Bug 508072 - Pulseaudio can't start when running with NFS4 home directories
Pulseaudio can't start when running with NFS4 home directories
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
11
All Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Dickson
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-25 09:37 EDT by Jonathan Dieter
Modified: 2010-06-28 09:17 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-06-28 09:17:32 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
In pa_make_secure_dir(), only check GID if group permissions != 0 (542 bytes, patch)
2009-06-25 09:37 EDT, Jonathan Dieter
no flags Details | Diff

  None (edit)
Description Jonathan Dieter 2009-06-25 09:37:12 EDT
Created attachment 349399 [details]
In pa_make_secure_dir(), only check GID if group permissions != 0

Description of problem:
When a user with a home directory that is accessed over NFS4 tries to run pulseaudio, he gets this error:

$ pulseaudio
E: core-util.c: getgrgid_r(513): Numerical result out of range
E: core-util.c: Failed to create secure directory: Permission denied
$

Version-Release number of selected component (if applicable):
pulseaudio-0.9.15-14.fc11.i586

How reproducible:
Always

Steps to Reproduce:
1. Log in
  
Actual results:
Pulseaudio doesn't start

Expected results:
Pulseaudio starts

Additional info:
We are running a network with about 100 Fedora 10 computers with user authentication using LDAP and home directories shared using NFSv3.  We're looking into moving to NFS4 when we upgrade our systems to Fedora 11 because of file locking problems with NFSv3.

The problem is that we use POSIX acls on our NFS server, and, while NFSv3 handles them transparently, NFS4 insists on using NFS acls, which don't translate quite right (well, they do for the most part, but it insists on setting the gid to 99 "nobody" for the user's home directories on the client, long story there).

When pulseaudio detects that the .pulse directory doesn't have the right gid, it quits, even though the permissions are 0700.  I am including a patch that changes pa_make_secure_dir() to only check the gid of the newly created directory if the group permissions for that directory are something other than 0.
Comment 1 Jonathan Dieter 2009-06-25 09:39:21 EDT
I believe the "Permission denied" error actually comes from when pulseaudio attempts to change group ownership of the directory, which fails.  It's not a fatal error, though, if the supplied patch is applied to pulseaudio.
Comment 2 Lennart Poettering 2009-06-25 11:08:15 EDT
Hmm, I don't really understand why the chgrp() call actually fails. Isn't this an issue in NFS4 that shouldn't be fixed in PA but in NFS4? Can you refer me to some docs that explain what's going on with the gid on nfs4 here?
Comment 3 Jonathan Dieter 2009-06-25 11:32:25 EDT
Uh, okay, I'm not totally sure what NFS4 is *supposed* to do, but let me try to explain *why* it's doing what it's doing.

On our server, the users' home directory permissions look like this:
$ ls -l -d /networld/usershare/users/jd001
drwxrwx---+ 128 jd001 Domain Users 16384 Jun 25 18:18 jd001
$ getfacl /networld/usershare/users/jd001
# file: /networld/usershare/users/jd001
# owner: jd001
# group: Domain\040Users
user::rwx
user:apache:r-x
group::---
group:Domain\040Admins:rwx
mask::rwx
other::---
default:user::rwx
default:group::---
default:group:Domain\040Admins:rwx
default:mask::rwx
default:other::---

As you can see the group permissions are rwx in the ls (because of the ACL mask), but, as you can see from the getfacl, "Domain Users" doesn't actually have rwx permissions on the directory.  If someone from "Domain Users" tries to access the directory, they will get "Permission denied" (which is what we want).

Now, NFS4 doesn't use POSIX acls, but instead they use NFS acls, which, at least from what I read somewhere, are closer to Windows acls than POSIX acls.  However, they are good enough to provide an automatic POSIX->NFS acl translator when your backend filesystem uses POSIX acls.  The down side is that the client sees something very different than what the server sees.

On the client:
$ ls -l -d /networld/usershare/users/jd001
drwxrwx--- 128 jd001 nobody 16384 2009-06-25 18:18 jd001
$ getfacl /networld/usershare/users/jd001
# file: /networld/usershare/users/jd001
# owner: jd001
# group: nobody
user::rwx
group::rwx
other::---
$ nfs4_getfacl /networld/usershare/users/jd001
A::OWNER@:rwaDxtTcCy
D::OWNER@:
D::apache@lesbg.loc:TC
A::apache@lesbg.loc:rxtcy
D::apache@lesbg.loc:waDTC
D:g:GROUP@:TC
A:g:GROUP@:tcy
D:g:Domain Admins@lesbg.loc:TC
A:g:Domain Admins@lesbg.loc:rwaDxtcy
D:g:GROUP@:rwaDxTC
D:g:Domain Admins@lesbg.loc:TC
A::EVERYONE@:tcy
D::EVERYONE@:rwaDxTC
A:fdi:OWNER@:rwaDxtTcCy
D:fdi:OWNER@:
D:fdig:GROUP@:TC
A:fdig:GROUP@:tcy
D:fdig:Domain Admins@lesbg.loc:TC
A:fdig:Domain Admins@lesbg.loc:rwaDxtcy
D:fdig:GROUP@:rwaDxTC
D:fdig:Domain Admins@lesbg.loc:TC
A:fdi:EVERYONE@:tcy
D:fdi:EVERYONE@:rwaDxTC

Both systems use the same LDAP server and neither have group 513 (Domain Users) set in /etc/group.
Comment 4 Lennart Poettering 2009-06-29 07:51:32 EDT
Hmm. I am tempted to say that this is brokeness in NFS4 if a directory created in $HOME by a normal user process and then chown()'ed to getuid()/getgid() is not owned by the group of that user.

To me it appears that this mishandling of chown() on NFS4 seems to be some serious departure from POSIX.

I will now reassign this to the kernel, in the hope that some NFS4 kernel folks will comment on this.

I mean, we can certainly patch PA not to verify the gid of the directory it creates. But before we do it I want to make sure this is really a good idea. So, NFS4 folks, feel free to reassign back to PA after commenting.
Comment 5 Dan Emmons 2009-07-02 00:37:27 EDT
I've had this problem on our client machines since we upgraded from Fedora 9 to Fedora 10. If I had to guess, I'd say this commit is where it started:

* Sun Jan 13 2008 Lubomir Kundrak <lkundrak@redhat.com> 0.9.8-4.1 - Actually add content to pulseaudio-0.9.8-create-dot-pulse.patch - Make the Source0 tag point to URL instead of a local file - Drop the nochown patch; it's not applied at all and no longer needed

(from http://koji.fedoraproject.org/koji/buildinfo?buildID=105436 )

Hope that helps at all. This has been a real pain for us here.
Comment 6 Jonathan Dieter 2009-07-30 05:32:48 EDT
Is there any comment on this from the NFS4 people?  It would be greatly appreciated.
Comment 7 Steve Dickson 2009-07-30 08:44:04 EDT
on the ls -l -d of the directory 'nobody' is coming up as the 
group. Is the gid exist (and the same) on both the client 
and server?

What is the  nfs4_setfacl command you are using to set
the ACL?
Comment 8 Jonathan Dieter 2009-07-30 09:34:05 EDT
The gid (513) does exist and is the same on both clients (as they are both using the same LDAP server for user information as well as authentication).

I'm not using nfs4_setfacl.  The server has several directories where permissions have been set using POSIX acls and they are getting automatically translated into NFS4 acls that I read using nfs4_getfacl.

The odd thing is that it's only the group "Domain Users", gid 513 that gives me this problem (which is a major problem as all of my users' primary group is "Domain Users").  Every other uid and gid maps perfectly (and there are 1000+ uids).

Both server and client are set up with the same nfs4 domain (lesbg.com).  Running rpc.idmapd -vvvv -f on the client gives me:
rpc.idmapd: nfs4_name_to_gid: calling nsswitch->name_to_gid
rpc.idmapd: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
rpc.idmapd: nfs4_name_to_gid: final return value is 0
rpc.idmapd: Client e: (group) name "Domain Admins@lesbg.loc" -> id "512"
rpc.idmapd: nfs4_name_to_gid: calling nsswitch->name_to_gid
rpc.idmapd: nfs4_name_to_gid: nsswitch->name_to_gid returned -22
rpc.idmapd: nfs4_name_to_gid: final return value is -22
rpc.idmapd: Client e: (group) name "nobody" -> id "99"

I would expect that second one should look like:
rpc.idmapd: nfs4_name_to_gid: calling nsswitch->name_to_gid
rpc.idmapd: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
rpc.idmapd: nfs4_name_to_gid: final return value is 0
rpc.idmapd: Client e: (group) name "Domain Users@lesbg.loc" -> id "513"

Server is running Centos 5.3 (fully updated).  This client is running Fedora 10 (fully updated), while other Fedora 11 clients have the same problem.
Comment 9 Steve Dickson 2009-07-30 14:28:19 EDT
>rpc.idmapd: nfs4_name_to_gid: calling nsswitch->name_to_gid
>rpc.idmapd: nfs4_name_to_gid: nsswitch->name_to_gid returned -22
This is telling me the nsswitch routines that convert a group
name into a gid failed because the routine can not find the
local domain. But that does not make sense since the local domain
name was found for the "Domain Admins" group name... 

Question: Does the server's rpc.idmapd debugging info look similar
to the clients? Meaning can the server resolve "Domain Admins"?

Also (just thinking out loud) I would if the fact the group
name is two words (with a space) instead of one word is
causing problems.... 

What versions are nfs-utils and nfs-utils-lib on both
the server and client? 

Also is 'Domain' set in /etc/idmapd.conf? If so is it
set to the same thing on both sides?
Comment 10 Jonathan Dieter 2009-07-30 18:00:56 EDT
Ok, I've tracked it down.  There were 168 users that were members of the "Domain Users" group.  This wasn't needed (and I'm not even sure why they were there) as all users should have "Domain Users" as their gid.

For some reason, rpc.idmapd craps out on the server (and it was the server, not the client; tested by mounting nfs4 on the server from the server) when the group has that many users.
Comment 11 Steve Dickson 2009-07-31 09:01:35 EDT
hmm... so your workaround was to decrease the number of users in
the "Domain Users" group. A couple of questions...

1) What was the number of users that stop the server from crapping out? 
2) would it be possible to give me a clue as to where and how the
   server was crapping out? 

Obviously decreasing the number of gids is not the best solution... :-\

tia...
Comment 12 Jonathan Dieter 2009-07-31 09:41:03 EDT
Well, I brought it down to eight in one go, so I don't have a magic number for you.  Remember, the users still have 513 (Domain Users) as their gid and are still members of the group.  It's just that the ldap group no longer lists them as memberUid's (equivalent to removing a user from a group in /etc/group, but leaving the user's gid in /etc/passwd to be the group's gid).

As for how the server was crapping out, this is what I was getting:

Jul 30 16:26:42 storage-server02 rpc.idmapd[27621]:  Server: (group) id "99" -> name "nobody@lesbg.loc"
Jul 30 16:27:44 storage-server02 rpc.idmapd[27621]: nfsdcb: authbuf=10.0.0.0/255.0.0.0 authtype=group
Jul 30 16:27:44 storage-server02 rpc.idmapd[27621]:  Server: (group) id "513" -> name "nobody"
Jul 30 16:28:27 storage-server02 rpc.idmapd[27621]: nfsdcb: authbuf=10.0.0.0/255.0.0.0 authtype=group
Jul 30 16:28:27 storage-server02 rpc.idmapd[27621]:  Server: (group) id "512" -> name "Domain Admins@lesbg.loc"
Jul 30 16:28:28 storage-server02 rpc.idmapd[27621]: nfsdcb: authbuf=10.0.0.0/255.0.0.0 authtype=group
Jul 30 16:28:28 storage-server02 rpc.idmapd[27621]:  Server: (group) id "1002" -> name "JuniorAdmin@lesbg.loc"

99, 512, and 1002 are all correct, but 513 is mapped to nobody.  After removing all the memberUid's, this is what I got:

Jul 31 00:53:43 storage-server02 rpc.idmapd[14422]:  Server: (user) id "500" -> name "jd001@lesbg.loc"
Jul 31 00:53:43 storage-server02 rpc.idmapd[14422]: nfsdcb: authbuf=10.0.0.0/255.0.0.0 authtype=group
Jul 31 00:53:43 storage-server02 rpc.idmapd[14422]:  Server: (group) id "513" -> name "Domain Users@lesbg.loc"
Jul 31 00:53:43 storage-server02 rpc.idmapd[14422]: nfsdcb: authbuf=10.0.0.0/255.0.0.0 authtype=group
Jul 31 00:53:43 storage-server02 rpc.idmapd[14422]:  Server: (group) id "512" -> name "Domain Admins@lesbg.loc"
Comment 13 Dan Emmons 2009-08-01 00:41:04 EDT
I think I should mention that the patch attached to this bug worked for me. I got the SRPMs for pulseaudio, made the necessary edits, recompiled them into a set of RPMs, and I have been using them on the clients on my network with great success.
Comment 14 Lennart Poettering 2009-08-01 12:41:42 EDT
(In reply to comment #13)
> I think I should mention that the patch attached to this bug worked for me. I
> got the SRPMs for pulseaudio, made the necessary edits, recompiled them into a
> set of RPMs, and I have been using them on the clients on my network with great
> success.  

Yepp, but the problem is an NFS4 problem, not a PA problem. It should be fixed in NFS4, not in PA.
Comment 15 Dan Emmons 2009-08-04 02:06:52 EDT
yes, I agree. I was just confirming that the workaround does work around it.
Comment 16 Bug Zapper 2010-04-27 11:15:08 EDT
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 17 Bug Zapper 2010-06-28 09:17:32 EDT
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.