Bug 763777 (GLUSTER-2045)

When using LDAP with a user that has multiple group memberships, only the primary group is honored when trying to write to an NFS or Gluster mounted volume.  Here is an excerpt from my system:

user1@wall-e:/mnt/cvol00$ groups
staff admins vcs 

user1@wall-e:/mnt/cvol00$ ls -lah
total 28K
drwxr-xr-x 4 root root     8.0K 2010-10-29 11:35 .
drwxr-xr-x 3 root root     4.0K 2010-10-27 14:28 ..
drwxrwsr-x 2 root staff 8.0K 2010-10-29 11:38 share


user1@wall-e:/mnt/cvol00$ touch share/test1
user1@wall-e:/mnt/cvol00$ ls -lah share/
total 24K
drwxrwsr-x 2 root     staff 8.0K 2010-10-29 11:48 .
drwxr-xr-x 4 root     root     8.0K 2010-10-29 11:35 ..
-rw-r--r-- 1 user1 staff    0 2010-10-29 11:48 test1

user1@wall-e:/mnt/cvol00$ sudo chgrp -R admins share/
user1@wall-e:/mnt/cvol00$ sudo chmod -R 2775 share/
user1@wall-e:/mnt/cvol00$ ls -lah
total 28K
drwxr-xr-x 4 root root   8.0K 2010-10-29 11:35 .
drwxr-xr-x 3 root root   4.0K 2010-10-27 14:28 ..
drwxrwsr-x 2 root admins 8.0K 2010-10-29 11:48 share

user1@wall-e:/mnt/cvol00$ touch share/test2
touch: cannot touch `share/test2': Permission denied

In the gluster brick log I see it only looking up the user's primary group ID (of the group 'staff'):

[2010-10-29 11:54:50.472210] T [auth-glusterfs.c:176:auth_glusterfs_authenticate] rpc-service: Auth Info: pid: 0, uid: 10001, gid: 11001, owner: 30

It never changes from "gid: 11001" which is the user's primary group ID.

I added some quick debug messages to find out if gluster was reporting the correct group ids -- forgive my "C" ignorance -- I'm horribly in-experienced:

[2010-11-01 17:03:18.389750] T [access-control.c:216:ac_test_access] access-control: Testing owner access
[2010-11-01 17:03:18.389763] T [access-control.c:226:ac_test_access] access-control: Testing group access
[2010-11-01 17:03:18.389775] T [access-control.c:126:ac_test_group_access] access-control: Group found
[2010-11-01 17:03:18.389787] T [access-control.c:129:ac_test_group_access] access-control: Group (owner) ID: 11000 Group ID: 0

[2010-11-01 17:03:18.389808] T [access-control.c:126:ac_test_group_access] access-control: Group found
[2010-11-01 17:03:18.389820] T [access-control.c:129:ac_test_group_access] access-control: Group (owner) ID: 11000 Group ID: 0

[2010-11-01 17:03:18.389841] T [access-control.c:126:ac_test_group_access] access-control: Group found
[2010-11-01 17:03:18.389852] T [access-control.c:129:ac_test_group_access] access-control: Group (owner) ID: 11000 Group ID: 0

[2010-11-01 17:03:18.389874] T [access-control.c:126:ac_test_group_access] access-control: Group found
[2010-11-01 17:03:18.389885] T [access-control.c:129:ac_test_group_access] access-control: Group (owner) ID: 11000 Group ID: 0

[2010-11-01 17:03:18.389906] T [access-control.c:238:ac_test_access] access-control: Testing other access
[2010-11-01 17:03:18.389924] T [access-control.c:246:ac_test_access] access-control: No access allowed

In the above it loops through 4 groups which is the amount of group memberships the user has.  It correctly finds the owner group. However, it keeps reporting a group id of 0 which doesn't correspond.  

This is about the extent that I can take it -- I'm not well versed in C enough to try to dig in the stat call.

The above is with the latest Git version.

Thanks Evans, that looks like a bug. I'll be looking into it.

Thanks!

Hi Evan

I tried to reproduce it and am not running into a permission denied.

Here is the sequence of commands I used. Can you please point out whether I am correctly replicating the commands:

shehjart@indus:glusterfs$ groups
shehjart adm dialout cdrom plugdev lpadmin admin sambashare
shehjart@indus:glusterfs$ mkdir /mnt/share
shehjart@indus:glusterfs$ ls /mnt -lah
total 16K
drwxrwxrwx  3 root     root     4.0K 2010-11-09 10:53 .
drwxr-xr-x 24 root     root     4.0K 2010-11-08 14:48 ..
drwxr-xr-x  2 shehjart shehjart 4.0K 2010-11-09 10:53 share
shehjart@indus:glusterfs$ chgrp admin /mnt/share
shehjart@indus:glusterfs$ ls /mnt -lah
total 16K
drwxrwxrwx  3 root     root  4.0K 2010-11-09 10:53 .
drwxr-xr-x 24 root     root  4.0K 2010-11-08 14:48 ..
drwxr-xr-x  2 shehjart admin 4.0K 2010-11-09 10:53 share
shehjart@indus:glusterfs$ chmod 2775  /mnt/share
shehjart@indus:glusterfs$ ls /mnt -lah
total 16K
drwxrwxrwx  3 root     root  4.0K 2010-11-09 10:53 .
drwxr-xr-x 24 root     root  4.0K 2010-11-08 14:48 ..
drwxrwsr-x  2 shehjart admin 4.0K 2010-11-09 10:53 share
shehjart@indus:glusterfs$ touch /mnt/share/testfile
shehjart@indus:glusterfs$ ls /mnt -lah
total 16K
drwxrwxrwx  3 root     root  4.0K 2010-11-09 10:53 .
drwxr-xr-x 24 root     root  4.0K 2010-11-08 14:48 ..
drwxrwsr-x  2 shehjart admin 4.0K 2010-11-09 10:54 share

Change the directory (/mnt/share) owner to root (or some other user) and leave the group as "admin".  I didn't get any error when my user owned the directory -- only when I tried to use my group membership.  (I was also using LDAP but since I assume it's an issue with the gluster access lookup it shouldn't matter).

I'm experiencing the same behavior on my 3.1 install.

I have a GlusterFS 3.1 volume being mounted on a client using NFS. From the client I created a directory under the mount point and set the permissions to root:groupa 750

My user account is a member of groupa on the client, yet I am unable to list the contents of the directory:

$ ls -l /gfs/dir1
ls: /gfs/dir1/: Permission denied

$ ls -ld /gfs/dir1
rwxr-x--- 9 root groupa 73728 Nov  9 09:44 /gfs/dir1/

$ groups
myuser groupa

I am able to list the directory as the user root. If I change the group ownership to my primary group, myuser, then I can successfully list the contents of the directory.

$ sudo chgrp myuser /gfs/dir1
$ ls -ld /gfs/dir1
rwxr-x--- 9 root myuser 73728 Nov  9 09:44 /gfs/dir1/

$ ls -l /gfs/dir1
drwxr-xr-x 5 root     root      73728 Mar 26  2010 testdir1
drwxr-x--- 4 root     root      73728 Apr  8  2010 testdir2
drwxr-x--- 2 root     root      73728 Aug  4 21:23 testdir3

The volume is being exported using the builtin GlusterFS NFS server. The servers and client are all CentOS 5.5 x86_64 boxes.

The bug is in auth-glusterfs.c. The authentication code is not copying the auxiliary group ids that were sent by protocol/client. Patch coming soon.

PATCH: http://patches.gluster.com/patch/5712 in master (rpc-lib: Copy auxiliary GIDs into request from de-XDRed buffer)

PATCH: http://patches.gluster.com/patch/5715 in master (rpc-lib: Fail auth-unix if more than 16 gids found)

PATCH: http://patches.gluster.com/patch/5716 in master (nfsrpc: Fail auth-unix if more than 16 gids found)

(In reply to comment #8)
> The bug is in auth-glusterfs.c. The authentication code is not copying the
> auxiliary group ids that were sent by protocol/client. Patch coming soon.

Thank you very much!

(In reply to comment #9)
> PATCH: http://patches.gluster.com/patch/5712 in master (rpc-lib: Copy auxiliary
> GIDs into request from de-XDRed buffer)

Confirmed on my end.  Tested with multiple group ownerships and everything is working as expected.  Thanks for the quick fix!

Evan

After upgrading my servers to 3.1.1 I'm still experiencing the issue with secondary groups not working with the client using NFS:

In this scenario root owns the directory and the admins non primary group (of which my id is a member) is the group:

$ groups
mikeh admins

$ ls -ld /research/project/
drwxrwx--- 10 root admins 73728 Nov 22 12:13 /gluster/project/

$ ls -l /gluster/project/
ls: /gluster/project/: Permission denied

$ mount|grep research
nas-01.mydom:/research-storage on /gluster type nfs (rw,addr=192.168.1.11)

Both Gluster servers and the client are CentOS 5 x86_64 systems. The servers had glusterfs-core / glusterfs-fuse 3.0.1-1 updated to 3.1.1-1 via the RPMs. The storage is a simple distributed file system, no striping or replication.

Following the update I rebooted both servers and remounted the NFS file system on the client.

(In reply to comment #14)
> After upgrading my servers to 3.1.1 I'm still experiencing the issue with
> secondary groups not working with the client using NFS:
> 
> In this scenario root owns the directory and the admins non primary group (of
> which my id is a member) is the group:
> 

Thanks. The problem being faced now is slightly different. A patch is on the way.

This is what is going on:

nfs3_stat_to_accessbits (buf=0x7fff246b2840, request=31, uid=1000, gid=1000) at nfs3-helpers.c:682
682	        uint32_t        accresult = 0;
(gdb) n
683	        ia_prot_t       prot = {0, };
(gdb) n
684	        ia_type_t       type = 0;
(gdb) n
686	        prot = buf->ia_prot;
(gdb) n
687	        type = buf->ia_type;
(gdb) n
689	        if (uid == 0)
(gdb) n
691	        else if (buf->ia_uid == uid)
(gdb) n
693	        else if (buf->ia_gid == gid)
(gdb) n
696	                accresult = nfs3_other_accessbits (prot, type, request);

That is, NFS server access callback is not accounting for supplementary group ids to check for non-primary membership.

Created attachment 394

(In reply to comment #15)
> (In reply to comment #14)
> > After upgrading my servers to 3.1.1 I'm still experiencing the issue with
> > secondary groups not working with the client using NFS:
> > 
> > In this scenario root owns the directory and the admins non primary group (of
> > which my id is a member) is the group:
> > 
> 
> Thanks. The problem being faced now is slightly different. A patch is on the
> way.

Mike, please test with the patch I've attached to this bug. It fixes the problem on my setup.

To apply it, you'll have to download the source on the NFS server machine:

$ git clone git://git.gluster.com/glusterfs.git ./glusterfs
$ cd glusterfs
$ git am 0001-nfs3-Access-cbk-must-account-for-auxgids-on-group-ac.patch

Then build and install the source. Once thats done, restart Glusterfs volumes and run the same commands again. Thanks

(In reply to comment #18)
> (In reply to comment #15)
> > (In reply to comment #14)
> Mike, please test with the patch I've attached to this bug. It fixes the
> problem on my setup.
> 
> To apply it, you'll have to download the source on the NFS server machine:
> 
> $ git clone git://git.gluster.com/glusterfs.git ./glusterfs
> $ cd glusterfs
> $ git am 0001-nfs3-Access-cbk-must-account-for-auxgids-on-group-ac.patch
> 
> Then build and install the source. Once thats done, restart Glusterfs volumes
> and run the same commands again. Thanks

I installed my servers via the CentOS rpms, so here's how I tested the above patch:

Downloaded the 3.1.1 src.rpm, extracted it, untar'd the source code, applied the patch, tar'd it back up, bumped the release to 3.1.1-2, rpmbuild and rpm -U

Following that the simple test listed in my previous test now works via an nfs client. I'll have to do some further testing to ensure that users who are not members of the secondary group aren't allowed access, etc...

Since the above method is using patched 3.1.1 source, the Gluster client access is still broken.

How do I generate a gluster-3.1.1.tar.gz from the git tree to get all of the latest changes so that I can test both NFS and Gluster clients? Do I simply do the git steps above, rename the directory to gluster-3.1.1 and compress, rpmbuild?

PATCH: http://patches.gluster.com/patch/5916 in master (nfs3: Access cbk must account for auxgids on group access checks)

This seems to have regressed, I'm experiencing the problem with with glusterfs-3.2.6-2.el6.x86_64 (from EPEL).  Is it possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=764789 ?  The gid handling code was removed from nfs3-helpers.c.