Description of problem: the problem is seen till now only in one scenario, where root-squash is enabled on volume, after a umount and a subsequent mount of the same volume throws ls only for a single file, not for others. workaround, follow this sequence 1. umount the volume 2. disable root-squash on the volume 3. mount the volume 4. enable the root-squash on the same volume, the ls does not throw any error. Version-Release number of selected component (if applicable): glusterfs-3.6.0.7-1.el6rhs.x86_64 How reproducible: always Steps to Reproduce: 1. create a volume of 6x2 type 2. mount the volume over nfs 3. create directory say "dir3" 4. create a file say "3" and chmod 770 <mount-point>/dir3/3 5. enable root-squash 6. umount <mount-point> 7. mount <mount-point> 8. ls -lR <mount-point> Actual results: [root@rhsauto054 ~]# ls -lR /mnt/nfs-test/ /mnt/nfs-test/: total 0 drwxrwxrwx. 2 root root 76 May 27 12:05 dir1 drwxr-xr-x. 2 root root 44 May 27 11:48 dir2 drwxr-xr-x. 2 root root 44 May 27 11:48 dir3 drwxr-xr-x. 2 root root 36 May 27 11:48 dir4 /mnt/nfs-test/dir1: total 1 -rwxrwxrwx. 1 root root 6 May 27 11:49 1 -rw-r--r--. 1 root root 0 May 27 01:17 newfile -rw-r--r--. 1 nfsnobody nfsnobody 17 May 27 01:27 nobody-file /mnt/nfs-test/dir2: total 1 -rwxrwxrwx. 1 root root 6 May 27 11:49 2 /mnt/nfs-test/dir3: ls: /mnt/nfs-test/dir3/3: Permission denied total 0 -rwxrwx---. 1 root root 0 May 27 01:06 3 /mnt/nfs-test/dir4: total 0 check the permission denied error for dir3/3 nfs.log glimpse, [2014-05-26 20:59:03.738805] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) [2014-05-26 20:59:03.739852] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) [2014-05-26 20:59:08.975778] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) [2014-05-26 20:59:08.976858] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) Expected results: no permission denied expected Additional info:
Created attachment 899401 [details] nfs logs
The behaviour is seen on all the files that are created following the steps that are listed in the "Steps to reproduce" section. Packet trace shows expected ACCESS reply, allowing read and lookup permissions for the file. There are errors seen in server_getxattr_cbk.Yet to root cause the issue.
Capturing what we discussed earlier today. Permission denied comes from xlators/system/posix-acl/src/posix-acl.c:acl_permits(): ... 212 for (i = 0; i < acl->count; i++) { 213 switch (ace->tag) { ... 245 case POSIX_ACL_OTHER: 246 perm = (ctx->perm & S_IRWXO); 247 if (!found) 248 goto perm_check; ... 269 perm_check: 270 if ((perm & want) == want) { 271 goto green; 272 } else { 273 goto red; 274 } ... acl_permits() lands in the switch statement at POSIX_ACL_OTHER. 'found' is false, so it goes to 'perm_check', where the if-statement results in the 'else' branch. I do not remember what the 'want' was though... Please attach a tcpdump and output from rpcdebug captured on the NFS-client so that I can review it. Thanks!
The value of want was 4 in this case.
Created attachment 954119 [details] root-squash logs
Created attachment 954361 [details] nfs-client-logs Attaching rpcdebug output on the NFS client.
The fix as discussed needs more investigation and testing. Populating xdata with posix acl xattrs didn't work as expected.
https://code.engineering/gerrit/#/c/36603/
Build used is this, glusterfs-3.6.0.33-1.el6rhs.x86_64 Executed the below mentioned step and didn't the see the issue this time and nfs logs didn't report the similar logs as mentioned in description section, [root@rhsauto012 nfs-test]# ls -R dir3/ dir3/: 3 [root@rhsauto012 nfs-test]# cd [root@nfs1 ~]# gluster volume set vol0 server.root-squash on volume set: success [root@nfs1 ~]# gluster volume info Volume Name: vol0 Type: Distributed-Replicate Volume ID: 2527d332-11aa-4f4b-8163-7d44fda5f16c Status: Started Snap Volume: no Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: 10.70.37.74:/rhs/brick1/d1r1 Brick2: 10.70.37.89:/rhs/brick1/d1r2 Brick3: 10.70.37.91:/rhs/brick1/d2r1 Brick4: 10.70.37.133:/rhs/brick1/d2r2 Brick5: 10.70.37.74:/rhs/brick1/d3r1 Brick6: 10.70.37.89:/rhs/brick1/d3r2 Brick7: 10.70.37.91:/rhs/brick1/d4r1 Brick8: 10.70.37.133:/rhs/brick1/d4r2 Brick9: 10.70.37.74:/rhs/brick1/d5r1 Brick10: 10.70.37.89:/rhs/brick1/d5r2 Brick11: 10.70.37.91:/rhs/brick1/d6r1 Brick12: 10.70.37.133:/rhs/brick1/d6r2 Options Reconfigured: server.root-squash: on performance.readdir-ahead: on auto-delete: disable snap-max-soft-limit: 90 snap-max-hard-limit: 256 [root@rhsauto012 ~]# umount /mnt/nfs-test [root@rhsauto012 ~]# mount -t nfs -o vers=3 10.70.37.74:/vol0 /mnt/nfs-test [root@rhsauto012 ~]# ls -R /mnt/nfs-test/dir3/ /mnt/nfs-test/dir3/: 3
Meghana, Please review the edited doc text and sign-off.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0038.html