Description of problem: Directory deletion fails with 'Permission denied' error , its not removed from only one of the subvolume after performing rebalance and rm -rf from nfs mount Version-Release number of selected component (if applicable): 3.6.0.25-1.el6rhs.x86_64 How reproducible: Tried only once Steps to Reproduce: 1. created a 4 brick distribute volume and create a testuser 2. mount the volume as NFS mount, grant full permissions on the mount point 3. login as testuser 4. started linux kernel untar 5. once untarring is done did add-brick and rebalance 6. once rebalance is over , did rm -rf on untarred directory Actual results: though rm -rf succeeded , it throws an error " [test@localhost another]$ rm -rf linux-3.15.5 rm: cannot remove `linux-3.15.5': Permission denied " In the backend deletion failed on only one of the subvolume, it still has the directory with no files in it, this is the newly added brick before rebalance " [root@rhs-client4 an3]# getfattr -d -m . -e hex /home/an3/linux-3.15.5/ getfattr: Removing leading '/' from absolute path names # file: home/an3/linux-3.15.5/ trusted.gfid=0x6ea1c52162244cba8c37c77b9b7e75b3 trusted.glusterfs.dht=0x00000001000000006666666699999998 " NFS log on the server says ==================== [2014-07-22 10:06:16.250713] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-another-client-4: remote operation failed: No such file or directory. Path: <gfid:76021bb6-da04-41dd-b21b-482b9434a87e> (76021bb6-da04-41dd-b21b-482b9434a87e) [2014-07-22 10:06:16.462763] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-another-client-2: remote operation failed: No such file or directory. Path: <gfid:894b34f4-1ce6-4b3d-a869-bc5753765722> (894b34f4-1ce6-4b3d-a869-bc5753765722) [2014-07-22 10:06:16.468425] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-another-client-2: remote operation failed: No such file or directory. Path: <gfid:f3abb38f-f174-4525-8f26-26bc86ae254a> (f3abb38f-f174-4525-8f26-26bc86ae254a) [2014-07-22 22:05:01.230436] I [MSGID: 100011] [glusterfsd.c:1247:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2014-07-22 22:05:01.231457] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2014-07-23 05:27:31.835678] W [client-rpc-fops.c:679:client3_3_rmdir_cbk] 0-another-client-4: remote operation failed: Permission denied [2014-07-23 05:27:31.836308] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 6127c863: /linux-3.15.5 => -1 (Permission denied) [root@rhs-client4 an3]# gluster v info another Volume Name: another Type: Distribute Volume ID: 9b8708d5-745f-4cb2-874f-edf0aa55b927 Status: Started Snap Volume: no Number of Bricks: 5 Transport-type: tcp Bricks: Brick1: rhs-client4.lab.eng.blr.redhat.com:/home/an0 Brick2: rhs-client39.lab.eng.blr.redhat.com:/home/an1 Brick3: rhs-client4.lab.eng.blr.redhat.com:/home/an2 Brick4: rhs-client39.lab.eng.blr.redhat.com:/home/an2 Brick5: rhs-client4.lab.eng.blr.redhat.com:/home/an3 Options Reconfigured: performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable attaching the sosreports