Bug 1122374

Description of problem:

Directory deletion fails with 'Permission denied' error , its not removed from only one of the subvolume after performing rebalance and rm -rf from nfs mount

Version-Release number of selected component (if applicable):
3.6.0.25-1.el6rhs.x86_64


How reproducible:
Tried only once

Steps to Reproduce:
1. created a 4 brick distribute volume and create a testuser
2. mount the volume as NFS mount, grant full permissions on the mount point 
3. login as testuser
4. started linux kernel untar
5. once untarring is done did add-brick and rebalance
6. once rebalance is over , did rm -rf on untarred directory

Actual results:
though rm -rf succeeded , it throws an error " [test@localhost another]$ rm -rf linux-3.15.5
rm: cannot remove `linux-3.15.5': Permission denied

"

In the backend deletion failed on only one of the subvolume, it still has the directory with no files in it, this is the newly added brick before rebalance "

[root@rhs-client4 an3]# getfattr -d -m . -e hex /home/an3/linux-3.15.5/
getfattr: Removing leading '/' from absolute path names
# file: home/an3/linux-3.15.5/
trusted.gfid=0x6ea1c52162244cba8c37c77b9b7e75b3
trusted.glusterfs.dht=0x00000001000000006666666699999998

"


NFS log on the server says 
====================
[2014-07-22 10:06:16.250713] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-another-client-4: remote operation failed: No such file or directory. Path: <gfid:76021bb6-da04-41dd-b21b-482b9434a87e> (76021bb6-da04-41dd-b21b-482b9434a87e)
[2014-07-22 10:06:16.462763] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-another-client-2: remote operation failed: No such file or directory. Path: <gfid:894b34f4-1ce6-4b3d-a869-bc5753765722> (894b34f4-1ce6-4b3d-a869-bc5753765722)
[2014-07-22 10:06:16.468425] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-another-client-2: remote operation failed: No such file or directory. Path: <gfid:f3abb38f-f174-4525-8f26-26bc86ae254a> (f3abb38f-f174-4525-8f26-26bc86ae254a)
[2014-07-22 22:05:01.230436] I [MSGID: 100011] [glusterfsd.c:1247:reincarnate] 0-glusterfsd: Fetching the volume file from server...
[2014-07-22 22:05:01.231457] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2014-07-23 05:27:31.835678] W [client-rpc-fops.c:679:client3_3_rmdir_cbk] 0-another-client-4: remote operation failed: Permission denied
[2014-07-23 05:27:31.836308] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 6127c863: /linux-3.15.5 => -1 (Permission denied)




[root@rhs-client4 an3]# gluster v info  another
 
Volume Name: another
Type: Distribute
Volume ID: 9b8708d5-745f-4cb2-874f-edf0aa55b927
Status: Started
Snap Volume: no
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: rhs-client4.lab.eng.blr.redhat.com:/home/an0
Brick2: rhs-client39.lab.eng.blr.redhat.com:/home/an1
Brick3: rhs-client4.lab.eng.blr.redhat.com:/home/an2
Brick4: rhs-client39.lab.eng.blr.redhat.com:/home/an2
Brick5: rhs-client4.lab.eng.blr.redhat.com:/home/an3
Options Reconfigured:
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable


attaching the sosreports