Description of problem: While deleting a directory on fuse mount, we see ENOENT errors. The issue persisted for a couple of "rm -rf" attempts and later we are able to successfuly delete the directory. 


Version-Release number of selected component (if applicable):
glusterfs-3.4.0.14rhs-1.el6rhs.x86_64


How reproducible: Saw this issue once


Steps to Reproduce:
1. Created a distributed-replicate volume.
2. Created a directory with 1000 files. 
3. Added 2 bricks and executed rebalance
4. Tried deleting the directory

Actual results:

Saw ENOENT errors while deleting the directory for a couple of times.

# tail /var/log/glusterfs/mnt-gluster.log 
[2013-07-31 11:35:56.974726] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-acl-client-3: remote operation failed: No such file or directory. Path: /files (f423eb7e-ee33-4a05-a36e-c11cad25dd3d)
[2013-07-31 11:35:56.981423] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-acl-client-2: remote operation failed: No such file or directory. Path: /files (f423eb7e-ee33-4a05-a36e-c11cad25dd3d)
[2013-07-31 11:35:56.981689] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-acl-client-3: remote operation failed: No such file or directory. Path: /files (f423eb7e-ee33-4a05-a36e-c11cad25dd3d)
[2013-07-31 11:35:57.093259] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-3: remote operation failed: No such file or directory
[2013-07-31 11:35:57.093859] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-2: remote operation failed: No such file or directory
[2013-07-31 11:35:57.095903] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-2: remote operation failed: No such file or directory
[2013-07-31 11:35:57.096430] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-3: remote operation failed: No such file or directory
[2013-07-31 11:35:57.097184] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-acl-client-3: remote operation failed: No such file or directory
[2013-07-31 11:35:57.097370] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-acl-client-2: remote operation failed: No such file or directory
[2013-07-31 11:35:57.097419] W [fuse-bridge.c:1673:fuse_unlink_cbk] 0-glusterfs-fuse: 184019: RMDIR() /files => -1 (No such file or directory)


Expected results:

Repeated attempts to remove the directory is an unacceptable behaviour.

Additional info:
# gluster volume info acl
 
Volume Name: acl
Type: Distributed-Replicate
Volume ID: 4c8234b4-8d74-4048-97a9-0d2f519996d1
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.37.49:/glance2/s1
Brick2: 10.70.37.120:/glance2/s2
Brick3: 10.70.37.132:/glance2/s3_new
Brick4: 10.70.37.120:/glance2/s4_new
Options Reconfigured:
diagnostics.client-log-level: INFO
diagnostics.brick-log-level: INFO
performance.stat-prefetch: off

# gluster volume status acl
Status of volume: acl
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.49:/glance2/s1                           49153   Y       13328
Brick 10.70.37.120:/glance2/s2                          49153   Y       13083
Brick 10.70.37.132:/glance2/s3_new                      49156   Y       18104
Brick 10.70.37.120:/glance2/s4_new                      49157   Y       11545
NFS Server on localhost                                 2049    Y       22750
Self-heal Daemon on localhost                           N/A     Y       22286
NFS Server on 10.70.37.120                              2049    Y       11844
Self-heal Daemon on 10.70.37.120                        N/A     Y       11564
NFS Server on 10.70.37.208                              2049    Y       17905
Self-heal Daemon on 10.70.37.208                        N/A     Y       17633
NFS Server on 10.70.37.132                              2049    Y       18358
Self-heal Daemon on 10.70.37.132                        N/A     Y       18123
 
           Task                                      ID         Status
           ----                                      --         ------
      Rebalance    0821e548-9f64-4426-8da6-69ecadd88b18              3


# df -h

10.70.37.49:acl       100G  112M  100G   1% /mnt/gluster