Bug 820580

Description of problem: I was running bonnie++ with a distributed-replicated volume with RHS Beta2 AMI, bonnie++ failed with ENOTEMPTY. This was on a nfs mount



Version-Release number of selected component (if applicable): 3.3.0qa38


How reproducible: Consistently


Steps to Reproduce:
1. cd /mnt/gluster; bonnie++ -u root 
2.
3.
  
Actual results:

bonnie++ -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...Bonnie: drastic I/O error (rmdir): Directory not empty
Cleaning up test directory after error. 


Additional info:

NFS log:


[2012-05-09 10:45:03.606326] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-1: background  entry self-heal
completed on /run8145/p5
[2012-05-09 10:45:05.104762] I [afr-common.c:1347:afr_launch_self_heal] 0-distrep-replicate-0: background  entry self-heal triggered. path: /
run8145/p8/d4/d8, reason: lookup detected pending operations
[2012-05-09 10:45:05.112299] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-distrep-replicate-0: /run8145/p8/d4/d8: Performing conservative merge
[2012-05-09 10:45:05.141122] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-0: background  entry self-heal completed on /run8145/p8/d4/d8
[2012-05-09 10:45:05.435075] I [afr-common.c:1347:afr_launch_self_heal] 0-distrep-replicate-0: background  entry self-heal triggered. path: /run8145/p9/d1, reason: lookup detected pending operations
[2012-05-09 10:45:05.443347] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-distrep-replicate-0: /run8145/p9/d1: Performing conservative merge
[2012-05-09 10:45:05.481506] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-0: background  entry self-heal completed on /run8145/p9/d1
[2012-05-09 10:48:56.606212] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 79da588f: /Bonnie.6988 => -1 (Directory not empty)
[2012-05-09 10:50:10.886630] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument
[2012-05-09 10:50:10.886722] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d
[2012-05-09 10:50:10.902569] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: bcb598f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
[2012-05-09 10:50:10.906248] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument
[2012-05-09 10:50:10.906287] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d
[2012-05-09 10:50:10.906306] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: ccb598f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
[2012-05-09 10:50:10.908904] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument
[2012-05-09 10:50:10.908934] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d
[2012-05-09 10:50:10.908951] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: dcb598f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
[2012-05-09 10:50:10.909950] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument
[2012-05-09 10:50:10.909983] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d