Description of problem: I was running bonnie++ with a distributed-replicated volume with RHS Beta2 AMI, bonnie++ failed with ENOTEMPTY. This was on a nfs mount Version-Release number of selected component (if applicable): 3.3.0qa38 How reproducible: Consistently Steps to Reproduce: 1. cd /mnt/gluster; bonnie++ -u root 2. 3. Actual results: bonnie++ -u root Using uid:0, gid:0. Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...Bonnie: drastic I/O error (rmdir): Directory not empty Cleaning up test directory after error. Additional info: NFS log: [2012-05-09 10:45:03.606326] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-1: background entry self-heal completed on /run8145/p5 [2012-05-09 10:45:05.104762] I [afr-common.c:1347:afr_launch_self_heal] 0-distrep-replicate-0: background entry self-heal triggered. path: / run8145/p8/d4/d8, reason: lookup detected pending operations [2012-05-09 10:45:05.112299] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-distrep-replicate-0: /run8145/p8/d4/d8: Performing conservative merge [2012-05-09 10:45:05.141122] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-0: background entry self-heal completed on /run8145/p8/d4/d8 [2012-05-09 10:45:05.435075] I [afr-common.c:1347:afr_launch_self_heal] 0-distrep-replicate-0: background entry self-heal triggered. path: /run8145/p9/d1, reason: lookup detected pending operations [2012-05-09 10:45:05.443347] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-distrep-replicate-0: /run8145/p9/d1: Performing conservative merge [2012-05-09 10:45:05.481506] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-0: background entry self-heal completed on /run8145/p9/d1 [2012-05-09 10:48:56.606212] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 79da588f: /Bonnie.6988 => -1 (Directory not empty) [2012-05-09 10:50:10.886630] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument [2012-05-09 10:50:10.886722] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d [2012-05-09 10:50:10.902569] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: bcb598f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2012-05-09 10:50:10.906248] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument [2012-05-09 10:50:10.906287] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d [2012-05-09 10:50:10.906306] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: ccb598f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2012-05-09 10:50:10.908904] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument [2012-05-09 10:50:10.908934] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d [2012-05-09 10:50:10.908951] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: dcb598f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2012-05-09 10:50:10.909950] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b650e4d6-4cd8-4a10-8eb2-493aecc5506d>: Invalid argument [2012-05-09 10:50:10.909983] E [nfs3.c:1517:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.248.66.215:793) distrep : b650e4d6-4cd8-4a10-8eb2-493aecc5506d
[2012-05-09 10:45:05.481506] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-distrep-replicate-0: background entry self-heal completed on /run8145/p9/d1 [2012-05-09 10:48:56.606212] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 79da588f: /Bonnie.6988 => -1 (Directory not empty) AFR returns ENOTEMPTY after selfheal. Reassigning it to AFR.
*** This bug has been marked as a duplicate of bug 814052 ***