Description of problem: ======================= On a distributed-replicate volume, Bonnie test suite failed with "Can't open file" error. ===========================TESTS RUNNING=========================== Changing to the specified mountpoint /mnt/nfs/run15370 executing bonnie Using uid:0, gid:0. Writing with putc()... done Can't open file ./Bonnie.15391.005 <---------------------------- real 5m22.818s user 4m22.679s sys 0m47.933s bonnie failed 0 Total 0 tests were successful Switching over to the previous working directory Removing /mnt/nfs//run15370/ rmdir: failed to remove ‘/mnt/nfs//run15370/’: Directory not empty rmdir failed:Directory not empty Version-Release number of selected component (if applicable): 3.8.4-11.el7rhgs.x86_64 How reproducible: ================ 1/1 Steps to Reproduce: =================== 1) Create a distributed-replicate volume and start it (Please see the vol info output for enabled volume settings). 2) Mount it on multiples clients via gNFS. 3) Start bonnie test suite from one client and from the remaining clients start a infinite loop of lookups. Bonnie test suite failed with "Can't open file" error and I can see the below errors in nfs logs. [2017-01-16 10:52:08.042767] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.001: dentry not found in a32137f8-6941-4db4-b723-10019fd4f359 [2017-01-16 10:52:08.386420] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.003: dentry not found in 3c05ba77-5bad-4da8-8d5c-135fa9366d5f [2017-01-16 10:52:08.956338] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.004: dentry not found in 670abac1-f5d3-42b2-b0b2-4900c1915a9a [2017-01-16 10:52:09.067140] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.005: dentry not found in 3f678a33-317b-4cde-a959-0e038cdc04ca [2017-01-16 10:52:09.125807] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0 [2017-01-16 10:52:09.126007] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0 [2017-01-16 10:52:09.126267] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0 [2017-01-16 10:52:09.126465] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0 [2017-01-16 10:52:09.126533] W [MSGID: 108019] [afr-lk-common.c:1090:is_blocking_locks_count_sufficient] 0-newdr-replicate-4: Unable to obtain blocking inode lock on even one child for gfid:00000000-0000-0000-0000-000000000000. [2017-01-16 10:52:09.126614] I [MSGID: 108019] [afr-transaction.c:1829:afr_post_blocking_inodelk_cbk] 0-newdr-replicate-4: Blocking inodelks failed. [2017-01-16 10:52:09.126758] W [MSGID: 112199] [nfs3-helpers.c:3515:nfs3_log_newfh_res] 0-nfs-nfsv3: /run15370/Bonnie.15391.005 => (XID: 8f57191d, CREATE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)), FH: exportid 9e098339-67c4-4975-bdfa-7d5278a3aae8, gfid 61b4583d-b71a-4dfc-92ad-cb4dd7d0fe53, mountid 0e315778-0000-0000-0000-000000000000 [2017-01-16 10:52:11.821905] W [MSGID: 112032] [nfs3.c:3600:nfs3svc_rmdir_cbk] 0-nfs: a557191d: /run15370 => -1 (Directory not empty) [Directory not empty] Actual results: =============== Bonnie test suite failed with "Can't open file" error. Expected results: ================ Bonnie test suite should complete without any errors/issues.
upstream patch : http://review.gluster.org/16421