Description of problem:
=======================
On a distributed-replicate volume, Bonnie test suite failed with "Can't open file" error.
===========================TESTS RUNNING===========================
Changing to the specified mountpoint
/mnt/nfs/run15370
executing bonnie
Using uid:0, gid:0.
Writing with putc()...
done
Can't open file ./Bonnie.15391.005 <----------------------------
real 5m22.818s
user 4m22.679s
sys 0m47.933s
bonnie failed
0
Total 0 tests were successful
Switching over to the previous working directory
Removing /mnt/nfs//run15370/
rmdir: failed to remove ‘/mnt/nfs//run15370/’: Directory not empty
rmdir failed:Directory not empty
Version-Release number of selected component (if applicable):
3.8.4-11.el7rhgs.x86_64
How reproducible:
================
1/1
Steps to Reproduce:
===================
1) Create a distributed-replicate volume and start it (Please see the vol info output for enabled volume settings).
2) Mount it on multiples clients via gNFS.
3) Start bonnie test suite from one client and from the remaining clients start a infinite loop of lookups.
Bonnie test suite failed with "Can't open file" error and I can see the below errors in nfs logs.
[2017-01-16 10:52:08.042767] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.001: dentry not found in a32137f8-6941-4db4-b723-10019fd4f359
[2017-01-16 10:52:08.386420] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.003: dentry not found in 3c05ba77-5bad-4da8-8d5c-135fa9366d5f
[2017-01-16 10:52:08.956338] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.004: dentry not found in 670abac1-f5d3-42b2-b0b2-4900c1915a9a
[2017-01-16 10:52:09.067140] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.005: dentry not found in 3f678a33-317b-4cde-a959-0e038cdc04ca
[2017-01-16 10:52:09.125807] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126007] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126267] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126465] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126533] W [MSGID: 108019] [afr-lk-common.c:1090:is_blocking_locks_count_sufficient] 0-newdr-replicate-4: Unable to obtain blocking inode lock on even one child for gfid:00000000-0000-0000-0000-000000000000.
[2017-01-16 10:52:09.126614] I [MSGID: 108019] [afr-transaction.c:1829:afr_post_blocking_inodelk_cbk] 0-newdr-replicate-4: Blocking inodelks failed.
[2017-01-16 10:52:09.126758] W [MSGID: 112199] [nfs3-helpers.c:3515:nfs3_log_newfh_res] 0-nfs-nfsv3: /run15370/Bonnie.15391.005 => (XID: 8f57191d, CREATE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)), FH: exportid 9e098339-67c4-4975-bdfa-7d5278a3aae8, gfid 61b4583d-b71a-4dfc-92ad-cb4dd7d0fe53, mountid 0e315778-0000-0000-0000-000000000000
[2017-01-16 10:52:11.821905] W [MSGID: 112032] [nfs3.c:3600:nfs3svc_rmdir_cbk] 0-nfs: a557191d: /run15370 => -1 (Directory not empty) [Directory not empty]
Actual results:
===============
Bonnie test suite failed with "Can't open file" error.
Expected results:
================
Bonnie test suite should complete without any errors/issues.