Bug 1413584 - [GNFS] Bonnie test suite failed with "Can't open file" error
Summary: [GNFS] Bonnie test suite failed with "Can't open file" error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: gluster-nfs
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Jiffin
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On: 1413971
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-16 12:59 UTC by Prasad Desala
Modified: 2019-07-23 04:43 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1413971 (view as bug list)
Environment:
Last Closed: 2019-07-23 04:43:37 UTC
Embargoed:


Attachments (Terms of Use)

Description Prasad Desala 2017-01-16 12:59:35 UTC
Description of problem:
=======================
On a distributed-replicate volume, Bonnie test suite failed with "Can't open file" error.

===========================TESTS RUNNING===========================
Changing to the specified mountpoint
/mnt/nfs/run15370
executing bonnie
Using uid:0, gid:0.
Writing with putc()...
done
Can't open file ./Bonnie.15391.005      <----------------------------

real	5m22.818s
user	4m22.679s
sys	0m47.933s
bonnie failed
0
Total 0 tests were successful
Switching over to the previous working directory
Removing /mnt/nfs//run15370/
rmdir: failed to remove ‘/mnt/nfs//run15370/’: Directory not empty
rmdir failed:Directory not empty

Version-Release number of selected component (if applicable):
3.8.4-11.el7rhgs.x86_64

How reproducible:
================
1/1

Steps to Reproduce:
===================
1) Create a distributed-replicate volume and start it (Please see the vol info output for enabled volume settings).
2) Mount it on multiples clients via gNFS.
3) Start bonnie test suite from one client and from the remaining clients start a infinite loop of lookups.

Bonnie test suite failed with "Can't open file" error and I can see the below errors in nfs logs.

[2017-01-16 10:52:08.042767] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.001: dentry not found in a32137f8-6941-4db4-b723-10019fd4f359
[2017-01-16 10:52:08.386420] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.003: dentry not found in 3c05ba77-5bad-4da8-8d5c-135fa9366d5f
[2017-01-16 10:52:08.956338] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.004: dentry not found in 670abac1-f5d3-42b2-b0b2-4900c1915a9a
[2017-01-16 10:52:09.067140] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: 09388be3-eb94-4cb1-94b0-0518e00d3c90/Bonnie.15391.005: dentry not found in 3f678a33-317b-4cde-a959-0e038cdc04ca
[2017-01-16 10:52:09.125807] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126007] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126267] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126465] E [client-common.c:844:client_pre_inodelk] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0xc740) [0x7f866b734740] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x28a70) [0x7f866b750a70] -->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x3aa66) [0x7f866b762a66] ) 0-: Assertion failed: 0
[2017-01-16 10:52:09.126533] W [MSGID: 108019] [afr-lk-common.c:1090:is_blocking_locks_count_sufficient] 0-newdr-replicate-4: Unable to obtain blocking inode lock on even one child for gfid:00000000-0000-0000-0000-000000000000.
[2017-01-16 10:52:09.126614] I [MSGID: 108019] [afr-transaction.c:1829:afr_post_blocking_inodelk_cbk] 0-newdr-replicate-4: Blocking inodelks failed.
[2017-01-16 10:52:09.126758] W [MSGID: 112199] [nfs3-helpers.c:3515:nfs3_log_newfh_res] 0-nfs-nfsv3: /run15370/Bonnie.15391.005 => (XID: 8f57191d, CREATE: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)), FH: exportid 9e098339-67c4-4975-bdfa-7d5278a3aae8, gfid 61b4583d-b71a-4dfc-92ad-cb4dd7d0fe53, mountid 0e315778-0000-0000-0000-000000000000
[2017-01-16 10:52:11.821905] W [MSGID: 112032] [nfs3.c:3600:nfs3svc_rmdir_cbk] 0-nfs: a557191d: /run15370 => -1 (Directory not empty) [Directory not empty]

Actual results:
===============
Bonnie test suite failed with "Can't open file" error.

Expected results:
================
Bonnie test suite should complete without any errors/issues.

Comment 5 Atin Mukherjee 2017-01-17 13:22:20 UTC
upstream patch : http://review.gluster.org/16421


Note You need to log in before you can comment on or make changes to this bug.