Bug 1717784

Summary: Ganesha-gfapi logs are flooded with error messages related to "gf_uuid_is_null(gfid)) [Invalid argument]" when lookups are running from multiple clients
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: libgfapiAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: medium Docs Contact:
Priority: medium    
Version: rhgs-3.5CC: amukherj, dang, ffilz, grajoria, jthottan, mbenjamin, pasik, rhs-bugs, skoduri, storage-qa-internal, vdas
Target Milestone: ---   
Target Release: RHGS 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1718316 (view as bug list) Environment:
Last Closed: 2019-10-30 12:21:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1718316, 1718338, 1720633, 1720634, 1720635, 1720636, 1729221, 1729223    
Bug Blocks: 1696809    

Description Manisha Saini 2019-06-06 08:10:23 UTC
Description of problem:
=========================

Ganesha-gfapi logs are flooded with errors messages related to gf_uuid_is_null(gfid), when linux untars and lookups are running from multiple clients-

---------
[2019-06-06 07:56:12.503603] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7f7e91e8b0ae] -->/lib64/libgfapi.so.0(+0x258f1) [0x7f7e91ea28f1] -->/lib64/libgfapi.so.0(+0x257c4) [0x7f7e91ea27c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]

---------


Version-Release number of selected component (if applicable):
===========================
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 Beta (Maipo)

# rpm -qa | grep ganesha
nfs-ganesha-2.7.3-3.el7rhgs.x86_64
glusterfs-ganesha-6.0-3.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.7.3-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.7.3-3.el7rhgs.x86_64



How reproducible:
=====================
2/2


Steps to Reproduce:
======================
1.Create 4 node Ganesha cluster
2.Create 4*3 Distribute-replicate Volume.Export the volume via Ganesha
3.Mount the volume on 4 clients via v4.1 protocol 
4.Run the following workload

Client 1: Run linux untars
Client 2: du -sh in loop
Client 3: ls -lRt in loop
Client 4: find's in loop

Actual results:
==================

While test is running,ganesha-gfapi logs are flooded with errors related to "gf_uuid_is_null"

======
[2019-06-03 16:54:19.829136] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7ff6902d00ae] -->/lib64/libgfapi.so.0(+0x2594a) [0x7ff6902e794a] -->/lib64/libgfapi.so.0(+0x257c4) [0x7ff6902e77c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
[2019-06-03 16:54:20.006163] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7ff6902d00ae] -->/lib64/libgfapi.so.0(+0x2594a) [0x7ff6902e794a] -->/lib64/libgfapi.so.0(+0x257c4) [0x7ff6902e77c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
[2019-06-03 16:54:20.320293] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7ff6902d00ae] -->/lib64/libgfapi.so.0(+0x2594a) [0x7ff6902e794a] -->/lib64/libgfapi.so.0(+0x257c4) [0x7ff6902e77c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
=====

# cat /var/log/ganesha/ganesha-gfapi.log | grep gf_uuid_is_null | wc -l
605340


Expected results:
===================
There should not be error messages in ganesha-gfapi.logs

Additional info:
===================

On narrowing down the test scenario,Seems to be the error messages are coming when only du -sh and ls -lRt are running in loop from two different clients

Comment 11 Manisha Saini 2019-08-09 10:03:56 UTC
Verified this BZ with 

# rpm -qa | grep ganesha
nfs-ganesha-2.7.3-7.el7rhgs.x86_64
glusterfs-ganesha-6.0-11.el7rhgs.x86_64
nfs-ganesha-gluster-2.7.3-7.el7rhgs.x86_64

Steps performed:
================
1.Create 4 node Ganesha cluster
2.Create 4*3 Distribute-replicate Volume.Export the volume via Ganesha
3.Mount the volume on 4 clients via v4.1 protocol 
4.Run the following workload

Client 1: Run linux untars
Client 2: du -sh in loop
Client 3: ls -lRt in loop
Client 4: find's in loop

No such error messages were observed.Moving this BZ to verified state.

Comment 13 errata-xmlrpc 2019-10-30 12:21:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249