Ganesha-gfapi logs are flooded with error messages related to "gf_uuid_is_null(gfid)) [Invalid argument]" when lookups are running from multiple clients
Description of problem:
=========================
Ganesha-gfapi logs are flooded with errors messages related to gf_uuid_is_null(gfid), when linux untars and lookups are running from multiple clients-
---------
[2019-06-06 07:56:12.503603] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7f7e91e8b0ae] -->/lib64/libgfapi.so.0(+0x258f1) [0x7f7e91ea28f1] -->/lib64/libgfapi.so.0(+0x257c4) [0x7f7e91ea27c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
---------
Version-Release number of selected component (if applicable):
===========================
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 Beta (Maipo)
# rpm -qa | grep ganesha
nfs-ganesha-2.7.3-3.el7rhgs.x86_64
glusterfs-ganesha-6.0-3.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.7.3-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.7.3-3.el7rhgs.x86_64
How reproducible:
=====================
2/2
Steps to Reproduce:
======================
1.Create 4 node Ganesha cluster
2.Create 4*3 Distribute-replicate Volume.Export the volume via Ganesha
3.Mount the volume on 4 clients via v4.1 protocol
4.Run the following workload
Client 1: Run linux untars
Client 2: du -sh in loop
Client 3: ls -lRt in loop
Client 4: find's in loop
Actual results:
==================
While test is running,ganesha-gfapi logs are flooded with errors related to "gf_uuid_is_null"
======
[2019-06-03 16:54:19.829136] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7ff6902d00ae] -->/lib64/libgfapi.so.0(+0x2594a) [0x7ff6902e794a] -->/lib64/libgfapi.so.0(+0x257c4) [0x7ff6902e77c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
[2019-06-03 16:54:20.006163] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7ff6902d00ae] -->/lib64/libgfapi.so.0(+0x2594a) [0x7ff6902e794a] -->/lib64/libgfapi.so.0(+0x257c4) [0x7ff6902e77c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
[2019-06-03 16:54:20.320293] E [glfs-handleops.c:1892:glfs_h_find_handle] (-->/lib64/libgfapi.so.0(+0xe0ae) [0x7ff6902d00ae] -->/lib64/libgfapi.so.0(+0x2594a) [0x7ff6902e794a] -->/lib64/libgfapi.so.0(+0x257c4) [0x7ff6902e77c4] ) 0-glfs_h_find_handle: invalid argument: !(gf_uuid_is_null(gfid)) [Invalid argument]
=====
# cat /var/log/ganesha/ganesha-gfapi.log | grep gf_uuid_is_null | wc -l
605340
Expected results:
===================
There should not be error messages in ganesha-gfapi.logs
Additional info:
===================
On narrowing down the test scenario,Seems to be the error messages are coming when only du -sh and ls -lRt are running in loop from two different clients
Verified this BZ with
# rpm -qa | grep ganesha
nfs-ganesha-2.7.3-7.el7rhgs.x86_64
glusterfs-ganesha-6.0-11.el7rhgs.x86_64
nfs-ganesha-gluster-2.7.3-7.el7rhgs.x86_64
Steps performed:
================
1.Create 4 node Ganesha cluster
2.Create 4*3 Distribute-replicate Volume.Export the volume via Ganesha
3.Mount the volume on 4 clients via v4.1 protocol
4.Run the following workload
Client 1: Run linux untars
Client 2: du -sh in loop
Client 3: ls -lRt in loop
Client 4: find's in loop
No such error messages were observed.Moving this BZ to verified state.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2019:3249