Bug 1336630

Summary: ERROR and Warning message on writing a file from mount point "null gfid for path (null)" repeated 3 times between"
Product: [Community] GlusterFS Reporter: Karan Sandha <ksandha>
Component: posixAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: mainlineCC: bugs, jthottan, ksandha, nbalacha, pkarampu, rgowdapp
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: glusterfs-3.9.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1341477 (view as bug list) Environment:
Last Closed: 2017-03-27 18:25:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1334664, 1341477, 1341482, 1368608    

Description Karan Sandha 2016-05-17 05:50:20 UTC
Description of problem:
Multiple error messages as below while writing a file on  mount point.

0-nfs-posix: null gfid for path (null)" repeated 3 times between [2016-05-17 11:04:49.001762] and [2016-05-17 11:04:49.005541]

0-nfs-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-05-17 11:04:49.001812] and [2016-05-17 11:04:49.005542]
Version-Release number of selected component (if applicable):

sosreports are placed at qe.eng.blr.redhat.com:/var/www/html/sosreports/<bug>
How reproducible:
2/2 

Steps to Reproduce:
1. Create 1X3 volume with arbiter as 1 brick
2. From the mount point using fuse, dd files in a directory. (using urandom)
3. Continuous ERROR and Warning message appear in the brick logs 

Actual results:
[root@dhcp43-153 ~]# tailf /var/log/glusterfs/bricks/rhs-brick2-nfs.log 
[2016-05-17 11:04:49.001762] E [MSGID: 113091] [posix.c:179:posix_lookup] 0-nfs-posix: null gfid for path (null)
[2016-05-17 11:04:49.001812] W [MSGID: 113018] [posix.c:197:posix_lookup] 0-nfs-posix: lstat on null failed [Invalid argument]
[2016-05-17 11:04:54.915753] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 66fc0b36-2c1f-4e01-b21d-729b7b8896e8
[2016-05-17 11:04:54.915795] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-nfs-server: accepted client from dhcp43-153.lab.eng.blr.redhat.com-2920-2016/05/17-11:04:54:481605-nfs-client-2-0-0 (version: 3.9dev)
[2016-05-17 11:04:54.931408] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-nfs-server: disconnecting connection from dhcp43-153.lab.eng.blr.redhat.com-2920-2016/05/17-11:04:54:481605-nfs-client-2-0-0
[2016-05-17 11:04:54.931462] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-nfs-server: Shutting down connection dhcp43-153.lab.eng.blr.redhat.com-2920-2016/05/17-11:04:54:481605-nfs-client-2-0-0
The message "E [MSGID: 113091] [posix.c:179:posix_lookup] 0-nfs-posix: null gfid for path (null)" repeated 3 times between [2016-05-17 11:04:49.001762] and [2016-05-17 11:04:49.005541]
The message "W [MSGID: 113018] [posix.c:197:posix_lookup] 0-nfs-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-05-17 11:04:49.001812] and [2016-05-17 11:04:49.005542]
[2016-05-17 11:14:50.001911] E [MSGID: 113091] [posix.c:179:posix_lookup] 0-nfs-posix: null gfid for path (null)
[2016-05-17 11:14:50.001944] W [MSGID: 113018] [posix.c:197:posix_lookup] 0-nfs-posix: lstat on null failed [Invalid argument]
The message "E [MSGID: 113091] [posix.c:179:posix_lookup] 0-nfs-posix: null gfid for path (null)" repeated 3 times between [2016-05-17 11:14:50.001911] and [2016-05-17 11:14:50.005700]
The message "W [MSGID: 113018] [posix.c:197:posix_lookup] 0-nfs-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-05-17 11:14:50.001944] and [2016-05-17 11:14:50.005702]


Expected results:
There shouldn't be any error logs while writing the files on the mount point.

Additional info:

Comment 1 Jiffin 2016-05-31 08:55:30 UTC
Clearing the needinfo which was put by mistake

Comment 3 Nithya Balachandran 2016-05-31 15:15:07 UTC
Additional points:

1) I see this only with dist-rep volumes. These messages do not show up with pure distribute volumes.
2) These messages are repeated every 10 mins.
[2016-05-31 14:40:55.806055] E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)
[2016-05-31 14:40:55.806057] W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]
[2016-05-31 14:50:58.009270] E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)
[2016-05-31 14:50:58.009322] W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]
The message "E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)" repeated 3 times between [2016-05-31 14:50:58.009270] and [2016-05-31 14:50:58.037658]
The message "W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-05-31 14:50:58.009322] and [2016-05-31 14:50:58.037661]
[2016-05-31 15:00:59.014309] E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)
[2016-05-31 15:00:59.014367] W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]
The message "E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)" repeated 3 times between [2016-05-31 15:00:59.014309] and [2016-05-31 15:00:59.040114]
The message "W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-05-31 15:00:59.014367] and [2016-05-31 15:00:59.040117]
[2016-05-31 15:11:00.005350] E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)
[2016-05-31 15:11:00.005401] W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]
The message "E [MSGID: 113091] [posix.c:179:posix_lookup] 0-gs1-posix: null gfid for path (null)" repeated 3 times between [2016-05-31 15:11:00.005350] and [2016-05-31 15:11:00.033801]
The message "W [MSGID: 113018] [posix.c:197:posix_lookup] 0-gs1-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-05-31 15:11:00.005401] and [2016-05-31 15:11:00.033803]

Comment 4 Pranith Kumar K 2016-05-31 16:02:44 UTC
Nithya, Do you know of any steps to recreate this issue?

Comment 5 Nithya Balachandran 2016-05-31 16:12:11 UTC
I managed to see these by simply creating a dist-rep volume (4x2) , creating a few files from a Fuse mount, and waiting. :)

Comment 6 Vijay Bellur 2016-05-31 16:40:08 UTC
REVIEW: http://review.gluster.org/14589 (features/index: Exclude gfid-type for '.', '..') posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 7 Vijay Bellur 2016-06-01 07:34:42 UTC
COMMIT: http://review.gluster.org/14589 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit a89a9d266c439800286f281655d67f4e362dec32
Author: Pranith Kumar K <pkarampu>
Date:   Tue May 31 21:51:22 2016 +0530

    features/index: Exclude gfid-type for '.', '..'
    
    index_get_gfid_type is assuming all names are gfids where as some of these
    entry->d_names can be '.' or '..'
    
    Thanks a lot to Nithya for RC
    
    BUG: 1336630
    Change-Id: I06ad688a5865ab25b4f6c8a91af8c7fb2ed62186
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/14589
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Krutika Dhananjay <kdhananj>

Comment 8 Karan Sandha 2016-11-04 10:50:16 UTC
Verified the issue on gluster build 3.8.4.3 and the error messsages and warning are not seen now. Moving the bug to verified.

Comment 9 Shyamsundar 2017-03-27 18:25:08 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report.

glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html
[2] https://www.gluster.org/pipermail/gluster-users/