Bug 1379720

Summary: errors appear in brick and nfs logs and getting stale files on NFS clients
Product: [Community] GlusterFS Reporter: Mohammed Rafi KC <rkavunga>
Component: nfsAssignee: Mohammed Rafi KC <rkavunga>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: mainlineCC: hklein, jbuchta, jthottan, kkeithle, mmalhotr, mzywusko, ndevos, olim, pkarampu, rgowdapp, rhs-bugs, skoduri, storage-qa-internal, tpetr
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.10.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1358096
: 1394634 1394635 (view as bug list) Environment:
Last Closed: 2017-03-06 17:28:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1358096, 1394634, 1394635    

Description Mohammed Rafi KC 2016-09-27 14:00:43 UTC 
Description of problem:

Rotated logs are not accessible from replicated gluster node via NFS mounts. We've gluster with distributed replica between two sites. After shared log is rotated by a server on site A many (not all!) nodes accessing the replicated log from a node at the B site get "cannot access .log: Stale file handle" error.   

RCA from Soumya and Pranith,


The theory which we have come up with is that -->

The reason NFS server is not doing fresh lookup after it received ESTALE could be that cs->lookuptype must have been set to GF_NFS3_FRESH instead of GF_NFS3_REVALIDATE. 

nfs_lookup() fop starts with setting lookuptype to GF_NFS3_REVALIDATE. But as per the code changes done in the patch mentioned in the comment#52, before doing STACK_WIND on the child xlator lookup, in case if we find cached inode for that file/entry name in the inode table but with inode_ctx not set, we reset lookuptype to GF_NFS3_FRESH. This may have led to nfs xlator not sending fresh lookup on receiving ESTALE.

So there could be various reasons for inode_ctx not being set. Underlying xlators could have done inode_link which we are ruling out for now as we do not see inode_link done with a file entry name (except for tiered volumes). Other possibility is that we do not set inode_ctx in readdirp_cbk path. Maybe 
* client1 has done readdirp on the directory 'jmsdomain6'..got the file handle of the file 'gcjrockit_jms06admin00a.log' as part of readdirp response.
That means we have inode entry of this file but with no inode_ctx set.
* meanwhile client2 has deleted and re-created this log file (probably as part of logrotate)
* Now client1 does lookup on the earlier filehandle it received resulting in ESTALE.
Comment 1 Niels de Vos 2016-09-29 13:06:45 UTC 
Patch posted: http://review.gluster.org/15580
Comment 2 Worker Ant 2016-10-12 07:57:17 UTC 
REVIEW: http://review.gluster.org/15580 (nfs: revalidate lookup converted to fresh lookup) posted (#2) for review on master by mohammed rafi  kc (rkavunga)
Comment 3 Worker Ant 2016-10-12 10:54:36 UTC 
REVIEW: http://review.gluster.org/15580 (nfs: revalidate lookup converted to fresh lookup) posted (#3) for review on master by mohammed rafi  kc (rkavunga)
Comment 4 Worker Ant 2016-10-12 10:56:47 UTC 
REVIEW: http://review.gluster.org/15580 (nfs: revalidate lookup converted to fresh lookup) posted (#4) for review on master by mohammed rafi  kc (rkavunga)
Comment 5 Worker Ant 2016-10-13 12:02:54 UTC 
REVIEW: http://review.gluster.org/15580 (nfs: revalidate lookup converted to fresh lookup) posted (#5) for review on master by mohammed rafi  kc (rkavunga)
Comment 6 Worker Ant 2016-11-10 22:19:13 UTC 
COMMIT: http://review.gluster.org/15580 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit ba7a737b1260bbafe22097bea08814035c8b655d
Author: Mohammed Rafi KC <rkavunga>
Date:   Tue Sep 27 19:01:48 2016 +0530

    nfs: revalidate lookup converted to fresh lookup
    
    when an inode ctx is missing for a linked inode the revalidate
    lookups are converted to fresh.
    
    This could result in sending ESTALE when the gfid are recreated
    
    We are not able to reproduce the issue with normal setup, most part of
    RCA was done with code reading.
    
    Possible scenario in which  this bug can reproduce,
    Delete a file and recreate a new file with same name, at the same time
    from another client process try to list/or access the file.
    
    In this case the second client may throw an ESTALE error for such files
    
    Thanks to Soumya and Pranith for doing the complete RCA
    
    Change-Id: I73992a65844b09a169cefaaedc0dcfb129d66ea1
    BUG: 1379720
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/15580
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: soumya k <skoduri>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Comment 7 Shyamsundar 2017-03-06 17:28:10 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/