Bug 1258196 - gNFSd: NFS mount fails with "Remote I/O error"
Summary: gNFSd: NFS mount fails with "Remote I/O error"
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: nfs
Version: mainline
Hardware: All
OS: All
unspecified
medium
Target Milestone: ---
Assignee: Niels de Vos
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1258069 1258197
TreeView+ depends on / blocked
 
Reported: 2015-08-30 07:47 UTC by Niels de Vos
Modified: 2016-06-16 13:34 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Clone Of: 1258069
: 1258197 (view as bug list)
Environment:
Last Closed: 2016-06-16 13:34:27 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Niels de Vos 2015-08-30 07:47:48 UTC
+++ This bug was initially created as a clone of Bug #1258069 +++

Description of problem:
gNFSd throws Remote IO error for mounts to directories which have changed OOB from the target gNFSd (say from a FUSE mount).  Internally this is due to ESTALE (op_errno == 116) being returned to mnt3_resolve_subdir_cbk, this causes the code path to unroll with an error.  Per the AFR2 code comments, the correct behavior is for gNFSd to purge the inode from it's inode table and do a fresh lookup on the inode.

The question might follow why does mnt3_resolve_subdir_cbk get ESTALE?  This is because the LOOKUP request is actually sent to the bricks via gfid vs a full path lookup, and this optimization happens because the path successfully grep's the gNFSd inode table for the GFID.  This isn't incorrect behavior, but is the root cause of the ESTALE.

Version-Release number of selected component (if applicable):
v3.6.x (verified), probably 3.7.x but unverified.

How reproducible:
100%, see prove test.


Steps to Reproduce:
See prove test.

Actual results:
Mount returns with "Remote I/O error"

Expected results:
The mount should succeed.


Additional info:
See attached prove test and patch which resolves the bug.

--- Additional comment from  on 2015-08-28 22:57:54 CEST ---

Patch based off of FB GlusterFS v3.6.3, might not line up exactly but patching mnt3_resolve_subdir_cbk of mounts3.c per this patch should do the trick.

--- Additional comment from  on 2015-08-29 00:37:58 CEST ---

Comment 1 Anand Avati 2015-08-30 07:52:14 UTC
REVIEW: http://review.gluster.org/12046 (nfs: Fixes "Remote I/O error" mount failures) posted (#1) for review on master by Niels de Vos (ndevos)

Comment 2 Anand Avati 2015-08-30 19:01:19 UTC
REVIEW: http://review.gluster.org/12046 (nfs: Fixes "Remote I/O error" mount failures) posted (#2) for review on master by Niels de Vos (ndevos)

Comment 3 Anand Avati 2015-09-01 12:56:23 UTC
COMMIT: http://review.gluster.org/12046 committed in master by Jeff Darcy (jdarcy) 
------
commit 31000d1a62da4e8beafb6f5a7b30ae593479a1ce
Author: Richard Wareing <rwareing>
Date:   Thu Aug 27 21:06:37 2015 -0700

    nfs: Fixes "Remote I/O error" mount failures
    
    - Fixes issue where NFS mount fail with "Remove I/O error" after the
      target directory has been deleted and re-created after the gNFSd has
      already cached the inode of the first generation of the target
      directory.
    - The solution is to follow the guidance of the AFR2 comments and
      refresh the inode by deleting it from cache and looking it up
      again.
    
    BUG: 1258196
    Change-Id: I9c7d8bd460ee9e5ea0b5b47d23886b1afcdcd563
    Reported-by: Richard Wareing <rwareing>
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/12046
    Tested-by: Gluster Build System <jenkins.com>

Comment 4 Niels de Vos 2016-06-16 13:34:27 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.