Bug 1258196

Summary:	gNFSd: NFS mount fails with "Remote I/O error"
Product:	[Community] GlusterFS	Reporter:	Niels de Vos <ndevos>
Component:	nfs	Assignee:	Niels de Vos <ndevos>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs
Target Milestone:	---	Keywords:	Patch, Triaged
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	glusterfs-3.8rc2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1258069
Clones:	1258197 (view as bug list)		Environment:
Last Closed:	2016-06-16 13:34:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1258069, 1258197

Description Niels de Vos 2015-08-30 07:47:48 UTC

+++ This bug was initially created as a clone of Bug #1258069 +++

Description of problem:
gNFSd throws Remote IO error for mounts to directories which have changed OOB from the target gNFSd (say from a FUSE mount).  Internally this is due to ESTALE (op_errno == 116) being returned to mnt3_resolve_subdir_cbk, this causes the code path to unroll with an error.  Per the AFR2 code comments, the correct behavior is for gNFSd to purge the inode from it's inode table and do a fresh lookup on the inode.

The question might follow why does mnt3_resolve_subdir_cbk get ESTALE?  This is because the LOOKUP request is actually sent to the bricks via gfid vs a full path lookup, and this optimization happens because the path successfully grep's the gNFSd inode table for the GFID.  This isn't incorrect behavior, but is the root cause of the ESTALE.

Version-Release number of selected component (if applicable):
v3.6.x (verified), probably 3.7.x but unverified.

How reproducible:
100%, see prove test.


Steps to Reproduce:
See prove test.

Actual results:
Mount returns with "Remote I/O error"

Expected results:
The mount should succeed.


Additional info:
See attached prove test and patch which resolves the bug.

--- Additional comment from  on 2015-08-28 22:57:54 CEST ---

Patch based off of FB GlusterFS v3.6.3, might not line up exactly but patching mnt3_resolve_subdir_cbk of mounts3.c per this patch should do the trick.

--- Additional comment from  on 2015-08-29 00:37:58 CEST ---

Comment 1 Anand Avati 2015-08-30 07:52:14 UTC

REVIEW: http://review.gluster.org/12046 (nfs: Fixes "Remote I/O error" mount failures) posted (#1) for review on master by Niels de Vos (ndevos)

Comment 2 Anand Avati 2015-08-30 19:01:19 UTC

REVIEW: http://review.gluster.org/12046 (nfs: Fixes "Remote I/O error" mount failures) posted (#2) for review on master by Niels de Vos (ndevos)

Comment 3 Anand Avati 2015-09-01 12:56:23 UTC

COMMIT: http://review.gluster.org/12046 committed in master by Jeff Darcy (jdarcy) 
------
commit 31000d1a62da4e8beafb6f5a7b30ae593479a1ce
Author: Richard Wareing <rwareing>
Date:   Thu Aug 27 21:06:37 2015 -0700

    nfs: Fixes "Remote I/O error" mount failures
    
    - Fixes issue where NFS mount fail with "Remove I/O error" after the
      target directory has been deleted and re-created after the gNFSd has
      already cached the inode of the first generation of the target
      directory.
    - The solution is to follow the guidance of the AFR2 comments and
      refresh the inode by deleting it from cache and looking it up
      again.
    
    BUG: 1258196
    Change-Id: I9c7d8bd460ee9e5ea0b5b47d23886b1afcdcd563
    Reported-by: Richard Wareing <rwareing>
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/12046
    Tested-by: Gluster Build System <jenkins.com>

Comment 4 Niels de Vos 2016-06-16 13:34:27 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user