Bug 1132391

Summary:	NFS interoperability problem: stripe-xlator removes EOF at end of READDIR
Product:	[Community] GlusterFS	Reporter:	Niels de Vos <ndevos>
Component:	nfs	Assignee:	Niels de Vos <ndevos>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	low	Docs Contact:
Priority:	high
Version:	3.5.2	CC:	bugs, E3GH75, gluster-bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.5.3	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-11-21 16:01:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1130969
Bug Blocks:	1125231

Description Niels de Vos 2014-08-21 09:30:24 UTC

+++ This bug was initially created as a clone of Bug #1130969 +++
+++                                                           +++
+++ This bug is for backporting the fix to glusterfs-3.5      +++
+++                                                           +++

Description of problem (from Emmanuel Dreyfus):

NFS READDIR replies ar e made of a header, a sequence of
entriers, and a EOF flag. When GlusterFS's NFS server is
used along with stripe xlator, it fails to set the EOF
flag, which violates NFS RFC and confuses some clients.

The bug is caused because nfs xlator sets EOF if it gets
op_errno set to ENOENT. That value is produced in storage
xlator and propagated through server, client, and other
xlators until stripe xlator handles it. stripe only passed
op_errno if op_ret < 0, which is not the case here.

--- Additional comment from Niels de Vos on 2014-08-18 14:01:37 CEST ---

Example of missing EOF after the "Value Follows: No" marker.

    Frame 834: 1356 bytes on wire (10848 bits), 1356 bytes captured (10848 bits)
    Internet Protocol Version 4, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
    Remote Procedure Call, Type:Reply XID:0xe8e42eee
    Network File System, READDIRPLUS Reply
        [Program Version: 3]
        [V3 Procedure: READDIRPLUS (17)]
        Status: NFS3_OK (0)
        dir_attributes  Directory mode: 0755 uid: 0 gid: 0
            ...
        Verifier: Opaque Data
        Value Follows: Yes
        Entry: name e246f09c-2f58-417b-8c3e-6f7f5307154f
        ...
        Value Follows: Yes
        Entry: name 17cec36b-c553-47bc-a7da-4663f3efbac2
            ...
        Value Follows: No
        EOF: 0

--- Additional comment from Anand Avati on 2014-08-18 14:44:55 CEST ---

REVIEW: http://review.gluster.org/8493 (NFS: stripe-xlator should pass EOF at end of READDIR) posted (#3) for review on master by Niels de Vos (ndevos)

--- Additional comment from Niels de Vos on 2014-08-18 14:47:31 CEST ---

Confirmation that EOF is set with the patch:

    Frame 654: 1356 bytes on wire (10848 bits), 1356 bytes captured (10848 bits)
    Internet Protocol Version 4, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
    Transmission Control Protocol, Src Port: nfs (2049), Dst Port: accessnetwork (699), Seq: 19741, Ack: 1777, Len: 1288
    Remote Procedure Call, Type:Reply XID:0xa604116c
    Network File System, READDIRPLUS Reply
        [Program Version: 3]
        [V3 Procedure: READDIRPLUS (17)]
        Status: NFS3_OK (0)
        dir_attributes  Directory mode: 0755 uid: 0 gid: 0
            ...
        Entry: name e246f09c-2f58-417b-8c3e-6f7f5307154f
        ...
        Value Follows: Yes
        Entry: name 17cec36b-c553-47bc-a7da-4663f3efbac2
            ...
        Value Follows: No
        EOF: 1

--- Additional comment from Anand Avati on 2014-08-19 14:23:25 CEST ---

COMMIT: http://review.gluster.org/8493 committed in master by Niels de Vos (ndevos) 
------
commit 9b5231e5c98b8cfa116838287c7a14042702795f
Author: Niels de Vos <ndevos>
Date:   Mon Aug 18 12:11:35 2014 +0200

    NFS: stripe-xlator should pass EOF at end of READDIR
    
    NFS READDIR replies are made of a header, a sequence of
    entries, and a EOF flag. When GlusterFS's NFS server is
    used along with stripe xlator, it fails to set the EOF
    flag, which violates NFS RFC and confuses some clients.
    
    The bug is caused because nfs xlator sets EOF if it gets
    op_errno set to ENOENT. That value is produced in storage
    xlator and propagated through server, client, and other
    xlators until stripe xlator handles it. stripe only passed
    op_errno if op_ret < 0, which is not the case here. This
    change set adds a special case for that situation to fix
    the problem.
    
    Change-Id: Ie6db94b0515292387cfb04c1e4a9363f34fcd19a
    BUG: 1130969
    Reported-by: Emmanuel Dreyfus <manu>
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/8493
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Santosh Pradhan <spradhan>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Reviewed-by: Emmanuel Dreyfus <manu>
    Tested-by: Emmanuel Dreyfus <manu>

Comment 1 Anand Avati 2014-08-21 16:24:33 UTC

REVIEW: http://review.gluster.org/8509 (NFS: stripe-xlator should pass EOF at end of READDIR) posted (#1) for review on release-3.5 by Niels de Vos (ndevos)

Comment 2 Anand Avati 2014-08-26 16:53:13 UTC

COMMIT: http://review.gluster.org/8509 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit b47ce75a49dee95da17630e735db50baa9420729
Author: Niels de Vos <ndevos>
Date:   Thu Aug 21 18:23:38 2014 +0200

    NFS: stripe-xlator should pass EOF at end of READDIR
    
    NFS READDIR replies are made of a header, a sequence of
    entries, and a EOF flag. When GlusterFS's NFS server is
    used along with stripe xlator, it fails to set the EOF
    flag, which violates NFS RFC and confuses some clients.
    
    The bug is caused because nfs xlator sets EOF if it gets
    op_errno set to ENOENT. That value is produced in storage
    xlator and propagated through server, client, and other
    xlators until stripe xlator handles it. stripe only passed
    op_errno if op_ret < 0, which is not the case here. This
    change set adds a special case for that situation to fix
    the problem.
    
    Cherry picked from commit 9b5231e5c98b8cfa116838287c7a14042702795f:
    > Change-Id: Ie6db94b0515292387cfb04c1e4a9363f34fcd19a
    > BUG: 1130969
    > Reported-by: Emmanuel Dreyfus <manu>
    > Signed-off-by: Niels de Vos <ndevos>
    > Reviewed-on: http://review.gluster.org/8493
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Santosh Pradhan <spradhan>
    > Reviewed-by: Kaleb KEITHLEY <kkeithle>
    > Reviewed-by: Emmanuel Dreyfus <manu>
    > Tested-by: Emmanuel Dreyfus <manu>
    
    Change-Id: Ie6db94b0515292387cfb04c1e4a9363f34fcd19a
    BUG: 1132391
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/8509
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Santosh Pradhan <spradhan>

Comment 3 Niels de Vos 2014-09-09 07:26:31 UTC

*** Bug 1128820 has been marked as a duplicate of this bug. ***

Comment 4 Niels de Vos 2014-10-05 12:59:37 UTC

The first (and last?) Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-October/018990.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 5 Niels de Vos 2014-11-05 09:24:20 UTC

The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions have been made available on [2] to make testing easier.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html
[2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/

Comment 6 Niels de Vos 2014-11-21 16:01:58 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report.

glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/