Bug 1010241 - nfs: crash with nfs process
nfs: crash with nfs process
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
mainline
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Raghavendra G
:
Depends On: 1010239
Blocks: 1099270 1166278 1196898
  Show dependency treegraph
 
Reported: 2013-09-20 06:54 EDT by santosh pradhan
Modified: 2015-02-26 18:30 EST (History)
6 users (show)

See Also:
Fixed In Version: glusterfs-3.6.0beta1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1010239
: 1196898 (view as bug list)
Environment:
Last Closed: 2014-11-11 03:24:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description santosh pradhan 2013-09-20 06:54:37 EDT
+++ This bug was initially created as a clone of Bug #1010239 +++

Description of problem:

executed the tool nfstest_posix tool

Version-Release number of selected component (if applicable):
glusterfs-3.4.0.33rhs-1.el6rhs

How reproducible:
always

Steps to Reproduce:
1. execute nfstest-posix testsuite.
2.
3.

Actual results:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2013-09-20 10:13:38configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.0.33rhs
/lib64/libc.so.6[0x3cdd832960]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/nfs/server.so(nfs3_stat_to_fattr3+0x28)[0x7f56114568f8]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/nfs/server.so(nfs3_fill_getattr3res+0x35)[0x7f5611456b25]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/nfs/server.so(nfs3_getattr_reply+0x3a)[0x7f561144494a]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/nfs/server.so(nfs3svc_getattr_stat_cbk+0x4d)[0x7f561144776d]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/nfs/server.so(nfs_fop_stat_cbk+0x41)[0x7f561143bc91]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/debug/io-stats.so(io_stats_stat_cbk+0xf6)[0x7f56118f7196]
/usr/lib64/libglusterfs.so.0(default_stat_cbk+0xc2)[0x3981426742]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/cluster/distribute.so(dht_attr2+0x234)[0x7f5611d438a4]
/usr/lib64/glusterfs/3.4.0.33rhs/xlator/cluster/distribute.so(+0xb196)[0x7f5611d1a196]
/usr/lib64/libglusterfs.so.0(synctask_wrap+0x2a)[0x398144983a]
/lib64/libc.so.6[0x3cdd843bb0]
---------

Expected results:
crash is not expected

Additional info:
Comment 1 santosh pradhan 2013-09-24 07:19:46 EDT
NFS must do a NULL check in nfs3_stat_to_fattr3() routine avoid SEGV but the problem seems to be from DHT side. DHT sends down upto NFS with op_errno as 0 and op_ret as -1 which looks confusing. Needs more investigation who is masking the actual errno. The way dht_migration_complete_check() does lookup for the file, is also not clear.  Will work with DHT team.
Comment 2 Anand Avati 2013-10-03 01:22:47 EDT
REVIEW: http://review.gluster.org/6026 (gNFS: NFS segfaults with nfstest_posix tool) posted (#1) for review on master by Santosh Pradhan (spradhan@redhat.com)
Comment 3 Anand Avati 2013-11-04 02:24:21 EST
REVIEW: http://review.gluster.org/6026 (gNFS: NFS segfaults with nfstest_posix tool) posted (#2) for review on master by Santosh Pradhan (spradhan@redhat.com)
Comment 4 santosh pradhan 2013-11-04 05:55:31 EST
I am done with the FIX from NFS side. I am assigning the bug to Raghavendra Gowdappa so that he can put the FIX from DHT side. Idea is to commit the changes from same BZ id.
Comment 5 Anand Avati 2013-11-04 06:07:58 EST
REVIEW: http://review.gluster.org/6219 (cluster/dht: set op_errno correctly during migration.) posted (#1) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 6 Anand Avati 2013-11-04 20:33:44 EST
COMMIT: http://review.gluster.org/6026 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 0b2487d3bc8bc526d9b08698ea1434e94a6420d5
Author: Santosh Kumar Pradhan <spradhan@redhat.com>
Date:   Fri Sep 20 16:58:47 2013 +0530

    gNFS: NFS segfaults with nfstest_posix tool
    
    Problem:
    nfs3_stat_to_fattr3() missed a NULL check.
    
    FIX:
    (1) Added a NULL check.
    (2) In all fop cbk path, if the op_ret is -1 and op_errno is 0,
        then handle it as a special case. Set the NFS3 status as
        NFS3ERR_SERVERFAULT instead of NFS3_OK.
    (3) The other component of FIX would be in DHT module and
        is on the way.
    
    Change-Id: I6f03c9a02d794f8b807574f2755094dab1b90c92
    BUG: 1010241
    Signed-off-by: Santosh Kumar Pradhan <spradhan@redhat.com>
    Reviewed-on: http://review.gluster.org/6026
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
Comment 7 Anand Avati 2013-12-18 01:02:28 EST
REVIEW: http://review.gluster.org/6219 (cluster/dht: set op_errno correctly during migration.) posted (#2) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 8 Anand Avati 2013-12-18 23:14:10 EST
REVIEW: http://review.gluster.org/6219 (cluster/dht: set op_errno correctly during migration.) posted (#3) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 9 Anand Avati 2014-01-20 02:31:44 EST
REVIEW: http://review.gluster.org/6219 (cluster/dht: set op_errno correctly during migration.) posted (#4) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 10 Anand Avati 2014-01-22 20:09:08 EST
REVIEW: http://review.gluster.org/6219 (cluster/dht: set op_errno correctly during migration.) posted (#5) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 11 Anand Avati 2014-01-25 03:03:35 EST
COMMIT: http://review.gluster.org/6219 committed in master by Anand Avati (avati@redhat.com) 
------
commit c0c1210ffd5a45a47a3ad6a61545145f1a8e129c
Author: Raghavendra G <rgowdapp@redhat.com>
Date:   Mon Nov 4 16:21:14 2013 +0530

    cluster/dht: set op_errno correctly during migration.
    
    Change-Id: I65acedf92c1003975a584a2ac54527e9a2a1e52f
    BUG: 1010241
    Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-on: http://review.gluster.org/6219
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Anand Avati <avati@redhat.com>
Comment 12 Anand Avati 2014-01-27 05:32:59 EST
REVIEW: http://review.gluster.org/6817 (cluster/dht: set op_errno correctly during migration.) posted (#1) for review on release-3.5 by Raghavendra G (rgowdapp@redhat.com)
Comment 13 Anand Avati 2014-01-27 21:19:19 EST
REVIEW: http://review.gluster.org/6817 (cluster/dht: set op_errno correctly during migration.) posted (#2) for review on release-3.5 by Raghavendra G (rgowdapp@redhat.com)
Comment 14 Anand Avati 2014-01-28 11:47:08 EST
COMMIT: http://review.gluster.org/6817 committed in release-3.5 by Vijay Bellur (vbellur@redhat.com) 
------
commit 0483c68e0d05f1285baff22609f2ec6be65a5306
Author: Raghavendra G <rgowdapp@redhat.com>
Date:   Mon Nov 4 16:21:14 2013 +0530

    cluster/dht: set op_errno correctly during migration.
    
    Change-Id: I65acedf92c1003975a584a2ac54527e9a2a1e52f
    BUG: 1010241
    Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-on: http://review.gluster.org/6219
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Anand Avati <avati@redhat.com>
    Reviewed-on: http://review.gluster.org/6817
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Comment 15 Niels de Vos 2014-06-02 10:00:38 EDT
This can possibly get closed when bug 1099270 is fixed.
Comment 17 Niels de Vos 2014-09-22 08:32:18 EDT
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/
Comment 18 Niels de Vos 2014-11-11 03:24:10 EST
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users

Note You need to log in before you can comment on or make changes to this bug.