Bug 1480594 - nfs process crashed in "nfs3_getattr"
nfs process crashed in "nfs3_getattr"
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
3.10
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Niels de Vos
: Triaged
Depends On: 1479030
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-11 08:45 EDT by Niels de Vos
Modified: 2017-08-21 09:42 EDT (History)
1 user (show)

See Also:
Fixed In Version: glusterfs-3.10.5
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1479030
Environment:
Last Closed: 2017-08-21 09:42:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Niels de Vos 2017-08-11 08:45:50 EDT
+++ This bug was initially created as a clone of Bug #1479030 +++

+++ This bug was initially created as a clone of Bug #1476871 +++

Description:
============

add missing NULL check in nfs3_call_state_wipe() 

(gdb) bt
#0  0x00007ff1cfea9205 in _gf_ref_put (ref=ref@entry=0x0) at refcount.c:36
#1  0x00007ff1c1997455 in nfs3_call_state_wipe (cs=cs@entry=0x0) at nfs3.c:559
#2  0x00007ff1c1998931 in nfs3_getattr (req=req@entry=0x7ff1bc0b26d0, fh=fh@entry=0x7ff1c2f76ae0) at nfs3.c:962
#3  0x00007ff1c1998c8a in nfs3svc_getattr (req=0x7ff1bc0b26d0) at nfs3.c:987
#4  0x00007ff1cfbfd8c5 in rpcsvc_handle_rpc_call (svc=0x7ff1bc03e500, trans=trans@entry=0x7ff1bc0c8020, msg=<optimized out>) at rpcsvc.c:695
#5  0x00007ff1cfbfdaab in rpcsvc_notify (trans=0x7ff1bc0c8020, mydata=<optimized out>, event=<optimized out>, data=<optimized out>) at rpcsvc.c:789
#6  0x00007ff1cfbff9e3 in rpc_transport_notify (this=this@entry=0x7ff1bc0c8020, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7ff1bc0038d0)
    at rpc-transport.c:538
#7  0x00007ff1c4a2e3d6 in socket_event_poll_in (this=this@entry=0x7ff1bc0c8020, notify_handled=<optimized out>) at socket.c:2306
#8  0x00007ff1c4a3097c in socket_event_handler (fd=21, idx=9, gen=19, data=0x7ff1bc0c8020, poll_in=1, poll_out=0, poll_err=0) at socket.c:2458
#9  0x00007ff1cfe950f6 in event_dispatch_epoll_handler (event=0x7ff1c2f76e80, event_pool=0x5618154d5ee0) at event-epoll.c:572
#10 event_dispatch_epoll_worker (data=0x56181551cbd0) at event-epoll.c:648
#11 0x00007ff1cec99e25 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ff1ce56634d in clone () from /lib64/libc.so.6
(gdb) 


Version:
========
mainline


How to reproduce:
=================
[Unknown] Hit above crash multiple times

--- Additional comment from Worker Ant on 2017-08-07 20:48:38 CEST ---

REVIEW: https://review.gluster.org/17989 (nfs : add NULL check for call state in nfs3_call_state_wipe) posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2017-08-07 20:57:32 CEST ---

REVIEW: https://review.gluster.org/17989 (nfs : add NULL check for call state in nfs3_call_state_wipe) posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2017-08-08 10:31:02 CEST ---

COMMIT: https://review.gluster.org/17989 committed in master by Niels de Vos (ndevos@redhat.com) 
------
commit 111d6bda9259126b0429113c9b8ba479958a4398
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Mon Aug 7 23:47:00 2017 +0530

    nfs : add NULL check for call state in nfs3_call_state_wipe
    
    Refcounting added for nfs call state in https://review.gluster.org/17696.
    This is based on assumption that call state won't NULL when it is freed.
    But currently gluster nfs server is crashing in different scenarios at
    nfs3_getattr() with following bt
    
    #0  0x00007ff1cfea9205 in _gf_ref_put (ref=ref@entry=0x0) at refcount.c:36
    #1  0x00007ff1c1997455 in nfs3_call_state_wipe (cs=cs@entry=0x0) at nfs3.c:559
    #2  0x00007ff1c1998931 in nfs3_getattr (req=req@entry=0x7ff1bc0b26d0, fh=fh@entry=0x7ff1c2f76ae0) at nfs3.c:962
    #3  0x00007ff1c1998c8a in nfs3svc_getattr (req=0x7ff1bc0b26d0) at nfs3.c:987
    #4  0x00007ff1cfbfd8c5 in rpcsvc_handle_rpc_call (svc=0x7ff1bc03e500, trans=trans@entry=0x7ff1bc0c8020, msg=<optimized out>) at rpcsvc.c:695
    #5  0x00007ff1cfbfdaab in rpcsvc_notify (trans=0x7ff1bc0c8020, mydata=<optimized out>, event=<optimized out>, data=<optimized out>) at rpcsvc.c:789
    #6  0x00007ff1cfbff9e3 in rpc_transport_notify (this=this@entry=0x7ff1bc0c8020, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7ff1bc0038d0)
        at rpc-transport.c:538
    #7  0x00007ff1c4a2e3d6 in socket_event_poll_in (this=this@entry=0x7ff1bc0c8020, notify_handled=<optimized out>) at socket.c:2306
    #8  0x00007ff1c4a3097c in socket_event_handler (fd=21, idx=9, gen=19, data=0x7ff1bc0c8020, poll_in=1, poll_out=0, poll_err=0) at socket.c:2458
    #9  0x00007ff1cfe950f6 in event_dispatch_epoll_handler (event=0x7ff1c2f76e80, event_pool=0x5618154d5ee0) at event-epoll.c:572
    #10 event_dispatch_epoll_worker (data=0x56181551cbd0) at event-epoll.c:648
    #11 0x00007ff1cec99e25 in start_thread () from /lib64/libpthread.so.0
    #12 0x00007ff1ce56634d in clone () from /lib64/libc.so.6
    
    This patch add previous NULL check move from __nfs3_call_state_wipe() to nfs3_call_state_wipe()
    
    Change-Id: I2d73632f4be23f14d8467be3d908b09b3a2d87ea
    BUG: 1479030
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: https://review.gluster.org/17989
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
Comment 1 Worker Ant 2017-08-11 08:46:57 EDT
REVIEW: https://review.gluster.org/18027 (nfs: add NULL check for call state in nfs3_call_state_wipe) posted (#1) for review on release-3.10 by Niels de Vos (ndevos@redhat.com)
Comment 2 Worker Ant 2017-08-11 16:04:04 EDT
COMMIT: https://review.gluster.org/18027 committed in release-3.10 by Shyamsundar Ranganathan (srangana@redhat.com) 
------
commit cfcc0e9d4b76cdf3a64189463ea61c58edd540de
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Mon Aug 7 23:47:00 2017 +0530

    nfs: add NULL check for call state in nfs3_call_state_wipe
    
    Refcounting added for nfs call state in https://review.gluster.org/17696.
    This is based on assumption that call state won't NULL when it is freed.
    But currently gluster nfs server is crashing in different scenarios at
    nfs3_getattr() with following bt
    
    #0  0x00007ff1cfea9205 in _gf_ref_put (ref=ref@entry=0x0) at refcount.c:36
    #1  0x00007ff1c1997455 in nfs3_call_state_wipe (cs=cs@entry=0x0) at nfs3.c:559
    #2  0x00007ff1c1998931 in nfs3_getattr (req=req@entry=0x7ff1bc0b26d0, fh=fh@entry=0x7ff1c2f76ae0) at nfs3.c:962
    #3  0x00007ff1c1998c8a in nfs3svc_getattr (req=0x7ff1bc0b26d0) at nfs3.c:987
    #4  0x00007ff1cfbfd8c5 in rpcsvc_handle_rpc_call (svc=0x7ff1bc03e500, trans=trans@entry=0x7ff1bc0c8020, msg=<optimized out>) at rpcsvc.c:695
    #5  0x00007ff1cfbfdaab in rpcsvc_notify (trans=0x7ff1bc0c8020, mydata=<optimized out>, event=<optimized out>, data=<optimized out>) at rpcsvc.c:789
    #6  0x00007ff1cfbff9e3 in rpc_transport_notify (this=this@entry=0x7ff1bc0c8020, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7ff1bc0038d0)
        at rpc-transport.c:538
    #7  0x00007ff1c4a2e3d6 in socket_event_poll_in (this=this@entry=0x7ff1bc0c8020, notify_handled=<optimized out>) at socket.c:2306
    #8  0x00007ff1c4a3097c in socket_event_handler (fd=21, idx=9, gen=19, data=0x7ff1bc0c8020, poll_in=1, poll_out=0, poll_err=0) at socket.c:2458
    #9  0x00007ff1cfe950f6 in event_dispatch_epoll_handler (event=0x7ff1c2f76e80, event_pool=0x5618154d5ee0) at event-epoll.c:572
    #10 event_dispatch_epoll_worker (data=0x56181551cbd0) at event-epoll.c:648
    #11 0x00007ff1cec99e25 in start_thread () from /lib64/libpthread.so.0
    #12 0x00007ff1ce56634d in clone () from /lib64/libc.so.6
    
    This patch add previous NULL check move from __nfs3_call_state_wipe() to
    nfs3_call_state_wipe()
    
    Cherry picked from commit 111d6bda9259126b0429113c9b8ba479958a4398:
    > Change-Id: I2d73632f4be23f14d8467be3d908b09b3a2d87ea
    > BUG: 1479030
    > Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    > Reviewed-on: https://review.gluster.org/17989
    > Smoke: Gluster Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    > Reviewed-by: Niels de Vos <ndevos@redhat.com>
    
    Change-Id: I2d73632f4be23f14d8467be3d908b09b3a2d87ea
    BUG: 1480594
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: https://review.gluster.org/18027
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Comment 3 Shyamsundar 2017-08-21 09:42:02 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.5, please open a new bug report.

glusterfs-3.10.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-August/000079.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.