Bug 1396793 - [Ganesha] : Ganesha crashes intermittently during nfs-ganesha restarts.
Summary: [Ganesha] : Ganesha crashes intermittently during nfs-ganesha restarts.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: io-threads
Version: mainline
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
Assignee: Pranith Kumar K
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1393526 1397381 1397386
TreeView+ depends on / blocked
 
Reported: 2016-11-20 09:35 UTC by Pranith Kumar K
Modified: 2017-03-06 17:35 UTC (History)
11 users (show)

Fixed In Version: glusterfs-3.10.0
Clone Of: 1393526
: 1397381 (view as bug list)
Environment:
Last Closed: 2017-03-06 17:35:27 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Worker Ant 2016-11-20 09:36:45 UTC
REVIEW: http://review.gluster.org/15888 (performance/io-threads: Exit threads in fini() as well) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 2 Worker Ant 2016-11-22 07:01:21 UTC
COMMIT: http://review.gluster.org/15888 committed in master by Raghavendra G (rgowdapp) 
------
commit 25817a8c868b6c1b8149117f13e4216a99e453aa
Author: Pranith Kumar K <pkarampu>
Date:   Fri Nov 18 13:30:08 2016 +0530

    performance/io-threads: Exit threads in fini() as well
    
    Problem:
    io-threads starts the thread in 'init()' but doesn't clean them up
    on 'fini()'. It relies on PARENT_DOWN to exit threads but there can
    be cases where event before PARENT_UP the graph init code can think
    of issuing fini(). This code path is hit when glfs_init() is called
    on a volume that is in 'stopped' state. It leads to a crash in ganesha
    process, because the io-thread tries to access freed memory.
    
    Fix:
    Ideal fix would be to wait for all fops in io-thread list to be completed on
    PARENT_DOWN, and have fini() do cleanup of threads. Because there is no proper
    documentation about how PARENT_DOWN/fini are supposed to be used,
    we are getting different kinds of sequences in different higher level protocols.
    So for now cleaning up in both PARENT_DOWN and fini(). Fuse doesn't call fini()
    gfapi is not calling PARENT_DOWN in some cases, so for now I don't see
    another way out.
    
    BUG: 1396793
    Change-Id: I9c9154e7d57198dbaff0f30d3ffc25f6d8088aec
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/15888
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 3 Shyamsundar 2017-03-06 17:35:27 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.