Bug 1396793

Summary:	[Ganesha] : Ganesha crashes intermittently during nfs-ganesha restarts.
Product:	[Community] GlusterFS	Reporter:	Pranith Kumar K <pkarampu>
Component:	io-threads	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	mainline	CC:	amukherj, asoman, bturner, bugs, jthottan, ndevos, pkarampu, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.10.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1393526
Clones:	1397381 (view as bug list)		Environment:
Last Closed:	2017-03-06 17:35:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1393526, 1397381, 1397386

Comment 1 Worker Ant 2016-11-20 09:36:45 UTC

REVIEW: http://review.gluster.org/15888 (performance/io-threads: Exit threads in fini() as well) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 2 Worker Ant 2016-11-22 07:01:21 UTC

COMMIT: http://review.gluster.org/15888 committed in master by Raghavendra G (rgowdapp) 
------
commit 25817a8c868b6c1b8149117f13e4216a99e453aa
Author: Pranith Kumar K <pkarampu>
Date:   Fri Nov 18 13:30:08 2016 +0530

    performance/io-threads: Exit threads in fini() as well
    
    Problem:
    io-threads starts the thread in 'init()' but doesn't clean them up
    on 'fini()'. It relies on PARENT_DOWN to exit threads but there can
    be cases where event before PARENT_UP the graph init code can think
    of issuing fini(). This code path is hit when glfs_init() is called
    on a volume that is in 'stopped' state. It leads to a crash in ganesha
    process, because the io-thread tries to access freed memory.
    
    Fix:
    Ideal fix would be to wait for all fops in io-thread list to be completed on
    PARENT_DOWN, and have fini() do cleanup of threads. Because there is no proper
    documentation about how PARENT_DOWN/fini are supposed to be used,
    we are getting different kinds of sequences in different higher level protocols.
    So for now cleaning up in both PARENT_DOWN and fini(). Fuse doesn't call fini()
    gfapi is not calling PARENT_DOWN in some cases, so for now I don't see
    another way out.
    
    BUG: 1396793
    Change-Id: I9c9154e7d57198dbaff0f30d3ffc25f6d8088aec
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/15888
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 3 Shyamsundar 2017-03-06 17:35:27 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/