Bug 1387894

Summary:	Regression caused by enabling client-io-threads by default
Product:	[Community] GlusterFS	Reporter:	Pranith Kumar K <pkarampu>
Component:	io-threads	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.9	CC:	bugs, pkarampu, skoduri
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	glusterfs-3.9.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1381830	Environment:
Last Closed:	2016-12-06 05:59:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1381830, 1412941
Bug Blocks:	1350744

Description Pranith Kumar K 2016-10-23 09:38:17 UTC

+++ This bug was initially created as a clone of Bug #1381830 +++

Description of problem:

As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1380619#c11,   there is a regression caused to gfapi applications with making client-io-threads option on by default. 
    
iot-worker threads spawned are not cleaned up as part of xlator->fini() and they could end up accessing invalid/freed memory.

we need to fix io-thread->fini() to cleanup those threads before exiting. Since it could be an intricate fix, we could try disabling io-threads by default till then.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Worker Ant on 2016-10-05 05:35:56 EDT ---

REVIEW: http://review.gluster.org/15616 (Revert "mgmt/glusterd: Enable client-io-threads by default") posted (#2) for review on master by soumya k (skoduri)

--- Additional comment from Worker Ant on 2016-10-09 12:18:41 EDT ---

REVIEW: http://review.gluster.org/15620 (performance/io-threads: Exit all threads on PARENT_DOWN) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Worker Ant on 2016-10-21 07:38:04 EDT ---

REVIEW: http://review.gluster.org/15620 (performance/io-threads: Exit all threads on PARENT_DOWN) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Worker Ant on 2016-10-22 03:59:06 EDT ---

REVIEW: http://review.gluster.org/15620 (performance/io-threads: Exit all threads on PARENT_DOWN) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Worker Ant on 2016-10-23 05:32:10 EDT ---

COMMIT: http://review.gluster.org/15620 committed in master by Raghavendra G (rgowdapp) 
------
commit d7a5ca16911caca03cec1112d4be56a9cda2ee30
Author: Pranith Kumar K <pkarampu>
Date:   Sun Oct 9 21:36:40 2016 +0530

    performance/io-threads: Exit all threads on PARENT_DOWN
    
    Problem:
    When glfs_fini() is called on a volume where client.io-threads is enabled,
    fini() will free up iothread xl's private structure but there would be some
    threads that are sleeping which would wakeup after the timedwait completes
    leading to accessing already free'd memory.
    
    Fix:
    As part of parent-down, exit all sleeping threads.
    
    BUG: 1381830
    Change-Id: I0bb8d90241112c355fb22ee3fbfd7307f475b339
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/15620
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 1 Worker Ant 2016-10-23 09:41:07 UTC

REVIEW: http://review.gluster.org/15701 (performance/io-threads: Exit all threads on PARENT_DOWN) posted (#1) for review on release-3.9 by Pranith Kumar Karampuri (pkarampu)

Comment 2 Worker Ant 2016-10-25 06:52:49 UTC

COMMIT: http://review.gluster.org/15701 committed in release-3.9 by Pranith Kumar Karampuri (pkarampu) 
------
commit 4731717aca7a1bd778ba497440a64c4c6eef1daf
Author: Pranith Kumar K <pkarampu>
Date:   Sun Oct 9 21:36:40 2016 +0530

    performance/io-threads: Exit all threads on PARENT_DOWN
    
    Problem:
    When glfs_fini() is called on a volume where client.io-threads is enabled,
    fini() will free up iothread xl's private structure but there would be some
    threads that are sleeping which would wakeup after the timedwait completes
    leading to accessing already free'd memory.
    
    Fix:
    As part of parent-down, exit all sleeping threads.
    
     >BUG: 1381830
     >Change-Id: I0bb8d90241112c355fb22ee3fbfd7307f475b339
     >Signed-off-by: Pranith Kumar K <pkarampu>
     >Reviewed-on: http://review.gluster.org/15620
     >Smoke: Gluster Build System <jenkins.org>
     >CentOS-regression: Gluster Build System <jenkins.org>
     >NetBSD-regression: NetBSD Build System <jenkins.org>
     >Reviewed-by: Raghavendra G <rgowdapp>
     >(cherry picked from commit d7a5ca16911caca03cec1112d4be56a9cda2ee30)
    
    Change-Id: I450e5db84c83fae3237aaa7915c08cd3ee9bde2c
    BUG: 1387894
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/15701
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 3 Aravinda VK 2016-10-27 05:21:58 UTC

glusterfs-3.9.0rc2 is released[1] and packages are available for different distributions[2] to test.

[1] http://www.gluster.org/pipermail/maintainers/2016-October/001601.html
[2] http://www.gluster.org/pipermail/maintainers/2016-October/001605.html and http://www.gluster.org/pipermail/maintainers/2016-October/001606.html

Comment 4 Aravinda VK 2016-12-06 05:59:22 UTC

Gluster 3.9 GA is released http://blog.gluster.org/2016/11/announcing-gluster-3-9/