Bug 1314724

Summary:	Multi-threaded SHD support
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Sahina Bose <sabose>
Component:	replicate	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED ERRATA	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	medium	Docs Contact:
Priority:	high
Version:	rhgs-3.1	CC:	atalur, bugs, nchilaka, ndevos, pkarampu, ravishankar, rcyriac, rhinduja, rhs-bugs, rkavunga, rwareing, sasundar, smohan, storage-qa-internal
Target Milestone:	---	Keywords:	FutureFeature, ZStream
Target Release:	RHGS 3.1.3
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:	1221737	Environment:
Last Closed:	2016-06-23 05:10:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1221737, 1325857
Bug Blocks:	1258386, 1311386

Description Sahina Bose 2016-03-04 10:44:19 UTC

+++ This bug was initially created as a clone of Bug #1221737 +++

Multi-threading support is critical for two important use-cases:

Halo replication (separate patch) - Long distance replication are high
latency and parallel healing is required for performance. Use higher
(16-32 threads for such use-cases).

Traditional clusters where bricks are being healed from scratch w/ large
numbers of small files (4-8 threads should be sufficient for these
use-cases).

The net result is anywhere from 2-30x SHD performance depending on how
many threads you use and what kind of storage hardware you have backing
your bricks. For bricks with large numbers of small files, the effect is
especially dramatic.

NOTES: It's critical to ensure your bricks have a sufficient number of
threads available via the performance.io-thread-count volume options.
Based on my tests sizing this to 2x the number of SHD threads is a good
place to start. Failure to do so can DOS your bricks with SHD requests.

--- Additional comment from  on 2015-05-14 13:21:58 EDT ---



--- Additional comment from Mohammed Rafi KC on 2015-05-19 09:14:38 EDT ---

Thank you Richard!

--- Additional comment from Anand Avati on 2015-05-20 11:02:55 EDT ---

REVIEW: http://review.gluster.org/10851 (Multi-threaded SHD support) posted (#1) for review on master by Kaushal M (kaushal)

--- Additional comment from Anand Avati on 2015-08-28 03:07:46 EDT ---

REVIEW: http://review.gluster.org/10851 (Multi-threaded SHD support) posted (#2) for review on master by Kaushal M (kaushal)

Comment 3 Mike McCune 2016-03-28 23:19:36 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 6 Pranith Kumar K 2016-05-04 07:30:44 UTC

Documentation for the new feature is tracked as a separate bug, so moving doc-text -

Comment 7 Nag Pavan Chilakam 2016-06-03 10:15:51 UTC

QATP:
====
1)set the msh cluster.shd-max-threads to say 5
<number-of-threads>". 
now 5 files in a dir must be getting healed parallely.
If there are 8 files, 5 will get healed, then based on the wait queue the next set will be picked
If there are only 3 files then all 3 will be parallely healed
If there are 3 files in dir1 and 7 in dir2. then the heal is done parallely on 3 files in dir1 and then only dir2 is picked


Also some more cases are below:
http://etherpad.corp.redhat.com/1314724-bug-testcases


All the above cases passed, hence moving to verified

Also, raised a bug#1335070 change the misleading multi threaded self heal option 

test version:3.7.9-6

Moving to verified

Comment 9 errata-xmlrpc 2016-06-23 05:10:40 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240