Bug 1314724 - Multi-threaded SHD support
Summary: Multi-threaded SHD support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: replicate
Version: rhgs-3.1
Hardware: All
OS: All
high
medium
Target Milestone: ---
: RHGS 3.1.3
Assignee: Pranith Kumar K
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On: 1221737 1325857
Blocks: Gluster-HC-1 1311386
TreeView+ depends on / blocked
 
Reported: 2016-03-04 10:44 UTC by Sahina Bose
Modified: 2016-09-17 12:09 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of: 1221737
Environment:
Last Closed: 2016-06-23 05:10:40 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1240 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 Update 3 2016-06-23 08:51:28 UTC

Description Sahina Bose 2016-03-04 10:44:19 UTC
+++ This bug was initially created as a clone of Bug #1221737 +++

Multi-threading support is critical for two important use-cases:

Halo replication (separate patch) - Long distance replication are high
latency and parallel healing is required for performance. Use higher
(16-32 threads for such use-cases).

Traditional clusters where bricks are being healed from scratch w/ large
numbers of small files (4-8 threads should be sufficient for these
use-cases).

The net result is anywhere from 2-30x SHD performance depending on how
many threads you use and what kind of storage hardware you have backing
your bricks. For bricks with large numbers of small files, the effect is
especially dramatic.

NOTES: It's critical to ensure your bricks have a sufficient number of
threads available via the performance.io-thread-count volume options.
Based on my tests sizing this to 2x the number of SHD threads is a good
place to start. Failure to do so can DOS your bricks with SHD requests.

--- Additional comment from  on 2015-05-14 13:21:58 EDT ---



--- Additional comment from Mohammed Rafi KC on 2015-05-19 09:14:38 EDT ---

Thank you Richard!

--- Additional comment from Anand Avati on 2015-05-20 11:02:55 EDT ---

REVIEW: http://review.gluster.org/10851 (Multi-threaded SHD support) posted (#1) for review on master by Kaushal M (kaushal)

--- Additional comment from Anand Avati on 2015-08-28 03:07:46 EDT ---

REVIEW: http://review.gluster.org/10851 (Multi-threaded SHD support) posted (#2) for review on master by Kaushal M (kaushal)

Comment 3 Mike McCune 2016-03-28 23:19:36 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 6 Pranith Kumar K 2016-05-04 07:30:44 UTC
Documentation for the new feature is tracked as a separate bug, so moving doc-text -

Comment 7 Nag Pavan Chilakam 2016-06-03 10:15:51 UTC
QATP:
====
1)set the msh cluster.shd-max-threads to say 5
<number-of-threads>". 
now 5 files in a dir must be getting healed parallely.
If there are 8 files, 5 will get healed, then based on the wait queue the next set will be picked
If there are only 3 files then all 3 will be parallely healed
If there are 3 files in dir1 and 7 in dir2. then the heal is done parallely on 3 files in dir1 and then only dir2 is picked


Also some more cases are below:
http://etherpad.corp.redhat.com/1314724-bug-testcases


All the above cases passed, hence moving to verified

Also, raised a bug#1335070 change the misleading multi threaded self heal option 

test version:3.7.9-6

Moving to verified

Comment 9 errata-xmlrpc 2016-06-23 05:10:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240


Note You need to log in before you can comment on or make changes to this bug.