Bug 1314724 - Multi-threaded SHD support
Multi-threaded SHD support
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: replicate (Show other bugs)
All All
high Severity medium
: ---
: RHGS 3.1.3
Assigned To: Pranith Kumar K
: FutureFeature, ZStream
Depends On: 1221737 1325857
Blocks: Gluster-HC-1 1311386
  Show dependency treegraph
Reported: 2016-03-04 05:44 EST by Sahina Bose
Modified: 2016-09-17 08:09 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1221737
Last Closed: 2016-06-23 01:10:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Sahina Bose 2016-03-04 05:44:19 EST
+++ This bug was initially created as a clone of Bug #1221737 +++

Multi-threading support is critical for two important use-cases:

Halo replication (separate patch) - Long distance replication are high
latency and parallel healing is required for performance. Use higher
(16-32 threads for such use-cases).

Traditional clusters where bricks are being healed from scratch w/ large
numbers of small files (4-8 threads should be sufficient for these

The net result is anywhere from 2-30x SHD performance depending on how
many threads you use and what kind of storage hardware you have backing
your bricks. For bricks with large numbers of small files, the effect is
especially dramatic.

NOTES: It's critical to ensure your bricks have a sufficient number of
threads available via the performance.io-thread-count volume options.
Based on my tests sizing this to 2x the number of SHD threads is a good
place to start. Failure to do so can DOS your bricks with SHD requests.

--- Additional comment from  on 2015-05-14 13:21:58 EDT ---

--- Additional comment from Mohammed Rafi KC on 2015-05-19 09:14:38 EDT ---

Thank you Richard!

--- Additional comment from Anand Avati on 2015-05-20 11:02:55 EDT ---

REVIEW: http://review.gluster.org/10851 (Multi-threaded SHD support) posted (#1) for review on master by Kaushal M (kaushal@redhat.com)

--- Additional comment from Anand Avati on 2015-08-28 03:07:46 EDT ---

REVIEW: http://review.gluster.org/10851 (Multi-threaded SHD support) posted (#2) for review on master by Kaushal M (kaushal@redhat.com)
Comment 3 Mike McCune 2016-03-28 19:19:36 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 6 Pranith Kumar K 2016-05-04 03:30:44 EDT
Documentation for the new feature is tracked as a separate bug, so moving doc-text -
Comment 7 nchilaka 2016-06-03 06:15:51 EDT
1)set the msh cluster.shd-max-threads to say 5
now 5 files in a dir must be getting healed parallely.
If there are 8 files, 5 will get healed, then based on the wait queue the next set will be picked
If there are only 3 files then all 3 will be parallely healed
If there are 3 files in dir1 and 7 in dir2. then the heal is done parallely on 3 files in dir1 and then only dir2 is picked

Also some more cases are below:

All the above cases passed, hence moving to verified

Also, raised a bug#1335070 change the misleading multi threaded self heal option 

test version:3.7.9-6

Moving to verified
Comment 9 errata-xmlrpc 2016-06-23 01:10:40 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.