Bug 1473136 - rebalance: Allow admin to change thread count for rebalance
rebalance: Allow admin to change thread count for rebalance
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
3.10
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Susant Kumar Palai
:
Depends On: 1438370
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-20 02:05 EDT by Susant Kumar Palai
Modified: 2017-08-21 09:41 EDT (History)
1 user (show)

See Also:
Fixed In Version: glusterfs-3.10.5
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1438370
Environment:
Last Closed: 2017-08-21 09:41:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Susant Kumar Palai 2017-07-20 02:05:07 EDT
+++ This bug was initially created as a clone of Bug #1438370 +++

Description of problem:

Current rebalance throttle options: lazy/normal/aggressive may not always be sufficient for the purpose of throttling.  In our recent test, we observed for certain setups, normal and aggressive modes behaved similarly consuming full disk bandwidth. So in cases like this admin can tune it down(or vice versa) depending on the need.

--- Additional comment from Worker Ant on 2017-04-03 14:19:52 MVT ---

REVIEW: https://review.gluster.org/16980 (cluster/dht: Make rebalance throttle option tuned by number) posted (#2) for review on master by Susant Palai (spalai@redhat.com)

--- Additional comment from Worker Ant on 2017-04-27 14:28:38 MVT ---

REVIEW: https://review.gluster.org/16980 (cluster/dht: Make rebalance throttle option tuned by number) posted (#3) for review on master by Susant Palai (spalai@redhat.com)

--- Additional comment from Worker Ant on 2017-04-27 15:38:47 MVT ---

REVIEW: https://review.gluster.org/16980 (cluster/dht: Make rebalance throttle option tuned by number) posted (#4) for review on master by Susant Palai (spalai@redhat.com)

--- Additional comment from Worker Ant on 2017-04-27 15:43:03 MVT ---

REVIEW: https://review.gluster.org/16980 (cluster/dht: Make rebalance throttle option tuned by number) posted (#5) for review on master by Susant Palai (spalai@redhat.com)

--- Additional comment from Worker Ant on 2017-04-27 19:06:23 MVT ---

REVIEW: https://review.gluster.org/16980 (cluster/dht: Make rebalance throttle option tuned by number) posted (#6) for review on master by Susant Palai (spalai@redhat.com)

--- Additional comment from Worker Ant on 2017-04-29 19:29:38 MVT ---

COMMIT: https://review.gluster.org/16980 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit d51288540241d1f7785bb17bdc0702c0879087a9
Author: Susant Palai <spalai@redhat.com>
Date:   Wed Mar 22 17:14:25 2017 +0530

    cluster/dht: Make rebalance throttle option tuned by number
    
    Current rebalance throttle options: lazy/normal/aggressive may not always be
    sufficient for the purpose of throttling.  In our recent test, we observed for
    certain setups, normal and aggressive modes behaved similarly consuming full
    disk bandwidth. So in cases like this admin should be able to  tune it
    down(or vice versa) depending on the need.
    
    Along with old throttle configurations, thread counts are tuned based on number.
    e.g. gluster v set vol-name cluster-rebal.throttle  5.
    
    Admin can tune up/down between 0 and the number of cores available.
    
    Note: For heterogenous servers, validation will fail on the old server if "number"
    is given for throttle configuration.
    The message looks something like this:
    "volume set: failed: Staging failed on vm2. Error: cluster.rebal-throttle should be {lazy|normal|aggressive}"
    
    Test: Manual test by logging active thread number after reconfiguring throttle option.
    testcase: tests/basic/distribute/throttle-rebal.t
    
    Change-Id: I46e3cde546900307831028b344ecf601fd9b02c3
    BUG: 1438370
    Signed-off-by: Susant Palai <spalai@redhat.com>
    Reviewed-on: https://review.gluster.org/16980
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>

--- Additional comment from Shyamsundar on 2017-05-30 23:48:52 MVT ---

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report.

glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html
[2] https://www.gluster.org/pipermail/gluster-users/
Comment 1 Worker Ant 2017-07-20 02:40:33 EDT
REVIEW: https://review.gluster.org/17835 (cluster/dht: Make rebalance throttle option tuned by number) posted (#1) for review on release-3.10 by Susant Palai (spalai@redhat.com)
Comment 2 Worker Ant 2017-07-20 06:09:53 EDT
REVIEW: https://review.gluster.org/17835 (cluster/dht: Make rebalance throttle option tuned by number) posted (#2) for review on release-3.10 by Susant Palai (spalai@redhat.com)
Comment 3 Worker Ant 2017-08-11 13:41:26 EDT
REVIEW: https://review.gluster.org/17835 (cluster/dht: Make rebalance throttle option tuned by number) posted (#3) for review on release-3.10 by Shyamsundar Ranganathan (srangana@redhat.com)
Comment 4 Worker Ant 2017-08-11 14:06:44 EDT
COMMIT: https://review.gluster.org/17835 committed in release-3.10 by Shyamsundar Ranganathan (srangana@redhat.com) 
------
commit c7101426c64db6723752c55f116dcbfbaf84a668
Author: Susant Palai <spalai@redhat.com>
Date:   Wed Mar 22 17:14:25 2017 +0530

    cluster/dht: Make rebalance throttle option tuned by number
    
    Current rebalance throttle options: lazy/normal/aggressive may not always be
    sufficient for the purpose of throttling.  In our recent test, we observed for
    certain setups, normal and aggressive modes behaved similarly consuming full
    disk bandwidth. So in cases like this admin should be able to  tune it
    down(or vice versa) depending on the need.
    
    Along with old throttle configurations, thread counts are tuned based on number.
    e.g. gluster v set vol-name cluster-rebal.throttle  5.
    
    Admin can tune up/down between 0 and the number of cores available.
    
    Note: For heterogenous servers, validation will fail on the old server if "number"
    is given for throttle configuration.
    The message looks something like this:
    "volume set: failed: Staging failed on vm2. Error: cluster.rebal-throttle should be {lazy|normal|aggressive}"
    
    Test: Manual test by logging active thread number after reconfiguring throttle option.
    testcase: tests/basic/distribute/throttle-rebal.t
    
    > Change-Id: I46e3cde546900307831028b344ecf601fd9b02c3
    > BUG: 1438370
    > Signed-off-by: Susant Palai <spalai@redhat.com>
    > Reviewed-on: https://review.gluster.org/16980
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > Smoke: Gluster Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    > Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
    > Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    > Signed-off-by: Susant Palai <spalai@redhat.com>
    
    Change-Id: I46e3cde546900307831028b344ecf601fd9b02c3
    BUG: 1473136
    Signed-off-by: Susant Palai <spalai@redhat.com>
    Reviewed-on: https://review.gluster.org/17835
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Comment 5 Shyamsundar 2017-08-21 09:41:22 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.5, please open a new bug report.

glusterfs-3.10.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-August/000079.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.