Bug 1366494
Summary: | Rebalance is not considering the brick sizes while fixing the layout | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Nithya Balachandran <nbalacha> | |
Component: | distribute | Assignee: | Nithya Balachandran <nbalacha> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
Severity: | unspecified | Docs Contact: | ||
Priority: | unspecified | |||
Version: | mainline | CC: | bugs, jdarcy, mzywusko, nbalacha, rcyriac, rgowdapp, rhinduja, smohan, spalai, storage-qa-internal | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.10.0 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | 1257182 | |||
: | 1374135 (view as bug list) | Environment: | ||
Last Closed: | 2017-03-06 17:22:10 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1257182 | |||
Bug Blocks: | 1374135 |
Description
Nithya Balachandran
2016-08-12 06:41:33 UTC
RCA: The volume was created with a single brick. On adding a second much larger brick and running a rebalance, the layout is recalculated for all existing directories by calling dht_fix_layout_of_directory (). This function generates a new weighted layout in dht_selfheal_layout_new_directory () but then calls dht_selfheal_layout_maximize_overlap () on the newly generated layout. This function does not consider the relative brick sizes and as the original brick had a complete layout (0x00000000-0xffffffff), the layout is swapped to maximize the overlap with the old layout. Nithya's analysis is correct. We generate a new layout based on brick sizes, then attempt to optimize it for maximum overlap with the current layout. That optimization is important to minimize data movement, but unfortunately it's broken in this case because it doesn't account properly for where each range already resides. I wrote that function BTW, so it's my fault. For now, we should probably just disable the optimization phase when we're weighting by brick size. Longer term, what we need to do is fix dht_selfheal_layout_maximize_overlap. There's a place where it tries to determine whether a particular swap would be an improvement or not. That particular calculation needs to be enhance to account for the *actual* current and proposed locations for a range, instead of (effectively) inferring those locations from ordinal positions. REVIEW: http://review.gluster.org/15403 (cluster/dht: Skip layout overlap maximization on weighted rebalance) posted (#1) for review on master by N Balachandran (nbalacha) REVIEW: http://review.gluster.org/15403 (cluster/dht: Skip layout overlap maximization on weighted rebalance) posted (#2) for review on master by N Balachandran (nbalacha) REVIEW: http://review.gluster.org/15403 (cluster/dht: Skip layout overlap maximization on weighted rebalance) posted (#3) for review on master by N Balachandran (nbalacha) REVIEW: http://review.gluster.org/15403 (cluster/dht: Skip layout overlap maximization on weighted rebalance) posted (#4) for review on master by N Balachandran (nbalacha) COMMIT: http://review.gluster.org/15403 committed in master by Jeff Darcy (jdarcy) ------ commit 2209f6d49bc8d16beed52b1d145be936d2da941a Author: N Balachandran <nbalacha> Date: Tue Sep 6 11:13:20 2016 +0530 cluster/dht: Skip layout overlap maximization on weighted rebalance dht_selfheal_layout_maximize_overlap () does not consider chunk sizes while calculating overlaps. Temporarily enabling this operation if only if weighted rebalance is disabled or all bricks are the same size. Change-Id: I5ed16cdff2551b826a1759ca8338921640bfc7b3 BUG: 1366494 Signed-off-by: N Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/15403 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp> NetBSD-regression: NetBSD Build System <jenkins.org> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report. glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html [2] https://www.gluster.org/pipermail/gluster-users/ |