Bug 1167789

Summary: DHT: Rebalance- Misleading log messages from __dht_check_free_space function
Product: Red Hat Gluster Storage Reporter: shylesh <shmohan>
Component: distributeAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED ERRATA QA Contact: Prasad Desala <tdesala>
Severity: medium Docs Contact:
Priority: low    
Version: rhgs-3.0CC: amukherj, kramdoss, nbalacha, nchilaka, rgowdapp, rhinduja, rhs-bugs, sheggodu, smohan
Target Milestone: ---Keywords: Reopened, ZStream
Target Release: RHGS 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: triaged, dht-rebalance-file, dht-log, rebase
Fixed In Version: glusterfs-3.12.2-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1341948 (view as bug list) Environment:
Last Closed: 2018-09-04 06:26:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1341948, 1503134    

Description shylesh 2014-11-25 12:33:13 UTC
Description of problem:
misleading message from logs about available disk space

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. create a volume and data 
2. add one more brick , make sure that there is some space discrepancy 
3. start the rebalance without force option
gluster volume rebalance <vol> start 
4. check the status of rebalance and make sure that some skipped files are there

Actual results:
2014-11-25 09:10:32.433141] I [dht-rebalance.c:902:dht_migrate_file] 0-gs-dht: /file9: attempting to move from gs-client-0 to gs-client-4
[2014-11-25 09:10:32.437164] W [MSGID: 109023] [dht-rebalance.c:568:__dht_check_free_space] 0-gs-dht: data movement attempted from node (gs-client-0:209540896) with higher disk space to a node (gs-client-4:209540960) with lesser disk space, file { blocks:2048, name:(/file9) }

[2014-11-25 09:10:32.438441] I [dht-common.c:1563:dht_lookup_everywhere_cbk] 0-gs-dht: attempting deletion of stale linkfile /file10 on gs-client-4 (hashed subvol is gs-client-6)

Here available space on source is gs-client-0:209540896 and on destination is gs-client-4:209540960 but log says source has higher space and destination has lesser which is misleading .

since calculation of free space is done in the following way

if ((dst_statfs_blocks - stbuf->ia_blocks) <
    (src_statfs_blocks + stbuf->ia_blocks)) 

the proper values has to be printed in the logs

Comment 2 Sakshi 2015-04-22 13:03:44 UTC
What is the size of the brick? Also what do you mean by space discrepancy? Should the brick added later be lesser in size compared to the existing bricks?

Comment 3 ankit 2016-11-02 05:42:35 UTC

*** This bug has been marked as a duplicate of bug 1341948 ***

Comment 4 Nithya Balachandran 2016-11-02 06:28:41 UTC
This is a downstream BZ whereas 1341948 is an upstream one. They cannot be marked Duplicates.


Comment 10 Prasad Desala 2018-02-20 07:22:27 UTC
Verified this BZ on glusterfs version: 3.12.2-4.el7rhgs.x86_64.

Now, we are seeing correct log message when the file is skipped.

[2018-02-20 05:35:11.063121] W [MSGID: 109023] [dht-rebalance.c:962:__dht_check_free_space] 0-distrepx3-dht: data movement of file {blocks:1 name:(/1/linux-4.6.4/Documentation/devicetree/bindings/display/panel/innolux,zj070na-01p.txt) } would result in dst node (distrepx3-replicate-1:41275176) having lower disk space then the source node (distrepx3-replicate-0:41449024).Skipping file.
[2018-02-20 05:35:11.400035] W [MSGID: 109023] [dht-rebalance.c:962:__dht_check_free_space] 0-distrepx3-dht: data movement of file {blocks:1 name:(/1/linux-4.6.4/Documentation/devicetree/bindings/display/panel/lg,lb070wv8.txt) } would result in dst node (distrepx3-replicate-3:41444376) having lower disk space then the source node (distrepx3-replicate-2:41445288).Skipping file.
[2018-02-20 05:35:12.421654] W [MSGID: 109023] [dht-rebalance.c:962:__dht_check_free_space] 0-distrepx3-dht: data movement of file {blocks:6 name:(/1/linux-4.6.4/Documentation/devicetree/bindings/display/ti/ti,omap4-dss.txt) } would result in dst node (distrepx3-replicate-1:41275184) having lower disk space then the source node (distrepx3-replicate-0:41449032).Skipping file.

[2018-02-20 05:35:14.544070] W [MSGID: 109023] [dht-rebalance.c:962:__dht_check_free_space] 0-distrepx3-dht: data movement of file {blocks:3 name:(/1/linux-4.6.4/Documentation/devicetree/bindings/dma/sun4i-dma.txt) } would result in dst node (distrepx3-replicate-1:41275384) having lower disk space then the source node (distrepx3-replicate-0:41449112).Skipping file.

Here available space on source is distrepx3-replicate-0:41449112 and on destination is distrepx3-replicate-1:41275384, which is correct and expected to skip the file migration.

Hence, moving this BZ to Verified.

Comment 12 errata-xmlrpc 2018-09-04 06:26:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.