Bug 1260479 - DHT:While removing the brick, rebalance is trying to migrate files to the brick which doesn't have space due to this migration is failing
Summary: DHT:While removing the brick, rebalance is trying to migrate files to the bri...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: RHGS 3.4.0
Assignee: Nithya Balachandran
QA Contact: Prasad Desala
Whiteboard: dht-rebalance-usability, dht-brick-fu...
Depends On:
Blocks: 1503134
TreeView+ depends on / blocked
Reported: 2015-09-07 04:35 UTC by RajeshReddy
Modified: 2018-09-17 10:51 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.12.2-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2018-09-04 06:26:56 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 06:28:39 UTC

Description RajeshReddy 2015-09-07 04:35:15 UTC
Description of problem:
While removing the brick, Rebalance is trying to migrate files from decommissioned brick to the brick which doesn't have enough space and due to this migration is failing  

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1. Create distributed volume with three bricks such that one brick has has less space and another two has more (say 10 G, 200G & 200G)
2. Once the brick is full, link files will be created on brick which has less space and data file will be on brick which has more space
3. Now remove the brick which has data files and as par of this re-balance is trying to migrate the files to the brick which has less space and due to this migration is failing 

Actual results:
While migrating the files to other bricks it is good to select target brick based on available space 


[root@rhs-client9 ~]# gluster vol status dht10
Status of volume: dht10
Gluster process                             TCP Port  RDMA Port  Online  Pid
Brick rhs-client9.lab.eng.blr.redhat.com:/r
hs/brick1/dht10                             49181     0          Y       8840 
Brick rhs-client39.lab.eng.blr.redhat.com:/
rhs/brick10/dht10                           49173     0          Y       9645 
Brick rhs-client39.lab.eng.blr.redhat.com:/
rhs/brick1/dht10                            49174     0          Y       9663 
NFS Server on localhost                     2049      0          Y       32216
NFS Server on rhs-client39.lab.eng.blr.redh
at.com                                      2049      0          Y       18874
Task Status of Volume dht10
Task                 : Remove brick        
ID                   : 5908c287-47df-4662-8f56-a93ed7241f41
Removed bricks:     
Status               : completed

Comment 6 Prasad Desala 2018-05-28 09:53:53 UTC
Verified this BZ on glusterfs version 3.12.2-11.el7rhgs.x86_64.

Followed the same steps as in the description. With cluster.min-free-disk, when started remove-brick operation rebalance will migrate the files from the decommissioned bricks to the bricks which has enough space. 

If file picked is attempted to migrate to a brick which doesn't have enough space, a warning will be thrown in logs and it will again look for a new subvol which has enough space to migrate.

[2018-05-28 09:46:00.113661] I [dht-rebalance.c:1516:dht_migrate_file] 0-rr-dht: /file_3709: attempting to move from rr-client-2 to rr-client-0
[2018-05-28 09:46:00.128659] W [MSGID: 0] [dht-rebalance.c:985:__dht_check_free_space] 0-rr-dht: Write will cross min-free-disk for file - /file_3709 on subvol - rr-client-0. Looking for new subvol
[2018-05-28 09:46:00.128744] I [MSGID: 0] [dht-rebalance.c:1034:__dht_check_free_space] 0-rr-dht: new target found - rr-client-1 for file - /file_3709
[2018-05-28 09:46:00.139228] I [MSGID: 0] [dht-rebalance.c:1715:dht_migrate_file] 0-rr-dht: destination for file - /file_3709 is changed to - rr-client-1
[2018-05-28 09:46:02.970999] I [MSGID: 109022] [dht-rebalance.c:2205:dht_migrate_file] 0-rr-dht: completed migration of /file_3709 from subvolume rr-client-2 to rr-client-1

Comment 8 errata-xmlrpc 2018-09-04 06:26:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.