Red Hat Bugzilla – Bug 1260479
DHT:While removing the brick, rebalance is trying to migrate files to the brick which doesn't have space due to this migration is failing
Last modified: 2018-09-17 06:51:06 EDT
Description of problem: ========================= While removing the brick, Rebalance is trying to migrate files from decommissioned brick to the brick which doesn't have enough space and due to this migration is failing Version-Release number of selected component (if applicable): ========================== glusterfs-server-3.7.1-13 Steps to Reproduce: ===================== 1. Create distributed volume with three bricks such that one brick has has less space and another two has more (say 10 G, 200G & 200G) 2. Once the brick is full, link files will be created on brick which has less space and data file will be on brick which has more space 3. Now remove the brick which has data files and as par of this re-balance is trying to migrate the files to the brick which has less space and due to this migration is failing Actual results: ================ While migrating the files to other bricks it is good to select target brick based on available space Notes: ============ [root@rhs-client9 ~]# gluster vol status dht10 Status of volume: dht10 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client9.lab.eng.blr.redhat.com:/r hs/brick1/dht10 49181 0 Y 8840 Brick rhs-client39.lab.eng.blr.redhat.com:/ rhs/brick10/dht10 49173 0 Y 9645 Brick rhs-client39.lab.eng.blr.redhat.com:/ rhs/brick1/dht10 49174 0 Y 9663 NFS Server on localhost 2049 0 Y 32216 NFS Server on rhs-client39.lab.eng.blr.redh at.com 2049 0 Y 18874 Task Status of Volume dht10 ------------------------------------------------------------------------------ Task : Remove brick ID : 5908c287-47df-4662-8f56-a93ed7241f41 Removed bricks: rhs-client9.lab.eng.blr.redhat.com:/rhs/brick1/dht10 Status : completed
Verified this BZ on glusterfs version 3.12.2-11.el7rhgs.x86_64. Followed the same steps as in the description. With cluster.min-free-disk, when started remove-brick operation rebalance will migrate the files from the decommissioned bricks to the bricks which has enough space. If file picked is attempted to migrate to a brick which doesn't have enough space, a warning will be thrown in logs and it will again look for a new subvol which has enough space to migrate. [2018-05-28 09:46:00.113661] I [dht-rebalance.c:1516:dht_migrate_file] 0-rr-dht: /file_3709: attempting to move from rr-client-2 to rr-client-0 [2018-05-28 09:46:00.128659] W [MSGID: 0] [dht-rebalance.c:985:__dht_check_free_space] 0-rr-dht: Write will cross min-free-disk for file - /file_3709 on subvol - rr-client-0. Looking for new subvol [2018-05-28 09:46:00.128744] I [MSGID: 0] [dht-rebalance.c:1034:__dht_check_free_space] 0-rr-dht: new target found - rr-client-1 for file - /file_3709 [2018-05-28 09:46:00.139228] I [MSGID: 0] [dht-rebalance.c:1715:dht_migrate_file] 0-rr-dht: destination for file - /file_3709 is changed to - rr-client-1 [2018-05-28 09:46:02.970999] I [MSGID: 109022] [dht-rebalance.c:2205:dht_migrate_file] 0-rr-dht: completed migration of /file_3709 from subvolume rr-client-2 to rr-client-1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607