Bug 1260479

Summary:	DHT:While removing the brick, rebalance is trying to migrate files to the brick which doesn't have space due to this migration is failing
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	RajeshReddy <rmekala>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED ERRATA	QA Contact:	Prasad Desala <tdesala>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	rhgs-3.1	CC:	amukherj, nchilaka, rgowdapp, rhinduja, rhs-bugs, sheggodu, spalai
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	dht-rebalance-usability, dht-brick-full, dht-retest, rebase
Fixed In Version:	glusterfs-3.12.2-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-09-04 06:26:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1503134

Description RajeshReddy 2015-09-07 04:35:15 UTC

Description of problem:
=========================
While removing the brick, Rebalance is trying to migrate files from decommissioned brick to the brick which doesn't have enough space and due to this migration is failing  

Version-Release number of selected component (if applicable):
==========================
glusterfs-server-3.7.1-13


Steps to Reproduce:
=====================
1. Create distributed volume with three bricks such that one brick has has less space and another two has more (say 10 G, 200G & 200G)
2. Once the brick is full, link files will be created on brick which has less space and data file will be on brick which has more space
3. Now remove the brick which has data files and as par of this re-balance is trying to migrate the files to the brick which has less space and due to this migration is failing 

Actual results:
================
While migrating the files to other bricks it is good to select target brick based on available space 

Notes:
============

[root@rhs-client9 ~]# gluster vol status dht10
Status of volume: dht10
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-client9.lab.eng.blr.redhat.com:/r
hs/brick1/dht10                             49181     0          Y       8840 
Brick rhs-client39.lab.eng.blr.redhat.com:/
rhs/brick10/dht10                           49173     0          Y       9645 
Brick rhs-client39.lab.eng.blr.redhat.com:/
rhs/brick1/dht10                            49174     0          Y       9663 
NFS Server on localhost                     2049      0          Y       32216
NFS Server on rhs-client39.lab.eng.blr.redh
at.com                                      2049      0          Y       18874
 
Task Status of Volume dht10
------------------------------------------------------------------------------
Task                 : Remove brick        
ID                   : 5908c287-47df-4662-8f56-a93ed7241f41
Removed bricks:     
rhs-client9.lab.eng.blr.redhat.com:/rhs/brick1/dht10
Status               : completed

Comment 6 Prasad Desala 2018-05-28 09:53:53 UTC

Verified this BZ on glusterfs version 3.12.2-11.el7rhgs.x86_64.

Followed the same steps as in the description. With cluster.min-free-disk, when started remove-brick operation rebalance will migrate the files from the decommissioned bricks to the bricks which has enough space. 

If file picked is attempted to migrate to a brick which doesn't have enough space, a warning will be thrown in logs and it will again look for a new subvol which has enough space to migrate.

[2018-05-28 09:46:00.113661] I [dht-rebalance.c:1516:dht_migrate_file] 0-rr-dht: /file_3709: attempting to move from rr-client-2 to rr-client-0
[2018-05-28 09:46:00.128659] W [MSGID: 0] [dht-rebalance.c:985:__dht_check_free_space] 0-rr-dht: Write will cross min-free-disk for file - /file_3709 on subvol - rr-client-0. Looking for new subvol
[2018-05-28 09:46:00.128744] I [MSGID: 0] [dht-rebalance.c:1034:__dht_check_free_space] 0-rr-dht: new target found - rr-client-1 for file - /file_3709
[2018-05-28 09:46:00.139228] I [MSGID: 0] [dht-rebalance.c:1715:dht_migrate_file] 0-rr-dht: destination for file - /file_3709 is changed to - rr-client-1
[2018-05-28 09:46:02.970999] I [MSGID: 109022] [dht-rebalance.c:2205:dht_migrate_file] 0-rr-dht: completed migration of /file_3709 from subvolume rr-client-2 to rr-client-1

Comment 8 errata-xmlrpc 2018-09-04 06:26:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607