+++ This bug was initially created as a clone of Bug #1441508 +++

Description of problem:

Currently rebalance/remove-brick process does not honor min-free-disk. Hence, often making the disk full scenario.

As part of solution they should check if brick can accommodate file within the min-free-disk boundary. If not, then it should find a new brick which has good enough space (obviously min-free-disk limit checked) to migrate the file.

--- Additional comment from Worker Ant on 2017-04-12 11:57:41 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#5) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-12 12:13:47 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#6) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Worker Ant on 2017-04-12 14:07:32 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#7) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-12 16:24:16 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#8) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-12 17:10:50 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#9) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-12 19:23:51 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#10) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-12 22:13:06 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#11) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-13 10:33:14 MVT ---

REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#12) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-13 14:57:45 MVT ---

COMMIT: https://review.gluster.org/17034 committed in master by Raghavendra G (rgowdapp) 
------
commit c2ea4463ab4ce04936a824f7e8c7c133aabe1381
Author: Susant Palai <spalai>
Date:   Tue Apr 11 17:27:17 2017 +0530

    cluster/dht: Make rebalance honor min-free-disk
    
    test:  Manual
    
    created files of size 1K on 2 brick(of size 1GB) setup .
    added a brick of size 16GB.
    set min-free-disk to 12GB(so that first two bricks won't receive any files).
    removed one of the 1st brick of size 1GB.
    
    Logs from test:
    [2017-04-12 08:52:08.196484] W [MSGID: 0] [dht-rebalance.c:895:__dht_check_free_space]
     0-test1-dht: Write will cross min-free-disk for file - /tile32 on subvol - test1-client-1.
    Looking for new subvol.
    
    [2017-04-12 08:52:08.196904] I [MSGID: 0] [dht-rebalance.c:925:__dht_check_free_space]
    0-test1-dht: new target found - test1-client-2 for file - /tile32
    
     - Post migration we have two files. The new destination (/brick/1) has the data file
    [root@vm1 ~]# ll /brick/1/tile32
    -rw-r--r--. 2 root root 0 Apr 12 14:22 /brick/1/tile32
    
     - On the old target the linkto file is there with linkto xattr pointing to /brick/1
    [root@vm1 ~]# ll /tmp/2/tile32
    ---------T. 2 root root 1000 Apr 12 14:22 /tmp/2/tile32
    [root@vm1 ~]# getfattr -m . -de text /tmp/2/tile32
    getfattr: Removing leading '/' from absolute path names
    security.selinux="unconfined_u:object_r:user_tmp_t:s0"
    trusted.gfid="����:Aс�#�/'b2"
    trusted.glusterfs.dht.linkto="test1-client-2"
    
    Marking ./tests/features/worm_sh.t as bad test.
    Reason being, this patch failed on master branch as well and it has nothing
    to do with rebalance/remove-brick.
    
    BUG: 1441508
    Change-Id: I90bae251cda3d957a49cdceda90cd08311a392fb
    Signed-off-by: Susant Palai <spalai>
    Reviewed-on: https://review.gluster.org/17034
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Amar Tumballi <amarts>
    Reviewed-by: Raghavendra G <rgowdapp>
    CentOS-regression: Gluster Build System <jenkins.org>

--- Additional comment from Worker Ant on 2017-04-17 12:35:26 MVT ---

REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#1) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-17 12:41:54 MVT ---

REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#2) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-17 12:45:39 MVT ---

REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#3) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-17 14:52:21 MVT ---

REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#4) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-17 15:09:08 MVT ---

REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#5) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-18 12:27:50 MVT ---

REVIEW: https://review.gluster.org/17064 (cluster/dht: Skip file migration if the subvol that meets min-free-disk criteria happens to be the same subvol containing data-file) posted (#6) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-19 10:35:18 MVT ---

COMMIT: https://review.gluster.org/17064 committed in master by Raghavendra G (rgowdapp) 
------
commit b350bcd6a3db2e92d1baa47a5ec02efd09f76f16
Author: Susant Palai <spalai>
Date:   Mon Apr 17 13:00:54 2017 +0530

    cluster/dht: Skip file migration if the subvol that meets min-free-disk
    criteria happens to be the same subvol containing data-file
    
    Rebalance need to figure out a new subvol in case the hashed subvol
    does not have enough space. In the process of figuring out the new subvol,
    we need to ignore the source subvol, otherwise it will lead to data loss.
    
    Test: Manual
    Ran the following
    sizeof /tmp/1: 1.5GB
    sizeof /brick/1: 16GB
    sizeof /tmp/2: 1.5GB
    <start>
    
    glusterd;  gluster v create test1 vm1:/brick/1 vm1:/tmp/1;
    gluster v start test1;
    mount -t glusterfs vm1:test1 /mnt;
    for i in {1..2000}
    do
    dd if=/dev/zero of=/mnt/file$i bs=1KB count=1 &> /dev/null;
    done
    gluster v add-brick test1 vm1:/tmp/2
    gluster v set test1 min-free-disk 12GB
    gluster v remove-brick test1 vm1:/tmp/1 star
    <end>
    
    file count and data were intact.
    
    Change-Id: Ib8fc8467a3d48a7c12958824c4f0b88e160b86c1
    BUG: 1441508
    Signed-off-by: Susant Palai <spalai>
    Reviewed-on: https://review.gluster.org/17064
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>

--- Additional comment from Worker Ant on 2017-04-24 16:06:58 MVT ---

REVIEW: https://review.gluster.org/17104 (cluster/dht: correct space check for rebalance) posted (#1) for review on master by Susant Palai (spalai)

--- Additional comment from Worker Ant on 2017-04-25 21:28:22 MVT ---

COMMIT: https://review.gluster.org/17104 committed in master by Raghavendra G (rgowdapp) 
------
commit dba51428dc16762a9489249f6d9b2e64c0b8303e
Author: Susant Palai <spalai>
Date:   Mon Apr 24 16:27:23 2017 +0530

    cluster/dht: correct space check for rebalance
    
    With rebalance doing fallocate on destination, we don't need to
    add file size to the "destination available space" to decide whether
    to migrate the file or not.
    
    Notes: Fallocate would have already occupied the file size space on
    destination
    
    Change-Id: If7f6a6654e6257726680cf20d618482a6e9095a6
    BUG: 1441508
    Signed-off-by: Susant Palai <spalai>
    Reviewed-on: https://review.gluster.org/17104
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Amar Tumballi <amarts>
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Raghavendra G <rgowdapp>

--- Additional comment from Worker Ant on 2017-05-16 09:58:51 MVT ---

REVIEW: https://review.gluster.org/17305 (cluster/dht: Fix crash in dht rmdir) posted (#1) for review on master by N Balachandran (nbalacha)

--- Additional comment from Nithya Balachandran on 2017-05-16 10:05:26 MVT ---

(In reply to Worker Ant from comment #19)
> REVIEW: https://review.gluster.org/17305 (cluster/dht: Fix crash in dht
> rmdir) posted (#1) for review on master by N Balachandran
> (nbalacha)


Please ignore this - I had entered the wrong BZ for the patch.

--- Additional comment from Shyamsundar on 2017-05-30 23:49:52 MVT ---

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report.

glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html
[2] https://www.gluster.org/pipermail/gluster-users/