Description of problem: Currently rebalance/remove-brick process does not honor min-free-disk. Hence, often making the disk full scenario. As part of solution they should check if brick can accommodate file within the min-free-disk boundary. If not, then it should find a new brick which has good enough space (obviously min-free-disk limit checked) to migrate the file.
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#5) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#6) for review on master by Atin Mukherjee (amukherj)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#7) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#8) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#9) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#10) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#11) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor min-free-disk) posted (#12) for review on master by Susant Palai (spalai)
COMMIT: https://review.gluster.org/17034 committed in master by Raghavendra G (rgowdapp) ------ commit c2ea4463ab4ce04936a824f7e8c7c133aabe1381 Author: Susant Palai <spalai> Date: Tue Apr 11 17:27:17 2017 +0530 cluster/dht: Make rebalance honor min-free-disk test: Manual created files of size 1K on 2 brick(of size 1GB) setup . added a brick of size 16GB. set min-free-disk to 12GB(so that first two bricks won't receive any files). removed one of the 1st brick of size 1GB. Logs from test: [2017-04-12 08:52:08.196484] W [MSGID: 0] [dht-rebalance.c:895:__dht_check_free_space] 0-test1-dht: Write will cross min-free-disk for file - /tile32 on subvol - test1-client-1. Looking for new subvol. [2017-04-12 08:52:08.196904] I [MSGID: 0] [dht-rebalance.c:925:__dht_check_free_space] 0-test1-dht: new target found - test1-client-2 for file - /tile32 - Post migration we have two files. The new destination (/brick/1) has the data file [root@vm1 ~]# ll /brick/1/tile32 -rw-r--r--. 2 root root 0 Apr 12 14:22 /brick/1/tile32 - On the old target the linkto file is there with linkto xattr pointing to /brick/1 [root@vm1 ~]# ll /tmp/2/tile32 ---------T. 2 root root 1000 Apr 12 14:22 /tmp/2/tile32 [root@vm1 ~]# getfattr -m . -de text /tmp/2/tile32 getfattr: Removing leading '/' from absolute path names security.selinux="unconfined_u:object_r:user_tmp_t:s0" trusted.gfid="����:Aс�#�/'b2" trusted.glusterfs.dht.linkto="test1-client-2" Marking ./tests/features/worm_sh.t as bad test. Reason being, this patch failed on master branch as well and it has nothing to do with rebalance/remove-brick. BUG: 1441508 Change-Id: I90bae251cda3d957a49cdceda90cd08311a392fb Signed-off-by: Susant Palai <spalai> Reviewed-on: https://review.gluster.org/17034 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Amar Tumballi <amarts> Reviewed-by: Raghavendra G <rgowdapp> CentOS-regression: Gluster Build System <jenkins.org>
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#1) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#2) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#3) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#4) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source for min-free-disk check) posted (#5) for review on master by Susant Palai (spalai)
REVIEW: https://review.gluster.org/17064 (cluster/dht: Skip file migration if the subvol that meets min-free-disk criteria happens to be the same subvol containing data-file) posted (#6) for review on master by Susant Palai (spalai)
COMMIT: https://review.gluster.org/17064 committed in master by Raghavendra G (rgowdapp) ------ commit b350bcd6a3db2e92d1baa47a5ec02efd09f76f16 Author: Susant Palai <spalai> Date: Mon Apr 17 13:00:54 2017 +0530 cluster/dht: Skip file migration if the subvol that meets min-free-disk criteria happens to be the same subvol containing data-file Rebalance need to figure out a new subvol in case the hashed subvol does not have enough space. In the process of figuring out the new subvol, we need to ignore the source subvol, otherwise it will lead to data loss. Test: Manual Ran the following sizeof /tmp/1: 1.5GB sizeof /brick/1: 16GB sizeof /tmp/2: 1.5GB <start> glusterd; gluster v create test1 vm1:/brick/1 vm1:/tmp/1; gluster v start test1; mount -t glusterfs vm1:test1 /mnt; for i in {1..2000} do dd if=/dev/zero of=/mnt/file$i bs=1KB count=1 &> /dev/null; done gluster v add-brick test1 vm1:/tmp/2 gluster v set test1 min-free-disk 12GB gluster v remove-brick test1 vm1:/tmp/1 star <end> file count and data were intact. Change-Id: Ib8fc8467a3d48a7c12958824c4f0b88e160b86c1 BUG: 1441508 Signed-off-by: Susant Palai <spalai> Reviewed-on: https://review.gluster.org/17064 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: https://review.gluster.org/17104 (cluster/dht: correct space check for rebalance) posted (#1) for review on master by Susant Palai (spalai)
COMMIT: https://review.gluster.org/17104 committed in master by Raghavendra G (rgowdapp) ------ commit dba51428dc16762a9489249f6d9b2e64c0b8303e Author: Susant Palai <spalai> Date: Mon Apr 24 16:27:23 2017 +0530 cluster/dht: correct space check for rebalance With rebalance doing fallocate on destination, we don't need to add file size to the "destination available space" to decide whether to migrate the file or not. Notes: Fallocate would have already occupied the file size space on destination Change-Id: If7f6a6654e6257726680cf20d618482a6e9095a6 BUG: 1441508 Signed-off-by: Susant Palai <spalai> Reviewed-on: https://review.gluster.org/17104 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Amar Tumballi <amarts> Reviewed-by: N Balachandran <nbalacha> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: https://review.gluster.org/17305 (cluster/dht: Fix crash in dht rmdir) posted (#1) for review on master by N Balachandran (nbalacha)
(In reply to Worker Ant from comment #19) > REVIEW: https://review.gluster.org/17305 (cluster/dht: Fix crash in dht > rmdir) posted (#1) for review on master by N Balachandran > (nbalacha) Please ignore this - I had entered the wrong BZ for the patch.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report. glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html [2] https://www.gluster.org/pipermail/gluster-users/