Description of problem: New files being written to a distributed volumes are not taking in to account the cluster.min-free-disk setting. New files are being written to bricks with free space below this setting value despite adding additional empty bricks. Version-Release number of selected component (if applicable): glusterfs 3.3.1, running on Ubuntu 12.04 x64. Steps to Reproduce: I'm currently testing Gluster on 5x HyperV machines; 4x Gluster servers and 1x Gluster client. Each Gluster server has a 1GB XFS formatted volume which is mounted as /export/brick1. The IP address assignment is as follows: server1: 172.16.2.203 server2: 172.16.2.204 server3: 172.16.2.206 server4: 172.16.2.207 client1: 172.16.2.205 I do a "peer probe" from Gluster server 1 to Gluster server 2 to form a cluster. I then create a volume using the following command: volume create gv0 172.16.2.203:/export/brick1 172.16.2.204:/export/brick1 I then issue: volume set gv0 cluster.min-free-disk 100MB The volume is then started: volume start gv0 I mount the volume on the client. In fstab I have the following: 172.16.2.203:/gv0 /mnt glusterfs defaults,_netdev 0 0 The client is running VSFTPD and I connect with my FTP client (FileZilla). I then CD to /mnt and create folders numbered 1 to 10, and upload some random ~60MB files across the structure. After a while, "volume status gv0 detail" shows this: Status of volume: gv0 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.203:/export/brick1 Port : 24009 Online : Y Pid : 8813 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 48.7MB Total Disk Space : 1014.0MB Inode Count : 199696 Free Inodes : 199466 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.204:/export/brick1 Port : 24009 Online : Y Pid : 7956 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 24.0MB Total Disk Space : 1014.0MB Inode Count : 98560 Free Inodes : 98348 ------------------------------------------------------------------------------ If I attempt to upload another ~60MB file it errors due to running out of disk space. "volume status gv0 detail" now looks like this: Status of volume: gv0 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.203:/export/brick1 Port : 24009 Online : Y Pid : 8813 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 20.0KB Total Disk Space : 1014.0MB Inode Count : 400 Free Inodes : 167 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.204:/export/brick1 Port : 24009 Online : Y Pid : 7956 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 24.0MB Total Disk Space : 1014.0MB Inode Count : 98560 Free Inodes : 98348 ------------------------------------------------------------------------------ This is what I expect. However when I add another brick (hosted on server 3) with the following commands: peer probe 172.16.2.206 volume add-brick gv0 172.16.2.206:/export/brick1 and then: volume rebalance fix-layout start I repeat the same upload (same file to the same path) and I get the same out of space error, and "volume status gv0 detail" looks like this: Status of volume: gv0 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.203:/export/brick1 Port : 24009 Online : Y Pid : 8813 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 20.0KB Total Disk Space : 1014.0MB Inode Count : 400 Free Inodes : 167 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.204:/export/brick1 Port : 24009 Online : Y Pid : 7956 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 24.0MB Total Disk Space : 1014.0MB Inode Count : 98560 Free Inodes : 98348 ------------------------------------------------------------------------------ Brick : Brick 172.16.2.206:/export/brick1 Port : 24013 Online : Y Pid : 7846 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 256 Disk Space Free : 981.8MB Total Disk Space : 1014.0MB Inode Count : 1048576 Free Inodes : 1048517 I would have expected the new file to go on to 172.16.2.206, given that cluster.min-free-disk is set to 100MB. However it's writing to 172.16.2.203. I've tried restarting the volume and remounting the volume on the client but the file wants to write to 172.16.2.203.
Hi Pete, With 3.3.0 branch, I have not been able to reproduce the issue at hand. Could you please upload the client logs?
Hi pete, Can you confirm, whether the issue is still being seen?
Pete, also see how does 3.4.0qa releases work out for you. (qa6 is the latest for now).
Where can I download 3.4.0qa for Ubuntu 12.04?
The qa repos(centos) are available @ http://bits.gluster.org/pub/gluster/glusterfs/ Alternatively, clone source from https://github.com/gluster/glusterfs.git and checkout out required branch or use https://github.com/gluster/glusterfs/tree/v3.4.0qa6 link to download zip files directly.
http://review.gluster.org/4420 was last of the patches to get the feature working.
REVIEW: http://review.gluster.org/4839 (cluster/dht: Correct min_free_disk behaviour) posted (#1) for review on release-3.3 by Varun Shastry (vshastry)
COMMIT: http://review.gluster.org/4839 committed in release-3.3 by Vijay Bellur (vbellur) ------ commit 0ab16bb29a2e242714a76a3bad31921142c7dd35 Author: Varun Shastry <vshastry> Date: Tue Apr 16 12:11:17 2013 +0530 cluster/dht: Correct min_free_disk behaviour Problem: Files were being created in subvol which had less than min_free_disk available even in the cases where other subvols with more space were available. Solution: Changed the logic to look for subvol which has more space available. In cases where all the subvols have lesser than Min_free_disk available , the one with max space and atleast one inode is available. Known Issue: Cannot ensure that first file that is created right after min-free-value is crossed on a brick will get created in other brick because disk usage stat takes some time to update in glusterprocess. Will fix that as part of another bug. Change-Id: Icaba552db053ad8b00be0914b1f4853fb7661bd3 BUG: 874554 Signed-off-by: Raghavendra Talur <rtalur> Signed-off-by: Varun Shastry <vshastry> Reviewed-on: http://review.gluster.org/4839 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
Hi there, I've finally had the chance to re-test and the behaviour is still the same. I'm using glusterfs 3.4.0qa8 on Centos 6.4. Again I started out with 2x1GB bricks and loaded them up with files. The volume status is the following: gluster> volume status gv0 detail Status of volume: gv0 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.6:/export/brick1 Port : 49153 Online : Y Pid : 27502 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 16014MB Total Disk Space : 1014.0MB Inode Count : 328296 Free Inodes : 328042 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.11:/export/brick1 Port : 49153 Online : Y Pid : 2699 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 9.6MB Total Disk Space : 1014.0MB Inode Count : 20136 Free Inodes : 19880 If I attempt to write a ~30MB file it fails as it looks like it's going to 10.100.2.11. The volume status confirms that: gluster> volume status gv0 detail Status of volume: gv0 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.6:/export/brick1 Port : 49153 Online : Y Pid : 27502 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 160.1MB Total Disk Space : 1014.0MB Inode Count : 328296 Free Inodes : 328042 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.11:/export/brick1 Port : 49153 Online : Y Pid : 2699 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 20.0KB Total Disk Space : 1014.0MB Inode Count : 424 Free Inodes : 165 From what I understand from the fix made, if I add another brick the file should go to that new brick as it has sufficient free space. Or, it could be either the new brick or 10.100.2.6 as it has sufficient space and is below the free space threshold. So I add a new brick: gluster> volume add-brick gv0 10.100.2.14:/export/brick1 volume add-brick: success And a rebalance: gluster> volume rebalance gv0 And then try to write the file again - which fails. The volume status shows: gluster> volume status gv0 detail Status of volume: gv0 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.6:/export/brick1 Port : 49153 Online : Y Pid : 27502 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 114.3MB Total Disk Space : 1014.0MB Inode Count : 234376 Free Inodes : 234120 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.11:/export/brick1 Port : 49153 Online : Y Pid : 2699 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 24.0KB Total Disk Space : 1014.0MB Inode Count : 432 Free Inodes : 173 ------------------------------------------------------------------------------ Brick : Brick 10.100.2.14:/export/brick1 Port : 49153 Online : Y Pid : 2748 File System : xfs Device : /dev/sdb Mount Options : rw Inode Size : 512 Disk Space Free : 981.7MB Total Disk Space : 1014.0MB Inode Count : 524288 Free Inodes : 524248
Hi Pete, Please provide us the log for this volume. Also, what was the min-free-disk value set to?
REVIEW: http://review.gluster.org/7687 (tests: min-free-disk test was incomplete) posted (#3) for review on master by Harshavardhana (harsha)
COMMIT: http://review.gluster.org/7687 committed in master by Vijay Bellur (vbellur) ------ commit 47c33dd27150039a6e5e3295eacd8d2d5a7e0ce0 Author: Harshavardhana <harsha> Date: Tue May 6 14:38:40 2014 -0700 tests: min-free-disk test was incomplete - use '%' when we mean it for clarity - in bash we need to evaluate counter decrements Change-Id: Ibd17126945e8a335fa2671d658a2e0c71049fd1e BUG: 874554 Signed-off-by: Harshavardhana <harsha> Reviewed-on: http://review.gluster.org/7687 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra Talur <rtalur> Reviewed-by: Vijay Bellur <vbellur>