Description of problem: A volume of size 40GB was created and a file of 39GB was created on it. The volume was expanded by 30GB and the total size of the volume became 70GB. But the used space on the volume increased from 39GB to 61.5GB Version-Release number of selected component (if applicable): cns-deploy-5.0.0-57.el7rhgs.x86_64 heketi-client-5.0.0-19.el7rhgs.x86_64 How reproducible: Steps to Reproduce: 1.[root@dhcp46-2 ~]# oc rsh busybox / # df -h Filesystem Size Used Available Use% Mounted on /dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687 10.0G 34.1M 9.9G 0% / tmpfs 11.7G 0 11.7G 0% /dev tmpfs 11.7G 0 11.7G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /dev/termination-log /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /run/secrets /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/resolv.conf /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hostname /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 40.0G 33.1M 39.9G 0% /usr/share/busybox tmpfs 11.7G 16.0K 11.7G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 11.7G 0 11.7G 0% /proc/kcore tmpfs 11.7G 0 11.7G 0% /proc/timer_list tmpfs 11.7G 0 11.7G 0% /proc/timer_stats tmpfs 11.7G 0 11.7G 0% /proc/sched_debug 2. / # cd /usr/share/busybox /usr/share/busybox # fallocate -l 39G 39gfile 3./usr/share/busybox # df -h Filesystem Size Used Available Use% Mounted on /dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687 10.0G 34.1M 9.9G 0% / tmpfs 11.7G 0 11.7G 0% /dev tmpfs 11.7G 0 11.7G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /dev/termination-log /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /run/secrets /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/resolv.conf /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hostname /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 40.0G 39.0G 966.9M 98% /usr/share/busybox tmpfs 11.7G 16.0K 11.7G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 11.7G 0 11.7G 0% /proc/kcore tmpfs 11.7G 0 11.7G 0% /proc/timer_list tmpfs 11.7G 0 11.7G 0% /proc/timer_stats tmpfs 11.7G 0 11.7G 0% /proc/sched_debug 4. [root@dhcp46-2 ~]# heketi-cli volume expand --expand-size=30 --volume=14a05e3c1f728afdebc484c15f7c6cad Name: vol_14a05e3c1f728afdebc484c15f7c6cad Size: 70 Volume Id: 14a05e3c1f728afdebc484c15f7c6cad Cluster Id: 096f8c70caa54788a00e4458c052656e Mount: 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad Mount Options: backup-volfile-servers=10.70.46.21,10.70.47.68 Block: false Free Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 5. / # df -h Filesystem Size Used Available Use% Mounted on /dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687 10.0G 34.1M 9.9G 0% / tmpfs 11.7G 0 11.7G 0% /dev tmpfs 11.7G 0 11.7G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /dev/termination-log /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /run/secrets /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /etc/resolv.conf /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /etc/hostname /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 70.0G 61.5G 8.4G 88% /usr/share/busybox tmpfs 11.7G 16.0K 11.7G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 11.7G 0 11.7G 0% /proc/kcore tmpfs 11.7G 0 11.7G 0% /proc/timer_list tmpfs 11.7G 0 11.7G 0% /proc/timer_stats tmpfs 11.7G 0 11.7G 0% /proc/sched_debug Actual results: The used space increases to 61.5GB Expected results: The used space should stay the same i.e., 39GB Additional info:
** gluster volume info after expansion: sh-4.2# gluster vol info vol_14a05e3c1f728afdebc484c15f7c6cad Volume Name: vol_14a05e3c1f728afdebc484c15f7c6cad Type: Distributed-Replicate Volume ID: 15550f1f-743b-494d-b761-81d03c8bb24f Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.70.46.21:/var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick Brick2: 10.70.47.68:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick Brick3: 10.70.46.153:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick Brick4: 10.70.46.153:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick Brick5: 10.70.46.21:/var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick Brick6: 10.70.47.68:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick Options Reconfigured: nfs.disable: on transport.address-family: inet cluster.brick-multiplex: on ** ls on the client after expansion: # ls -lash /usr/share/busybox/ total 40894472 4 drwxrwsr-x 4 root 2000 4.0K Jan 16 14:25 . 0 drwxr-xr-x 3 root root 21 Jan 18 07:12 .. 4 drwxr-sr-x 3 root 2000 4.0K Jan 16 14:25 .trashcan 40894464 -rw-r--r-- 1 root 2000 39.0G Jan 16 14:23 39gfile ** ls on the bricks after expansion * node 1 sh-4.2# ls -lash /var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick/ total 23G 0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:25 . 0 drwxr-xr-x. 3 root root 19 Jan 16 14:25 .. 0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs 0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan 23G ---------T. 2 root 2000 0 Jan 16 14:25 39gfile sh-4.2# ls -lash /var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick/ total 40G 0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:23 . 0 drwxr-xr-x. 3 root root 19 Jan 16 14:14 .. 0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs 0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan 40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile * node 2 sh-4.2# ls -lash /var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick/ total 40G 0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:23 . 0 drwxr-xr-x. 3 root root 19 Jan 16 14:14 .. 0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs 0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan 40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile sh-4.2# ls -lash /var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick/ total 23G 0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:25 . 0 drwxr-xr-x. 3 root root 19 Jan 16 14:25 .. 0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs 0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan 23G ---------T. 2 root 2000 0 Jan 16 14:25 39gfile * node 3 sh-4.2# ls -lash /var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick/ total 40G 0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:23 . 0 drwxr-xr-x. 3 root root 19 Jan 16 14:14 .. 0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs 0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan 40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile sh-4.2# ls -lash /var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick/ total 23G 0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:25 . 0 drwxr-xr-x. 3 root root 19 Jan 16 14:25 .. 0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs 0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan 23G ---------T. 2 root 2000 0 Jan 16 14:25 39gfile
df output seems to showing sum of space occupied by real file and link to file 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 70.0G 61.5G 8.4G 88% /usr/share/busybox
The system details are : OCP master 10.70.46.2 root/aplo
This is probably the root cause (found by vbellur): https://stackoverflow.com/questions/32799252/why-fallocate-on-linux-creates-a-non-empty-file-when-it-has-not-enough-space
https://code.engineering.redhat.com/gerrit/#/c/129236/1 tests the hypothesis. If this works, I will redo this cleanly.
This bug is now a tracker for RHGS bug 1540961