Bug 1535332
| Summary: | The used space in the volume increases when the volume is expanded | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rachael <rgeorge> | |
| Component: | rhgs-server-container | Assignee: | Raghavendra Talur <rtalur> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Rachael <rgeorge> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | cns-3.6 | CC: | amukherj, hchiramm, jmulligan, knarra, madam, pprakash, rcyriac, rgeorge, rhs-bugs, rtalur, sarumuga, storage-qa-internal | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1540961 (view as bug list) | Environment: | ||
| Last Closed: | 2020-02-19 14:16:31 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1540961 | |||
| Bug Blocks: | ||||
** gluster volume info after expansion:
sh-4.2# gluster vol info vol_14a05e3c1f728afdebc484c15f7c6cad
Volume Name: vol_14a05e3c1f728afdebc484c15f7c6cad
Type: Distributed-Replicate
Volume ID: 15550f1f-743b-494d-b761-81d03c8bb24f
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.46.21:/var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick
Brick2: 10.70.47.68:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick
Brick3: 10.70.46.153:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick
Brick4: 10.70.46.153:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick
Brick5: 10.70.46.21:/var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick
Brick6: 10.70.47.68:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
cluster.brick-multiplex: on
** ls on the client after expansion:
# ls -lash /usr/share/busybox/
total 40894472
4 drwxrwsr-x 4 root 2000 4.0K Jan 16 14:25 .
0 drwxr-xr-x 3 root root 21 Jan 18 07:12 ..
4 drwxr-sr-x 3 root 2000 4.0K Jan 16 14:25 .trashcan
40894464 -rw-r--r-- 1 root 2000 39.0G Jan 16 14:23 39gfile
** ls on the bricks after expansion
* node 1
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick/
total 23G
0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:25 .
0 drwxr-xr-x. 3 root root 19 Jan 16 14:25 ..
0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs
0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan
23G ---------T. 2 root 2000 0 Jan 16 14:25 39gfile
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick/
total 40G
0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:23 .
0 drwxr-xr-x. 3 root root 19 Jan 16 14:14 ..
0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs
0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile
* node 2
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick/
total 40G
0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:23 .
0 drwxr-xr-x. 3 root root 19 Jan 16 14:14 ..
0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs
0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick/
total 23G
0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:25 .
0 drwxr-xr-x. 3 root root 19 Jan 16 14:25 ..
0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs
0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan
23G ---------T. 2 root 2000 0 Jan 16 14:25 39gfile
* node 3
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick/
total 40G
0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:23 .
0 drwxr-xr-x. 3 root root 19 Jan 16 14:14 ..
0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs
0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile
sh-4.2# ls -lash /var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick/
total 23G
0 drwxrwsr-x. 4 root 2000 56 Jan 16 14:25 .
0 drwxr-xr-x. 3 root root 19 Jan 16 14:25 ..
0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs
0 drwxr-sr-x. 3 root 2000 25 Jan 16 14:25 .trashcan
23G ---------T. 2 root 2000 0 Jan 16 14:25 39gfile
df output seems to showing sum of space occupied by real file and link to file
10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
70.0G 61.5G 8.4G 88% /usr/share/busybox
The system details are : OCP master 10.70.46.2 root/aplo This is probably the root cause (found by vbellur): https://stackoverflow.com/questions/32799252/why-fallocate-on-linux-creates-a-non-empty-file-when-it-has-not-enough-space https://code.engineering.redhat.com/gerrit/#/c/129236/1 tests the hypothesis. If this works, I will redo this cleanly. This bug is now a tracker for RHGS bug 1540961 |
Description of problem: A volume of size 40GB was created and a file of 39GB was created on it. The volume was expanded by 30GB and the total size of the volume became 70GB. But the used space on the volume increased from 39GB to 61.5GB Version-Release number of selected component (if applicable): cns-deploy-5.0.0-57.el7rhgs.x86_64 heketi-client-5.0.0-19.el7rhgs.x86_64 How reproducible: Steps to Reproduce: 1.[root@dhcp46-2 ~]# oc rsh busybox / # df -h Filesystem Size Used Available Use% Mounted on /dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687 10.0G 34.1M 9.9G 0% / tmpfs 11.7G 0 11.7G 0% /dev tmpfs 11.7G 0 11.7G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /dev/termination-log /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /run/secrets /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/resolv.conf /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hostname /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 40.0G 33.1M 39.9G 0% /usr/share/busybox tmpfs 11.7G 16.0K 11.7G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 11.7G 0 11.7G 0% /proc/kcore tmpfs 11.7G 0 11.7G 0% /proc/timer_list tmpfs 11.7G 0 11.7G 0% /proc/timer_stats tmpfs 11.7G 0 11.7G 0% /proc/sched_debug 2. / # cd /usr/share/busybox /usr/share/busybox # fallocate -l 39G 39gfile 3./usr/share/busybox # df -h Filesystem Size Used Available Use% Mounted on /dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687 10.0G 34.1M 9.9G 0% / tmpfs 11.7G 0 11.7G 0% /dev tmpfs 11.7G 0 11.7G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /dev/termination-log /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /run/secrets /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/resolv.conf /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hostname /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.4M 58.3G 1% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 40.0G 39.0G 966.9M 98% /usr/share/busybox tmpfs 11.7G 16.0K 11.7G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 11.7G 0 11.7G 0% /proc/kcore tmpfs 11.7G 0 11.7G 0% /proc/timer_list tmpfs 11.7G 0 11.7G 0% /proc/timer_stats tmpfs 11.7G 0 11.7G 0% /proc/sched_debug 4. [root@dhcp46-2 ~]# heketi-cli volume expand --expand-size=30 --volume=14a05e3c1f728afdebc484c15f7c6cad Name: vol_14a05e3c1f728afdebc484c15f7c6cad Size: 70 Volume Id: 14a05e3c1f728afdebc484c15f7c6cad Cluster Id: 096f8c70caa54788a00e4458c052656e Mount: 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad Mount Options: backup-volfile-servers=10.70.46.21,10.70.47.68 Block: false Free Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 5. / # df -h Filesystem Size Used Available Use% Mounted on /dev/mapper/docker-253:3-100663625-7ec3447870a5ff96a7b34e0213b31ef1a771a4424968e2c19f6da667d20e0687 10.0G 34.1M 9.9G 0% / tmpfs 11.7G 0 11.7G 0% /dev tmpfs 11.7G 0 11.7G 0% /sys/fs/cgroup /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /dev/termination-log /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /run/secrets /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /etc/resolv.conf /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /etc/hostname /dev/mapper/vg_rhel_dhcp47----104--var-lv_var 59.0G 642.7M 58.3G 1% /etc/hosts shm 64.0M 0 64.0M 0% /dev/shm 10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad 70.0G 61.5G 8.4G 88% /usr/share/busybox tmpfs 11.7G 16.0K 11.7G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 11.7G 0 11.7G 0% /proc/kcore tmpfs 11.7G 0 11.7G 0% /proc/timer_list tmpfs 11.7G 0 11.7G 0% /proc/timer_stats tmpfs 11.7G 0 11.7G 0% /proc/sched_debug Actual results: The used space increases to 61.5GB Expected results: The used space should stay the same i.e., 39GB Additional info: