Description of problem: one block hosting volume of 500gb has been created. Then in that block hosting volume 40 block devices of 10gb has been created. Out of 40 devices 5 devices are got deleted. But df -Th is still showing that 401 gb is used out of 500gb. Version-Release number of selected component (if applicable): 3.3.1-17 How reproducible: Steps to Reproduce: 1. Create block hosting volume of 500gb. # heketi-cli volume create --size=100 --gluster-volume-options='user.heketi.arbiter true' --block 2. Create 40 block device of 10gb in that block hosting volume. # oc create -f glusterfs_pvc_claim.yml 3. delete 5 pvc claims. # oc delete pvc_name 4. 'df -Th' in the gluster pod Actual results: It is showing that 401 gb is used. sh-4.2# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/dm-12 xfs 10G 1001M 9.1G 10% / devtmpfs devtmpfs 16G 0 16G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm /dev/sdb1 xfs 60G 6.5G 54G 11% /run tmpfs tmpfs 16G 2.8M 16G 1% /run/lvm /dev/mapper/rhel_dhcp47--42-root xfs 50G 2.2G 48G 5% /etc/ssl tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs tmpfs 16G 16K 16G 1% /run/secrets/kubernetes.io/serviceaccount /dev/mapper/vg_f9bf2366dbb75d7400ef6c5198754c4d-brick_92dbc86dcb6f650bd2bf56979a900d94 xfs 2.0G 34M 2.0G 2% /var/lib/heketi/mounts/vg_f9bf2366dbb75d7400ef6c5198754c4d/brick_92dbc86dcb6f650bd2bf56979a900d94 /dev/mapper/vg_f9bf2366dbb75d7400ef6c5198754c4d-brick_becfa5094b001af8fb94f8694e5b4ee3 xfs 500G 401G 100G 81% /var/lib/heketi/mounts/vg_f9bf2366dbb75d7400ef6c5198754c4d/brick_becfa5094b001af8fb94f8694e5b4ee3 Expected results: It should show 350 gb is used. Additional info:
logs and cli outputs http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1593124/
Because of this issue after deleting block devices we can not use space again which was occupied by deleted block devices.
I have verified this bug with below packages. I was not able to see the issue. gluster-block-0.2.1-22.el7rhgs.x86_64 tcmu-runner-1.2.0-22.el7rhgs.x86_64 libtcmu-1.2.0-22.el7rhgs.x86_64 targetcli-2.1.fb46-6.el7_5.noarch python-configshell-1.1.fb23-4.el7_5.noarch python-rtslib-2.1.fb63-12.el7_5.noarch glusterfs-3.8.4-54.15.el7rhgs.x86_64 outputs of commands -> When block devices was created. [root@dhcp46-180 ~]# for i in {101..110}; do ./pvc_create.sh c$i 5; sleep 3; done persistentvolumeclaim "c101" created persistentvolumeclaim "c102" created persistentvolumeclaim "c103" created persistentvolumeclaim "c104" created persistentvolumeclaim "c105" created persistentvolumeclaim "c106" created persistentvolumeclaim "c107" created persistentvolumeclaim "c108" created persistentvolumeclaim "c109" created persistentvolumeclaim "c110" created [root@dhcp46-180 ~]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE c101 Bound pvc-7bbf827c-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 51s c102 Bound pvc-7da58856-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 48s c103 Bound pvc-7f8cd92a-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 45s c104 Bound pvc-817409d0-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 42s c105 Bound pvc-8363135e-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 38s c106 Bound pvc-854ad3ef-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 35s c107 Bound pvc-873203ae-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 32s c108 Bound pvc-891ccd0c-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 29s c109 Bound pvc-8b03b526-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 26s c110 Bound pvc-8cf71865-90d1-11e8-8a35-005056a53010 5Gi RWO block-sc 22s [root@dhcp46-180 ~]# oc rsh glusterfs-storage-x94vq df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/dm-25 xfs 10G 301M 9.7G 3% / devtmpfs devtmpfs 16G 0 16G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm /dev/sdb1 xfs 100G 979M 99G 1% /run tmpfs tmpfs 16G 2.8M 16G 1% /run/lvm /dev/mapper/rhel_dhcp46--180-root xfs 50G 2.2G 48G 5% /etc/ssl tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs tmpfs 16G 16K 16G 1% /run/secrets/kubernetes.io/serviceaccount /dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_5a562e570b7f53030d93152fb07bf15c xfs 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_5a562e570b7f53030d93152fb07bf15c /dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_1a98d9e8356349473912731037e1589b xfs 100G 51G 50G 51% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_1a98d9e8356349473912731037e1589b when block devices was deleted [root@dhcp46-180 ~]# for i in {101..110}; do oc delete pvc c$i; sleep 2; done persistentvolumeclaim "c101" deleted persistentvolumeclaim "c102" deleted persistentvolumeclaim "c103" deleted persistentvolumeclaim "c104" deleted persistentvolumeclaim "c105" deleted persistentvolumeclaim "c106" deleted persistentvolumeclaim "c107" deleted persistentvolumeclaim "c108" deleted persistentvolumeclaim "c109" deleted persistentvolumeclaim "c110" deleted [root@dhcp46-180 ~]# oc get pvc No resources found. [root@dhcp46-180 ~]# oc rsh glusterfs-storage-x94vq df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/dm-25 xfs 10G 301M 9.7G 3% / devtmpfs devtmpfs 16G 0 16G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm /dev/sdb1 xfs 100G 978M 99G 1% /run tmpfs tmpfs 16G 2.8M 16G 1% /run/lvm /dev/mapper/rhel_dhcp46--180-root xfs 50G 2.2G 48G 5% /etc/ssl tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs tmpfs 16G 16K 16G 1% /run/secrets/kubernetes.io/serviceaccount /dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_5a562e570b7f53030d93152fb07bf15c xfs 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_5a562e570b7f53030d93152fb07bf15c /dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_1a98d9e8356349473912731037e1589b xfs 100G 674M 100G 1% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_1a98d9e8356349473912731037e1589b
Updated doc text in the Doc Text field. Please review for technical accuracy.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2691