Bug 1593124

Summary: 'df -Th' is showing incorrect used size of brick of block hosting volume (replica with arbiter) in container
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nitin Goyal <nigoyal>
Component: gluster-blockAssignee: Prasanna Kumar Kalever <prasanna.kalever>
Status: CLOSED ERRATA QA Contact: Nitin Goyal <nigoyal>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.3CC: akrishna, bgoyal, kramdoss, madam, pkarampu, prasanna.kalever, rhs-bugs, sankarshan, vbellur, xiubli
Target Milestone: ---   
Target Release: CNS 3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tcmu-runner-1.2.0-21.el7rhgs Doc Type: Bug Fix
Doc Text:
Previously, the close operation of gluster volume entity was missing. This caused 'df -Th' to show an incorrect used size for the brick of the block hosting volume in container. With this fix, tcmu-runner closes the gluster volume entity when removing the target device and ‘df-th’ shows correct used size of brick of block hosting volume.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 09:26:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1568862    

Description Nitin Goyal 2018-06-20 06:55:01 UTC 
Description of problem: one block hosting volume of 500gb has been created. Then in that block hosting volume 40 block devices of 10gb has been created. Out of 40 devices 5 devices are got deleted. But df -Th is still showing that 401 gb is used out of 500gb.


Version-Release number of selected component (if applicable): 3.3.1-17


How reproducible:


Steps to Reproduce:
1. Create block hosting volume of 500gb.
# heketi-cli volume create --size=100 --gluster-volume-options='user.heketi.arbiter true' --block 

2. Create 40 block device of 10gb in that block hosting volume.
# oc create -f glusterfs_pvc_claim.yml

3. delete 5 pvc claims.
# oc delete pvc_name

4. 'df -Th' in the gluster pod

Actual results: It is showing that 401 gb is used.
sh-4.2# df -Th
Filesystem                                                                             Type      Size  Used Avail Use% Mounted on
/dev/dm-12                                                                             xfs        10G 1001M  9.1G  10% /
devtmpfs                                                                               devtmpfs   16G     0   16G   0% /dev
shm                                                                                    tmpfs      64M     0   64M   0% /dev/shm
/dev/sdb1                                                                              xfs        60G  6.5G   54G  11% /run
tmpfs                                                                                  tmpfs      16G  2.8M   16G   1% /run/lvm
/dev/mapper/rhel_dhcp47--42-root                                                       xfs        50G  2.2G   48G   5% /etc/ssl
tmpfs                                                                                  tmpfs      16G     0   16G   0% /sys/fs/cgroup
tmpfs                                                                                  tmpfs      16G   16K   16G   1% /run/secrets/kubernetes.io/serviceaccount
/dev/mapper/vg_f9bf2366dbb75d7400ef6c5198754c4d-brick_92dbc86dcb6f650bd2bf56979a900d94 xfs       2.0G   34M  2.0G   2% /var/lib/heketi/mounts/vg_f9bf2366dbb75d7400ef6c5198754c4d/brick_92dbc86dcb6f650bd2bf56979a900d94
/dev/mapper/vg_f9bf2366dbb75d7400ef6c5198754c4d-brick_becfa5094b001af8fb94f8694e5b4ee3 xfs       500G  401G  100G  81% /var/lib/heketi/mounts/vg_f9bf2366dbb75d7400ef6c5198754c4d/brick_becfa5094b001af8fb94f8694e5b4ee3

Expected results: It should show 350 gb is used.


Additional info:
Comment 2 Nitin Goyal 2018-06-20 07:19:16 UTC 
logs and cli outputs
http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1593124/
Comment 3 Nitin Goyal 2018-06-20 08:30:43 UTC 
Because of this issue after deleting block devices we can not use space again which was occupied by deleted block devices.
Comment 14 Nitin Goyal 2018-07-26 12:52:12 UTC 
I have verified this bug with below packages. I was not able to see the issue.

gluster-block-0.2.1-22.el7rhgs.x86_64
tcmu-runner-1.2.0-22.el7rhgs.x86_64
libtcmu-1.2.0-22.el7rhgs.x86_64
targetcli-2.1.fb46-6.el7_5.noarch
python-configshell-1.1.fb23-4.el7_5.noarch
python-rtslib-2.1.fb63-12.el7_5.noarch
glusterfs-3.8.4-54.15.el7rhgs.x86_64

outputs of commands ->

When block devices was created.

[root@dhcp46-180 ~]# for i in {101..110}; do ./pvc_create.sh c$i 5; sleep 3; done
persistentvolumeclaim "c101" created
persistentvolumeclaim "c102" created
persistentvolumeclaim "c103" created
persistentvolumeclaim "c104" created
persistentvolumeclaim "c105" created
persistentvolumeclaim "c106" created
persistentvolumeclaim "c107" created
persistentvolumeclaim "c108" created
persistentvolumeclaim "c109" created
persistentvolumeclaim "c110" created

[root@dhcp46-180 ~]# oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
c101      Bound     pvc-7bbf827c-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       51s
c102      Bound     pvc-7da58856-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       48s
c103      Bound     pvc-7f8cd92a-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       45s
c104      Bound     pvc-817409d0-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       42s
c105      Bound     pvc-8363135e-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       38s
c106      Bound     pvc-854ad3ef-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       35s
c107      Bound     pvc-873203ae-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       32s
c108      Bound     pvc-891ccd0c-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       29s
c109      Bound     pvc-8b03b526-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       26s
c110      Bound     pvc-8cf71865-90d1-11e8-8a35-005056a53010   5Gi        RWO            block-sc       22s

[root@dhcp46-180 ~]# oc rsh glusterfs-storage-x94vq df -Th
Filesystem                                                                             Type      Size  Used Avail Use% Mounted on
/dev/dm-25                                                                             xfs        10G  301M  9.7G   3% /
devtmpfs                                                                               devtmpfs   16G     0   16G   0% /dev
shm                                                                                    tmpfs      64M     0   64M   0% /dev/shm
/dev/sdb1                                                                              xfs       100G  979M   99G   1% /run
tmpfs                                                                                  tmpfs      16G  2.8M   16G   1% /run/lvm
/dev/mapper/rhel_dhcp46--180-root                                                      xfs        50G  2.2G   48G   5% /etc/ssl
tmpfs                                                                                  tmpfs      16G     0   16G   0% /sys/fs/cgroup
tmpfs                                                                                  tmpfs      16G   16K   16G   1% /run/secrets/kubernetes.io/serviceaccount
/dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_5a562e570b7f53030d93152fb07bf15c xfs       2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_5a562e570b7f53030d93152fb07bf15c
/dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_1a98d9e8356349473912731037e1589b xfs       100G   51G   50G  51% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_1a98d9e8356349473912731037e1589b


when block devices was deleted

[root@dhcp46-180 ~]# for i in {101..110}; do oc delete pvc c$i; sleep 2; done
persistentvolumeclaim "c101" deleted
persistentvolumeclaim "c102" deleted
persistentvolumeclaim "c103" deleted
persistentvolumeclaim "c104" deleted
persistentvolumeclaim "c105" deleted
persistentvolumeclaim "c106" deleted
persistentvolumeclaim "c107" deleted
persistentvolumeclaim "c108" deleted
persistentvolumeclaim "c109" deleted
persistentvolumeclaim "c110" deleted

[root@dhcp46-180 ~]# oc get pvc
No resources found.

[root@dhcp46-180 ~]# oc rsh glusterfs-storage-x94vq df -Th
Filesystem                                                                             Type      Size  Used Avail Use% Mounted on
/dev/dm-25                                                                             xfs        10G  301M  9.7G   3% /
devtmpfs                                                                               devtmpfs   16G     0   16G   0% /dev
shm                                                                                    tmpfs      64M     0   64M   0% /dev/shm
/dev/sdb1                                                                              xfs       100G  978M   99G   1% /run
tmpfs                                                                                  tmpfs      16G  2.8M   16G   1% /run/lvm
/dev/mapper/rhel_dhcp46--180-root                                                      xfs        50G  2.2G   48G   5% /etc/ssl
tmpfs                                                                                  tmpfs      16G     0   16G   0% /sys/fs/cgroup
tmpfs                                                                                  tmpfs      16G   16K   16G   1% /run/secrets/kubernetes.io/serviceaccount
/dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_5a562e570b7f53030d93152fb07bf15c xfs       2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_5a562e570b7f53030d93152fb07bf15c
/dev/mapper/vg_b9c503fe9990834c168d676619ea1f97-brick_1a98d9e8356349473912731037e1589b xfs       100G  674M  100G   1% /var/lib/heketi/mounts/vg_b9c503fe9990834c168d676619ea1f97/brick_1a98d9e8356349473912731037e1589b
Comment 15 Anjana KD 2018-08-31 00:52:55 UTC 
Updated doc text in the Doc Text field. Please review for technical accuracy.
Comment 18 errata-xmlrpc 2018-09-12 09:26:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2691