Hide Forgot
Description of problem: While testing with block hosting volumes,post deleting the volume from heketi cli,few volumes failed to delete with message "target is busy".However if we login to the gluster pod post that,the volume gets successfully deleted from backend. This leads to mismatch in heketi volume etries and gluster backend. On gluster pod,though the volume is deleted,but bricks were still mounted. Version-Release number of selected component (if applicable): openshift_storage_glusterfs_heketi_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-volmanager-rhel7:3.11.1-1' openshift_storage_glusterfs_block_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-gluster-block-prov-rhel7:3.11.1-1' openshift_storage_glusterfs_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-server-rhel7:3.11.1-1' # oc rsh heketi-storage-1-chhcs rsh-4.2# rpm -qa | grep heketi heketi-client-8.0.0-1.el7rhgs.x86_64 heketi-8.0.0-1.el7rhgs.x86_64 # oc version oc v3.11.43 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://dhcp47-138.lab.eng.blr.redhat.com:8443 openshift v3.11.43 kubernetes v1.11.0+d4cacc0 How reproducible: Hit 2-3 times Steps to Reproduce: 1.Create 50 block devices pvc's (5GB) hosting on 3 block hosting volumes 2.Create 50 cirros app pods from these block pvc's. 3.Delete the app pods and dc 4.Delete all the pvc's 5.Check for #heketi-cli blockvolume list (It shows 0 entry since all pvc's are deleted) [root@dhcp47-138 scripts]# heketi-cli blockvolume list [root@dhcp47-138 scripts]# 6.Start deleting blockhosting volume ======== # heketi-cli volume list Id:06f28604994dd5e87fbd85871968db4b Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:vol_06f28604994dd5e87fbd85871968db4b [block] Id:4a25a5dfdd217c589222e638a21fc3e9 Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:vol_4a25a5dfdd217c589222e638a21fc3e9 [block] Id:8f6e417522ea27ce3cbe194bac337499 Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:vol_8f6e417522ea27ce3cbe194bac337499 [block] Id:cd94e0c50b94551ced09244c198d58b1 Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:heketidbstorage ======= [root@dhcp47-138 scripts]# heketi-cli volume delete 06f28604994dd5e87fbd85871968db4b Error: umount: /var/lib/heketi/mounts/vg_dca41af8b5c15e419f66928440c4d9d6/brick_f225251573ef845f5591e578a1f10652: target is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) [root@dhcp47-138 scripts]# heketi-cli volume delete 4a25a5dfdd217c589222e638a21fc3e9 Error: umount: /var/lib/heketi/mounts/vg_55d43508331a4ba298eee11d6f3c39a1/brick_a40130d2d380979ecce853a5e40308f2: target is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) [root@dhcp47-138 scripts]# heketi-cli volume delete 8f6e417522ea27ce3cbe194bac337499 Volume 8f6e417522ea27ce3cbe194bac337499 deleted ============== 7.Check for heketi-cli volume list and volume present in gluster backend ====== # heketi-cli volume list Id:06f28604994dd5e87fbd85871968db4b Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:vol_06f28604994dd5e87fbd85871968db4b [block] Id:4a25a5dfdd217c589222e638a21fc3e9 Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:vol_4a25a5dfdd217c589222e638a21fc3e9 [block] Id:cd94e0c50b94551ced09244c198d58b1 Cluster:3b834980ae06d3950765eaf0c7bc20a1 Name:heketidbstorage # gluster v list heketidbstorage ====== Actual results: heketi-cli fails to delete the volume but the volume is deleted from gluster backend causing mismatch in volume entries Expected results: There should not be any mismatch in volume entries netween gluster backend and heketi db Additional info:
We need to reproduce and collect the lsof output and share back, otherwise we don't have sufficient information to debug this problem. Also make sure that gluster rpm version is reported in the comment.
I am hitting the same issue in my setup too where deleting block hosting volumes from heketi-cli throws the error below but got deleted from gluster backend. [root@dhcp46-220 ~]# heketi-cli volume delete 037a62a428cc34f540102d91333b8f34 Error: umount: /var/lib/heketi/mounts/vg_c921126c92d975b25d5903a5afdc4214/brick_0a37b52d11601de61a4d11c3a4281c99: target is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) Id:037a62a428cc34f540102d91333b8f34 Cluster:ee2fbd7973e8396a23811d54dd8ed985 Name:vol_037a62a428cc34f540102d91333b8f34 [block] Id:3dc699b513a2ce7b3a75f401fa0390bc Cluster:ee2fbd7973e8396a23811d54dd8ed985 Name:heketidbstorage Id:aa77fe2a513ff7008142b103b6360af9 Cluster:ee2fbd7973e8396a23811d54dd8ed985 Name:vol_aa77fe2a513ff7008142b103b6360af9 [block] sh-4.2# gluster volume list heketidbstorage vol_aa77fe2a513ff7008142b103b6360af9
Hello Mohit, Is there a workaround available for this issue ? Thanks kasturi
Hi Kasturi, I think it is difficult to provide any workaround to avoid the same. Thanks, Mohit Agrawal
I have modified the text slightly.
*** Bug 1665462 has been marked as a duplicate of this bug. ***
Hello Atin, Could you please provide a bug fix doc text (CCFR--> Format) and change the doctype too.
I've updated the doc text to highlight this bug has been fixed and can be captured in the release notes.
have made minor changes. Kindly review and update the flag.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0667