Bug 1591437

Summary:	Heketi delete of orphaned volume returns error
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Thom Carlin <tcarlin>
Component:	heketi	Assignee:	John Mulligan <jmulligan>
Status:	CLOSED ERRATA	QA Contact:	vinutha <vinug>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	cns-3.9	CC:	akrishna, andcosta, hchiramm, jmulligan, kramdoss, madam, pprakash, rhs-bugs, rtalur, sankarshan, storage-qa-internal, tcarlin
Target Milestone:	---
Target Release:	CNS 3.10
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	heketi-7.0.0-2.el7rhgs	Doc Type:	Bug Fix
Doc Text:	Previously, Heketi could not delete volumes that were fully or partially removed from the underlying storage system. Running heketi commands such as 'heketi-cli volume delete <ID>' on a volume that was deleted from gluster failed. With this fix, Heketi volume delete functionality can incrementally delete a volume, skipping over items that have already been deleted.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-09-12 09:23:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1724792, 1568862

Description Thom Carlin 2018-06-14 17:46:58 UTC

Description of problem:

For CNS, Multiple orphaned volumes were found where heketi volumes (besides heketidbstorage) did not have corresponding OpenShift PVs.

Upon heketi volume deletion, received unexpected error.

Version-Release number of selected component (if applicable):

6.0.0-7.4.el7rhgs

How reproducible:

100% for this specific volume

Steps to Reproduce:
1. oc rsh <<heketi-pod>>
2. heketi-cli volume delete <<volume-id>>

Actual results:

Error: Cannot delete thin pool <<thin_pool_name>> on <<openshift_node_name>> because it is used by [-1] snapshot(s) or cloned volume(s)

Expected results:

Successful volume deletion

Additional info:

Will update bz if this applies to multiple volumes

Comment 2 Thom Carlin 2018-06-14 17:56:00 UTC

Occurs for multiple volumes in this cluster

Comment 3 John Mulligan 2018-06-14 18:15:26 UTC

Let's first try and determine what's in the thin pool in question. On the host where heketi created the brick that is failing to delete, run 'lvs'  as well as getting the id of the brick (should be in heketi logs).

In addition, as a proactive measure let's get the heketi db dump (heketi-cli db dump) and topology info (heketi-cli topology info) and attach them to the bug please.

Comment 5 Thom Carlin 2018-06-14 18:37:19 UTC

Item 1: on OpenShift node: lvs
Item 2: oc logs <<heketi_pod_name>>
Item 3: oc rsh <<heketi_pod_name>>
        # heketi-cli db dump
Item 4: oc rsh <<heketi_pod_name>>
        # heketi-cli topology info
are being added as private attachments

Comment 6 Thom Carlin 2018-06-14 18:47:43 UTC

Clarification for comment 5:
Item 1: On OpenShift node: lvs > /tmp/lvs
Item 2: oc logs <<heketi_pod_name>> > /tmp/heketi.log
Item 3: oc rsh <<heketi_pod_name>> heketi-cli db dump > /tmp/heketi_db_dump
Item 4: oc rsh <<heketi_pod_name>> heketi-cli topology info > /tmp/heketi_topology_info

Comment 14 Thom Carlin 2018-06-15 03:30:37 UTC

Updated Item 1: lvs --all -o lv_all >/tmp/lvs

Does this provide you with additional useful information beyond a simple "lvs"?

Comment 25 Bipin Kunal 2018-07-20 11:34:33 UTC

*** Bug 1603098 has been marked as a duplicate of this bug. ***

Comment 26 Anjana KD 2018-08-30 23:43:26 UTC

Updated doc text in the Doc Text field. Please review for technical accuracy.

Comment 28 Anjana KD 2018-09-05 09:43:07 UTC

I have updated the text to failed.

Comment 29 John Mulligan 2018-09-05 19:24:02 UTC

(In reply to Anjana from comment #28)
> I have updated the text to failed.

Looks OK to me.

Comment 31 errata-xmlrpc 2018-09-12 09:23:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686