Bug 1591437 - Heketi delete of orphaned volume returns error
Summary: Heketi delete of orphaned volume returns error
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: cns-3.9
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: CNS 3.10
Assignee: John Mulligan
QA Contact: vinutha
: 1603098 (view as bug list)
Depends On:
Blocks: 1724792 1568862
TreeView+ depends on / blocked
Reported: 2018-06-14 17:46 UTC by Thom Carlin
Modified: 2019-06-28 16:04 UTC (History)
12 users (show)

Fixed In Version: heketi-7.0.0-2.el7rhgs
Doc Type: Bug Fix
Doc Text:
Previously, Heketi could not delete volumes that were fully or partially removed from the underlying storage system. Running heketi commands such as 'heketi-cli volume delete <ID>' on a volume that was deleted from gluster failed. With this fix, Heketi volume delete functionality can incrementally delete a volume, skipping over items that have already been deleted.
Clone Of:
Last Closed: 2018-09-12 09:23:45 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2686 0 None None None 2018-09-12 09:24:56 UTC

Description Thom Carlin 2018-06-14 17:46:58 UTC
Description of problem:

For CNS, Multiple orphaned volumes were found where heketi volumes (besides heketidbstorage) did not have corresponding OpenShift PVs.

Upon heketi volume deletion, received unexpected error.

Version-Release number of selected component (if applicable):


How reproducible:

100% for this specific volume

Steps to Reproduce:
1. oc rsh <<heketi-pod>>
2. heketi-cli volume delete <<volume-id>>

Actual results:

Error: Cannot delete thin pool <<thin_pool_name>> on <<openshift_node_name>> because it is used by [-1] snapshot(s) or cloned volume(s)

Expected results:

Successful volume deletion

Additional info:

Will update bz if this applies to multiple volumes

Comment 2 Thom Carlin 2018-06-14 17:56:00 UTC
Occurs for multiple volumes in this cluster

Comment 3 John Mulligan 2018-06-14 18:15:26 UTC
Let's first try and determine what's in the thin pool in question. On the host where heketi created the brick that is failing to delete, run 'lvs'  as well as getting the id of the brick (should be in heketi logs).

In addition, as a proactive measure let's get the heketi db dump (heketi-cli db dump) and topology info (heketi-cli topology info) and attach them to the bug please.

Comment 5 Thom Carlin 2018-06-14 18:37:19 UTC
Item 1: on OpenShift node: lvs
Item 2: oc logs <<heketi_pod_name>>
Item 3: oc rsh <<heketi_pod_name>>
        # heketi-cli db dump
Item 4: oc rsh <<heketi_pod_name>>
        # heketi-cli topology info
are being added as private attachments

Comment 6 Thom Carlin 2018-06-14 18:47:43 UTC
Clarification for comment 5:
Item 1: On OpenShift node: lvs > /tmp/lvs
Item 2: oc logs <<heketi_pod_name>> > /tmp/heketi.log
Item 3: oc rsh <<heketi_pod_name>> heketi-cli db dump > /tmp/heketi_db_dump
Item 4: oc rsh <<heketi_pod_name>> heketi-cli topology info > /tmp/heketi_topology_info

Comment 14 Thom Carlin 2018-06-15 03:30:37 UTC
Updated Item 1: lvs --all -o lv_all >/tmp/lvs

Does this provide you with additional useful information beyond a simple "lvs"?

Comment 25 Bipin Kunal 2018-07-20 11:34:33 UTC
*** Bug 1603098 has been marked as a duplicate of this bug. ***

Comment 26 Anjana KD 2018-08-30 23:43:26 UTC
Updated doc text in the Doc Text field. Please review for technical accuracy.

Comment 28 Anjana KD 2018-09-05 09:43:07 UTC
I have updated the text to failed.

Comment 29 John Mulligan 2018-09-05 19:24:02 UTC
(In reply to Anjana from comment #28)
> I have updated the text to failed.

Looks OK to me.

Comment 31 errata-xmlrpc 2018-09-12 09:23:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.