Bug 1591437 – Heketi delete of orphaned volume returns error

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1591437 - Heketi delete of orphaned volume returns error

Summary:	Heketi delete of orphaned volume returns error

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	heketi (Show other bugs)
Sub Component:
Version:	cns-3.9
Hardware:	Unspecified Unspecified

Priority	unspecified Severity medium
Target Milestone:	---
Target Release:	CNS 3.10
Assigned To:	John Mulligan
QA Contact:	vinutha
Docs Contact:

URL:
Whiteboard:
Keywords:

Duplicates:	1603098 (view as bug list)
Depends On:
Blocks:	1542093 1568862
	Show dependency tree / graph

Reported:	2018-06-14 13:46 EDT by Thom Carlin
Modified:	2018-10-29 04:44 EDT (History)
CC List:	12 users (show)

See Also:
Fixed In Version:	heketi-7.0.0-2.el7rhgs
Doc Type:	Bug Fix
Doc Text:	Previously, Heketi could not delete volumes that were fully or partially removed from the underlying storage system. Running heketi commands such as 'heketi-cli volume delete <ID>' on a volume that was deleted from gluster failed. With this fix, Heketi volume delete functionality can incrementally delete a volume, skipping over items that have already been deleted.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-09-12 05:23:45 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2018:2686	None	None	None	2018-09-12 05:24 EDT

Groups: None (edit)

Description Thom Carlin 2018-06-14 13:46:58 EDT

Description of problem:

For CNS, Multiple orphaned volumes were found where heketi volumes (besides heketidbstorage) did not have corresponding OpenShift PVs.

Upon heketi volume deletion, received unexpected error.

Version-Release number of selected component (if applicable):

6.0.0-7.4.el7rhgs

How reproducible:

100% for this specific volume

Steps to Reproduce:
1. oc rsh <<heketi-pod>>
2. heketi-cli volume delete <<volume-id>>

Actual results:

Error: Cannot delete thin pool <<thin_pool_name>> on <<openshift_node_name>> because it is used by [-1] snapshot(s) or cloned volume(s)

Expected results:

Successful volume deletion

Additional info:

Will update bz if this applies to multiple volumes

Comment 2 Thom Carlin 2018-06-14 13:56:00 EDT

Occurs for multiple volumes in this cluster

Comment 3 John Mulligan 2018-06-14 14:15:26 EDT

Let's first try and determine what's in the thin pool in question. On the host where heketi created the brick that is failing to delete, run 'lvs'  as well as getting the id of the brick (should be in heketi logs).

In addition, as a proactive measure let's get the heketi db dump (heketi-cli db dump) and topology info (heketi-cli topology info) and attach them to the bug please.

Comment 5 Thom Carlin 2018-06-14 14:37:19 EDT

Item 1: on OpenShift node: lvs
Item 2: oc logs <<heketi_pod_name>>
Item 3: oc rsh <<heketi_pod_name>>
        # heketi-cli db dump
Item 4: oc rsh <<heketi_pod_name>>
        # heketi-cli topology info
are being added as private attachments

Comment 6 Thom Carlin 2018-06-14 14:47:43 EDT

Clarification for comment 5:
Item 1: On OpenShift node: lvs > /tmp/lvs
Item 2: oc logs <<heketi_pod_name>> > /tmp/heketi.log
Item 3: oc rsh <<heketi_pod_name>> heketi-cli db dump > /tmp/heketi_db_dump
Item 4: oc rsh <<heketi_pod_name>> heketi-cli topology info > /tmp/heketi_topology_info

Comment 14 Thom Carlin 2018-06-14 23:30:37 EDT

Updated Item 1: lvs --all -o lv_all >/tmp/lvs

Does this provide you with additional useful information beyond a simple "lvs"?

Comment 25 Bipin Kunal 2018-07-20 07:34:33 EDT

*** Bug 1603098 has been marked as a duplicate of this bug. ***

Comment 26 Anjana 2018-08-30 19:43:26 EDT

Updated doc text in the Doc Text field. Please review for technical accuracy.

Comment 28 Anjana 2018-09-05 05:43:07 EDT

I have updated the text to failed.

Comment 29 John Mulligan 2018-09-05 15:24:02 EDT

(In reply to Anjana from comment #28)
> I have updated the text to failed.

Looks OK to me.

Comment 31 errata-xmlrpc 2018-09-12 05:23:45 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686

Note You need to log in before you can comment on or make changes to this bug.