Bug 1603098

Summary:	Unable to delete volumes in Heketi when volumes are not present on GlusterFS nodes
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Andre Costa <andcosta>
Component:	heketi	Assignee:	John Mulligan <jmulligan>
Status:	CLOSED DUPLICATE	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	cns-3.6	CC:	bkunal, hchiramm, jmulligan, kramdoss, pprakash, rhs-bugs, sankarshan, storage-qa-internal, vinug
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-07-20 11:34:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andre Costa 2018-07-19 07:29:12 UTC

Description of problem:
There are 3 volumes on heketi-cli volume list that are not present on the GlusterFS nodes anymore, and trying to delete them it gives this error:

  # oc exec heketi-2-v9ccj -- heketi-cli --user admin --secret "$HEKETI_ADMIN_KEY" volume delete 81407b43a4cc5e28452332e4eb73edc2
Error: Cannot delete thin pool tp_294f6047afd358734cd3e3363ec38412 on ocp-cns-xxx because it is used by [-1] snapshot(s) or cloned volume(s)

Trying to run against the database always gives this error:

sh-4.2# heketi db delete-bricks-with-empty-path --dbfile=/var/lib/heketi/heketi.db --all
[heketi] ERROR 2018/07/11 15:18:14 /src/github.com/heketi/heketi/apps/glusterfs/dbcommon.go:173: Unable to open database: timeout
Unable to open database: timeout

Version-Release number of selected component (if applicable):
rhgs3/rhgs-volmanager-rhel7:3.3.1-8.1 - heketi 6.0.0
GlusterFS 
How reproducible:
Everytime, but not sure why heketi still has a reference for non-existing volumes on the cluster.
Tried to scale down and up heketi pod didn't work either.

Actual results:
Reference for non-existing volumes stay present on heketi.db

Expected results:
heketi-cli should be able to delete volumes or stay sync with Gluster or have better error info, since there are no snapshots.

Additional info:

Comment 2 John Mulligan 2018-07-19 17:15:09 UTC

Agreed, work has gone into this area as part of the heketi 3.10 effort. Please see bz#1591437
If you agree, I think that this issue is similar enough to that bug to make this a duplicate.

Comment 4 Bipin Kunal 2018-07-20 11:34:33 UTC

closing this as duplicate as per comment 3 and 2.

-Bipin

*** This bug has been marked as a duplicate of bug 1591437 ***