Bug 1278397

Summary:	When volume creation fails, gluster volume and brick lvs are not deleted
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Anush Shetty <ashetty>
Component:	heketi	Assignee:	Luis Pabón <lpabon>
Status:	CLOSED ERRATA	QA Contact:	Anush Shetty <ashetty>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	madam, rcyriac, vagarwal
Target Milestone:	---	Keywords:	Reopened, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	heketi-1.0.2-1.fc23 heketi-1.0.2-1.fc22 heketi-1.0.2-1.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1292205 (view as bug list)		Environment:
Last Closed:	2015-12-16 17:41:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Anush Shetty 2015-11-05 11:51:43 UTC

Description of problem: When heketi volume creation fails (because of gluster volumes failing to start), heketi fails to clean up the logical volumes cit attempts to clean up the lvs without deleting the gluster volume. 


Version-Release number of selected component (if applicable): heketi-1.0.1-1.el7rhgs.x86_64


How reproducible: Always when volume creation fails.

Actual results:

Excerpt from the logs:

[sshexec] ERROR 2015/11/05 16:00:54 /builddir/build/BUILD/heketi-f9273193371ab92456a6b042e60fc883f5b67d8b/src/github.com/heketi/heketi/executors/sshexec/brick.go:125:
 Process exited with: 32. Reason was:  ()
[sshexec] ERROR 2015/11/05 16:00:54 /builddir/build/BUILD/heketi-f9273193371ab92456a6b042e60fc883f5b67d8b/src/github.com/heketi/heketi/utils/ssh/ssh.go:155: Failed to
 run command [sudo lvremove -f vg_f9188e0f1334126cdc51a5619de2e1c5/tp_813ffa9b7a442ff8234210bb648f8578] on rhshdp12.lab.eng.blr.redhat.com:22: Err[Process exited with
: 5. Reason was:  ()]: Stdout []: Stderr [  Logical volume vg_f9188e0f1334126cdc51a5619de2e1c5/brick_813ffa9b7a442ff8234210bb648f8578 contains a filesystem in use.

Expected results:

gluster volumes should be deleted before attempting to delete the logical volumes. 

Additional info:

Comment 2 Luis Pabón 2015-11-05 21:03:03 UTC

https://github.com/heketi/heketi/issues/247

Comment 3 Luis Pabón 2015-11-05 22:20:52 UTC

Please try out different max brick values using the Heketi version from https://github.com/heketi/heketi/issues/248 .

Comment 4 Luis Pabón 2015-11-06 20:14:32 UTC

https://github.com/heketi/heketi/issues/247 has been updated with a patch.  I created a functional test to reproduce the issue.  I noticed that having a large number of maximum bricks per volume did not produce reliable results from GlusterFS volume create.  I have limited the maximum number of bricks per volume to 100.

Comment 5 Fedora Update System 2015-12-04 00:42:23 UTC

heketi-1.0.2-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-34d6d5296a

Comment 6 Fedora Update System 2015-12-04 20:43:59 UTC

heketi-1.0.2-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-6b376a19a9

Comment 7 Fedora Update System 2015-12-04 20:44:35 UTC

heketi-1.0.2-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-d0abd928f1

Comment 8 Fedora Update System 2015-12-05 20:51:12 UTC

heketi-1.0.2-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'yum --enablerepo=epel-testing update heketi'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-34d6d5296a

Comment 9 Fedora Update System 2015-12-06 05:22:19 UTC

heketi-1.0.2-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update heketi'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-d0abd928f1

Comment 10 Fedora Update System 2015-12-06 17:20:56 UTC

heketi-1.0.2-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update heketi'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-6b376a19a9

Comment 12 Anush Shetty 2015-12-09 10:16:43 UTC

Verified with heketi-1.0.2-1.el7rhgs.x86_64

Comment 13 Fedora Update System 2015-12-14 10:22:03 UTC

heketi-1.0.2-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Fedora Update System 2015-12-14 11:51:13 UTC

heketi-1.0.2-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2015-12-30 18:04:14 UTC

heketi-1.0.2-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.