Bug 1278397 - When volume creation fails, gluster volume and brick lvs are not deleted
Summary: When volume creation fails, gluster volume and brick lvs are not deleted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Luis Pabón
QA Contact: Anush Shetty
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-05 11:51 UTC by Anush Shetty
Modified: 2016-11-08 22:25 UTC (History)
3 users (show)

Fixed In Version: heketi-1.0.2-1.fc23 heketi-1.0.2-1.fc22 heketi-1.0.2-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1292205 (view as bug list)
Environment:
Last Closed: 2015-12-16 17:41:37 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Anush Shetty 2015-11-05 11:51:43 UTC
Description of problem: When heketi volume creation fails (because of gluster volumes failing to start), heketi fails to clean up the logical volumes cit attempts to clean up the lvs without deleting the gluster volume. 


Version-Release number of selected component (if applicable): heketi-1.0.1-1.el7rhgs.x86_64


How reproducible: Always when volume creation fails.

Actual results:

Excerpt from the logs:

[sshexec] ERROR 2015/11/05 16:00:54 /builddir/build/BUILD/heketi-f9273193371ab92456a6b042e60fc883f5b67d8b/src/github.com/heketi/heketi/executors/sshexec/brick.go:125:
 Process exited with: 32. Reason was:  ()
[sshexec] ERROR 2015/11/05 16:00:54 /builddir/build/BUILD/heketi-f9273193371ab92456a6b042e60fc883f5b67d8b/src/github.com/heketi/heketi/utils/ssh/ssh.go:155: Failed to
 run command [sudo lvremove -f vg_f9188e0f1334126cdc51a5619de2e1c5/tp_813ffa9b7a442ff8234210bb648f8578] on rhshdp12.lab.eng.blr.redhat.com:22: Err[Process exited with
: 5. Reason was:  ()]: Stdout []: Stderr [  Logical volume vg_f9188e0f1334126cdc51a5619de2e1c5/brick_813ffa9b7a442ff8234210bb648f8578 contains a filesystem in use.

Expected results:

gluster volumes should be deleted before attempting to delete the logical volumes. 

Additional info:

Comment 2 Luis Pabón 2015-11-05 21:03:03 UTC
https://github.com/heketi/heketi/issues/247

Comment 3 Luis Pabón 2015-11-05 22:20:52 UTC
Please try out different max brick values using the Heketi version from https://github.com/heketi/heketi/issues/248 .

Comment 4 Luis Pabón 2015-11-06 20:14:32 UTC
https://github.com/heketi/heketi/issues/247 has been updated with a patch.  I created a functional test to reproduce the issue.  I noticed that having a large number of maximum bricks per volume did not produce reliable results from GlusterFS volume create.  I have limited the maximum number of bricks per volume to 100.

Comment 5 Fedora Update System 2015-12-04 00:42:23 UTC
heketi-1.0.2-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-34d6d5296a

Comment 6 Fedora Update System 2015-12-04 20:43:59 UTC
heketi-1.0.2-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-6b376a19a9

Comment 7 Fedora Update System 2015-12-04 20:44:35 UTC
heketi-1.0.2-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-d0abd928f1

Comment 8 Fedora Update System 2015-12-05 20:51:12 UTC
heketi-1.0.2-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'yum --enablerepo=epel-testing update heketi'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-34d6d5296a

Comment 9 Fedora Update System 2015-12-06 05:22:19 UTC
heketi-1.0.2-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update heketi'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-d0abd928f1

Comment 10 Fedora Update System 2015-12-06 17:20:56 UTC
heketi-1.0.2-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update heketi'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-6b376a19a9

Comment 12 Anush Shetty 2015-12-09 10:16:43 UTC
Verified with heketi-1.0.2-1.el7rhgs.x86_64

Comment 13 Fedora Update System 2015-12-14 10:22:03 UTC
heketi-1.0.2-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Fedora Update System 2015-12-14 11:51:13 UTC
heketi-1.0.2-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2015-12-30 18:04:14 UTC
heketi-1.0.2-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.