Bug 1001181 - [RFE] Provide clean up script for complete cleaning the hosted engine VM installation after failed installation.
[RFE] Provide clean up script for complete cleaning the hosted engine VM inst...
Status: CLOSED CURRENTRELEASE
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Tools (Show other bugs)
---
Unspecified Unspecified
high Severity low (vote)
: ovirt-4.1.0-beta
: 2.1.0
Assigned To: Simone Tiraboschi
Nikolai Sednev
: FutureFeature, Triaged
: 1306301 1378035 1386581 1401996 1404241 (view as bug list)
Depends On:
Blocks: 1421003
  Show dependency treegraph
 
Reported: 2013-08-26 11:53 EDT by Leonid Natapov
Modified: 2017-02-15 09:59 EST (History)
28 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
This release provides a cleanup script for completely cleaning the host after a failed attempt to install a self-hosted engine.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-02-15 09:59:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ylavi: ovirt‑4.1+
sherold: Triaged+
rule-engine: exception+
gklein: testing_plan_complete+
ylavi: planning_ack+
sbonazzo: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2450101 None None None 2016-07-19 01:36 EDT
oVirt gerrit 48834 master MERGED packaging: setup: add CLI disconnect-storage option 2017-01-10 15:15 EST
oVirt gerrit 67592 master MERGED cleanup: provide a cleanup script for leftovers 2017-01-10 15:15 EST
oVirt gerrit 69909 master MERGED storage: add an helper method to disconnect the storage server 2017-01-10 09:10 EST
oVirt gerrit 69931 v2.1.z MERGED storage: add an helper method to disconnect the storage server 2017-01-11 04:55 EST
oVirt gerrit 69949 ovirt-hosted-engine-setup-2.1 MERGED packaging: setup: add CLI disconnect-storage option 2017-01-11 05:49 EST
oVirt gerrit 69966 ovirt-hosted-engine-setup-2.1 MERGED cleanup: provide a cleanup script for leftovers 2017-01-11 05:49 EST

  None (edit)
Description Leonid Natapov 2013-08-26 11:53:51 EDT
We should provide clean up script for complete cleaning the engine VM installation.
Comment 1 Sandro Bonazzola 2013-09-03 03:04:24 EDT
With complete automated cleaning we're talking about:

 - ensure that the vm is not running on this host, else migrate the VM
 - stop and disable HA services
 - remove this host from the engine
 - remove hosted engine configuration files
 - revert firewall configuration
 - revert management bridge creation
 - revert libvirt configuration
 - revert vdsm configuration
 - if this is the last host:
   - terminate the VM
   - remove all VM related files (vm.conf, volumes / images included)
   - if we have created the storage domain, clean the shared storage path

Am I missing something?

The automated cleaning is not discussed upstream on features page: http://www.ovirt.org/Features/Self_Hosted_Engine

I think this should be treated as a RFE.
Comment 2 Ofer Schreiber 2013-09-03 06:12:06 EDT
I Agree, creating such a script might take few weeks, and we're to late in the game for 3.3 for such features
Comment 8 Red Hat Bugzilla Rules Engine 2016-02-17 08:45:45 EST
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
Comment 9 Simone Tiraboschi 2016-03-22 09:29:48 EDT
*** Bug 1306301 has been marked as a duplicate of this bug. ***
Comment 12 Yaniv Lavi 2016-11-22 07:32:30 EST
*** Bug 1378035 has been marked as a duplicate of this bug. ***
Comment 14 Yaniv Lavi 2016-12-06 18:27:36 EST
Do not add the require doc text flag unless you provided the doc text. Removing.
Comment 17 Yaniv Lavi 2016-12-12 05:19:45 EST
*** Bug 1401996 has been marked as a duplicate of this bug. ***
Comment 18 Simone Tiraboschi 2016-12-13 08:25:21 EST
*** Bug 1404241 has been marked as a duplicate of this bug. ***
Comment 20 Kobi Hakimi 2017-01-01 08:35:01 EST
IMHO we need to add cleanup functionality also in case of success.
as we have in "engine-setup" and "engine-cleanup"
I expect to have this functionality in hosted engine like:
"hosted-engine --deploy" and "hosted-engine --cleanup"
to be able to cleanup the hosted engine vm, storage, host and additional hosts.

Do you think it should be other RFE?
Comment 21 Yaniv Lavi 2017-01-01 08:41:37 EST
(In reply to Kobi Hakimi from comment #20)
> IMHO we need to add cleanup functionality also in case of success.
> as we have in "engine-setup" and "engine-cleanup"
> I expect to have this functionality in hosted engine like:
> "hosted-engine --deploy" and "hosted-engine --cleanup"
> to be able to cleanup the hosted engine vm, storage, host and additional
> hosts.
> 
> Do you think it should be other RFE?

No, you should first undeploy each host from UI\REST\SDK and then use this script on the last host to finish the cleanup.
Comment 22 Kobi Hakimi 2017-01-01 09:07:45 EST
(In reply to Yaniv Dary from comment #21)
> (In reply to Kobi Hakimi from comment #20)
> > IMHO we need to add cleanup functionality also in case of success.
> > as we have in "engine-setup" and "engine-cleanup"
> > I expect to have this functionality in hosted engine like:
> > "hosted-engine --deploy" and "hosted-engine --cleanup"
> > to be able to cleanup the hosted engine vm, storage, host and additional
> > hosts.
> > 
> > Do you think it should be other RFE?
> 
> No, you should first undeploy each host from UI\REST\SDK and then use this
> script on the last host to finish the cleanup.

My suggestion is to be able to run "hosted-engine --cleanup"
then it will identify the status of vms(like "hosted-engine --vm-status")
1. in case of more than one vm it will ask to remove the rest vms(hosts with HE capabilities) - run the "hosted-engine --cleanup" in the relevant hosts.
and it will remove it inside the script (without to use manually UI/API).

2. in case of only one vm with HE capabilities it will ask the user to destroy/clean
the storage, hosted engine vm and all its dependent services 
again it will remove it inside the script (without to use manually UI/API).
Comment 23 Yaniv Lavi 2017-01-01 09:11:01 EST
(In reply to Kobi Hakimi from comment #22)
> My suggestion is to be able to run "hosted-engine --cleanup"
> then it will identify the status of vms(like "hosted-engine --vm-status")
> 1. in case of more than one vm it will ask to remove the rest vms(hosts with
> HE capabilities) - run the "hosted-engine --cleanup" in the relevant hosts.
> and it will remove it inside the script (without to use manually UI/API).
> 
> 2. in case of only one vm with HE capabilities it will ask the user to
> destroy/clean
> the storage, hosted engine vm and all its dependent services 
> again it will remove it inside the script (without to use manually UI/API).

I don't see us doing this. As I don't see a user use case for it and it is not a common task. 
You can do this in CLI with the SDK.
Comment 24 Nikolai Sednev 2017-01-26 13:49:02 EST
Please supply reproduction steps for this RFE and some documentation.
Comment 25 Simone Tiraboschi 2017-01-31 11:54:12 EST
- Deploy hosted-engine setup till it starts the engine VM
- interrupt hosted-engine --deploy
- run /usr/sbin/ovirt-hosted-engine-cleanup to remove any leftover from the host
- clean the shared storage or choose a different one
- try deploying again without manually cleaning the host and without rebooting it
Comment 26 Yaniv Lavi 2017-02-06 05:42:09 EST
*** Bug 1386581 has been marked as a duplicate of this bug. ***
Comment 27 Nikolai Sednev 2017-02-06 08:50:34 EST
1)Deploy hosted-engine setup till it starts the engine VM:
[ INFO  ] Stage: Closing up
[ INFO  ] Creating VM
          You can now connect to the VM with the following command:
                hosted-engine --console
          You can also graphically connect to the VM from your system with the following command:
                remote-viewer vnc://puma19.scl.lab.tlv.redhat.com:5900
          Use temporary password "7832zePq" to connect to vnc console.
          Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation.
          Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info.
          If you need to reboot the VM you will need to start it manually using the command:
          hosted-engine --vm-start
          You can then set a temporary password using the command:
          hosted-engine --add-console-password
[ INFO  ] Running engine-setup on the appliance

2) Interrupt hosted-engine --deploy:
^C[ ERROR ] Failed to execute stage 'Closing up': SIG2
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170206145745.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170206144452-sj4lh8.log

3)Run /usr/sbin/ovirt-hosted-engine-cleanup to remove any leftover from the host:
# /usr/sbin/ovirt-hosted-engine-cleanup
 This will de-configure the host to run ovirt-hosted-engine-setup from scratch. 
Caution, this operation should be used with care.

Are you sure you want to proceed? [y/n]
y
  -=== Destroy hosted-engine VM ===- 
  -=== Stop HA services ===- 
  -=== Shutdown sanlock ===- 
shutdown force 1 wait 0
shutdown done 0
  -=== Disconnecting the hosted-engine storage domain ===- 
  -=== De-configure VDSM networks ===- 
  -=== Stop other services ===- 
  -=== De-configure external daemons ===- 
  -=== Removing configuration files ===- 
? /etc/init/libvirtd.conf already missing
- removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml
? /etc/ovirt-hosted-engine/answers.conf already missing
- removing /etc/ovirt-hosted-engine/hosted-engine.conf
- removing /etc/vdsm/vdsm.conf
- removing /etc/pki/vdsm/certs/cacert.pem
- removing /etc/pki/vdsm/certs/vdsmcert.pem
- removing /etc/pki/vdsm/keys/vdsmkey.pem
- removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem
- removing /etc/pki/vdsm/libvirt-spice/ca-key.pem
- removing /etc/pki/vdsm/libvirt-spice/server-cert.pem
- removing /etc/pki/vdsm/libvirt-spice/server-key.pem
? /etc/pki/CA/cacert.pem already missing
? /etc/pki/libvirt/*.pem already missing
? /etc/pki/libvirt/private/*.pem already missing
? /etc/pki/ovirt-vmconsole/*.pem already missing
- removing /var/cache/libvirt/qemu
- removing /var/run/ovirt-hosted-engine-ha/vm.conf
- removing /var/run/ovirt-hosted-engine-ha/vm.conf.20170206145727

4)Clean the shared storage or choose a different one:
rm -rf /mnt/nsednev_he_4/*

5)Try deploying again without manually cleaning the host and without rebooting it:
.
.
.
.
[ INFO  ] Hosted Engine successfully deployed


Works for me on these components on hosts:
rhvm-appliance-4.1.20170126.0-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
vdsm-4.19.4-1.el7ev.x86_64
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Note You need to log in before you can comment on or make changes to this bug.