Bug 1001181 - [RFE] Provide clean up script for complete cleaning the hosted engine VM installation after failed installation.
Summary: [RFE] Provide clean up script for complete cleaning the hosted engine VM inst...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Tools
Version: ---
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ovirt-4.1.0-beta
: 2.1.0
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
URL:
Whiteboard:
: 1306301 1378035 1386581 1401996 1404241 (view as bug list)
Depends On:
Blocks: 1421003
TreeView+ depends on / blocked
 
Reported: 2013-08-26 15:53 UTC by Leonid Natapov
Modified: 2019-09-12 07:49 UTC (History)
28 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-02-15 14:59:03 UTC
oVirt Team: Integration
Embargoed:
ylavi: ovirt-4.1+
sherold: Triaged+
rule-engine: exception+
gklein: testing_plan_complete+
ylavi: planning_ack+
sbonazzo: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1134873 0 urgent CLOSED Report error as 'Hosted Engine HA service are already running on this system' when first configure hosted engine 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1385401 0 unspecified CLOSED Can not start setup procedure 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1386581 0 medium CLOSED vdsm fails to start when installing hosted-engine on RHV-H after a failure 2023-09-14 03:33:00 UTC
Red Hat Bugzilla 1401996 0 unspecified CLOSED hosted-engine --deploy does not clean setup after instllation 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 2450101 0 None None None 2016-07-19 05:36:33 UTC
oVirt gerrit 48834 0 master MERGED packaging: setup: add CLI disconnect-storage option 2020-11-27 15:45:49 UTC
oVirt gerrit 67592 0 master MERGED cleanup: provide a cleanup script for leftovers 2020-11-27 15:45:25 UTC
oVirt gerrit 69909 0 master MERGED storage: add an helper method to disconnect the storage server 2020-11-27 15:45:25 UTC
oVirt gerrit 69931 0 v2.1.z MERGED storage: add an helper method to disconnect the storage server 2020-11-27 15:45:25 UTC
oVirt gerrit 69949 0 ovirt-hosted-engine-setup-2.1 MERGED packaging: setup: add CLI disconnect-storage option 2020-11-27 15:45:49 UTC
oVirt gerrit 69966 0 ovirt-hosted-engine-setup-2.1 MERGED cleanup: provide a cleanup script for leftovers 2020-11-27 15:45:24 UTC

Internal Links: 1134873 1385401 1386581 1401996

Description Leonid Natapov 2013-08-26 15:53:51 UTC
We should provide clean up script for complete cleaning the engine VM installation.

Comment 1 Sandro Bonazzola 2013-09-03 07:04:24 UTC
With complete automated cleaning we're talking about:

 - ensure that the vm is not running on this host, else migrate the VM
 - stop and disable HA services
 - remove this host from the engine
 - remove hosted engine configuration files
 - revert firewall configuration
 - revert management bridge creation
 - revert libvirt configuration
 - revert vdsm configuration
 - if this is the last host:
   - terminate the VM
   - remove all VM related files (vm.conf, volumes / images included)
   - if we have created the storage domain, clean the shared storage path

Am I missing something?

The automated cleaning is not discussed upstream on features page: http://www.ovirt.org/Features/Self_Hosted_Engine

I think this should be treated as a RFE.

Comment 2 Ofer Schreiber 2013-09-03 10:12:06 UTC
I Agree, creating such a script might take few weeks, and we're to late in the game for 3.3 for such features

Comment 8 Red Hat Bugzilla Rules Engine 2016-02-17 13:45:45 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 9 Simone Tiraboschi 2016-03-22 13:29:48 UTC
*** Bug 1306301 has been marked as a duplicate of this bug. ***

Comment 12 Yaniv Lavi 2016-11-22 12:32:30 UTC
*** Bug 1378035 has been marked as a duplicate of this bug. ***

Comment 14 Yaniv Lavi 2016-12-06 23:27:36 UTC
Do not add the require doc text flag unless you provided the doc text. Removing.

Comment 17 Yaniv Lavi 2016-12-12 10:19:45 UTC
*** Bug 1401996 has been marked as a duplicate of this bug. ***

Comment 18 Simone Tiraboschi 2016-12-13 13:25:21 UTC
*** Bug 1404241 has been marked as a duplicate of this bug. ***

Comment 20 Kobi Hakimi 2017-01-01 13:35:01 UTC
IMHO we need to add cleanup functionality also in case of success.
as we have in "engine-setup" and "engine-cleanup"
I expect to have this functionality in hosted engine like:
"hosted-engine --deploy" and "hosted-engine --cleanup"
to be able to cleanup the hosted engine vm, storage, host and additional hosts.

Do you think it should be other RFE?

Comment 21 Yaniv Lavi 2017-01-01 13:41:37 UTC
(In reply to Kobi Hakimi from comment #20)
> IMHO we need to add cleanup functionality also in case of success.
> as we have in "engine-setup" and "engine-cleanup"
> I expect to have this functionality in hosted engine like:
> "hosted-engine --deploy" and "hosted-engine --cleanup"
> to be able to cleanup the hosted engine vm, storage, host and additional
> hosts.
> 
> Do you think it should be other RFE?

No, you should first undeploy each host from UI\REST\SDK and then use this script on the last host to finish the cleanup.

Comment 22 Kobi Hakimi 2017-01-01 14:07:45 UTC
(In reply to Yaniv Dary from comment #21)
> (In reply to Kobi Hakimi from comment #20)
> > IMHO we need to add cleanup functionality also in case of success.
> > as we have in "engine-setup" and "engine-cleanup"
> > I expect to have this functionality in hosted engine like:
> > "hosted-engine --deploy" and "hosted-engine --cleanup"
> > to be able to cleanup the hosted engine vm, storage, host and additional
> > hosts.
> > 
> > Do you think it should be other RFE?
> 
> No, you should first undeploy each host from UI\REST\SDK and then use this
> script on the last host to finish the cleanup.

My suggestion is to be able to run "hosted-engine --cleanup"
then it will identify the status of vms(like "hosted-engine --vm-status")
1. in case of more than one vm it will ask to remove the rest vms(hosts with HE capabilities) - run the "hosted-engine --cleanup" in the relevant hosts.
and it will remove it inside the script (without to use manually UI/API).

2. in case of only one vm with HE capabilities it will ask the user to destroy/clean
the storage, hosted engine vm and all its dependent services 
again it will remove it inside the script (without to use manually UI/API).

Comment 23 Yaniv Lavi 2017-01-01 14:11:01 UTC
(In reply to Kobi Hakimi from comment #22)
> My suggestion is to be able to run "hosted-engine --cleanup"
> then it will identify the status of vms(like "hosted-engine --vm-status")
> 1. in case of more than one vm it will ask to remove the rest vms(hosts with
> HE capabilities) - run the "hosted-engine --cleanup" in the relevant hosts.
> and it will remove it inside the script (without to use manually UI/API).
> 
> 2. in case of only one vm with HE capabilities it will ask the user to
> destroy/clean
> the storage, hosted engine vm and all its dependent services 
> again it will remove it inside the script (without to use manually UI/API).

I don't see us doing this. As I don't see a user use case for it and it is not a common task. 
You can do this in CLI with the SDK.

Comment 24 Nikolai Sednev 2017-01-26 18:49:02 UTC
Please supply reproduction steps for this RFE and some documentation.

Comment 25 Simone Tiraboschi 2017-01-31 16:54:12 UTC
- Deploy hosted-engine setup till it starts the engine VM
- interrupt hosted-engine --deploy
- run /usr/sbin/ovirt-hosted-engine-cleanup to remove any leftover from the host
- clean the shared storage or choose a different one
- try deploying again without manually cleaning the host and without rebooting it

Comment 26 Yaniv Lavi 2017-02-06 10:42:09 UTC
*** Bug 1386581 has been marked as a duplicate of this bug. ***

Comment 27 Nikolai Sednev 2017-02-06 13:50:34 UTC
1)Deploy hosted-engine setup till it starts the engine VM:
[ INFO  ] Stage: Closing up
[ INFO  ] Creating VM
          You can now connect to the VM with the following command:
                hosted-engine --console
          You can also graphically connect to the VM from your system with the following command:
                remote-viewer vnc://puma19.scl.lab.tlv.redhat.com:5900
          Use temporary password "7832zePq" to connect to vnc console.
          Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation.
          Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info.
          If you need to reboot the VM you will need to start it manually using the command:
          hosted-engine --vm-start
          You can then set a temporary password using the command:
          hosted-engine --add-console-password
[ INFO  ] Running engine-setup on the appliance

2) Interrupt hosted-engine --deploy:
^C[ ERROR ] Failed to execute stage 'Closing up': SIG2
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170206145745.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170206144452-sj4lh8.log

3)Run /usr/sbin/ovirt-hosted-engine-cleanup to remove any leftover from the host:
# /usr/sbin/ovirt-hosted-engine-cleanup
 This will de-configure the host to run ovirt-hosted-engine-setup from scratch. 
Caution, this operation should be used with care.

Are you sure you want to proceed? [y/n]
y
  -=== Destroy hosted-engine VM ===- 
  -=== Stop HA services ===- 
  -=== Shutdown sanlock ===- 
shutdown force 1 wait 0
shutdown done 0
  -=== Disconnecting the hosted-engine storage domain ===- 
  -=== De-configure VDSM networks ===- 
  -=== Stop other services ===- 
  -=== De-configure external daemons ===- 
  -=== Removing configuration files ===- 
? /etc/init/libvirtd.conf already missing
- removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml
? /etc/ovirt-hosted-engine/answers.conf already missing
- removing /etc/ovirt-hosted-engine/hosted-engine.conf
- removing /etc/vdsm/vdsm.conf
- removing /etc/pki/vdsm/certs/cacert.pem
- removing /etc/pki/vdsm/certs/vdsmcert.pem
- removing /etc/pki/vdsm/keys/vdsmkey.pem
- removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem
- removing /etc/pki/vdsm/libvirt-spice/ca-key.pem
- removing /etc/pki/vdsm/libvirt-spice/server-cert.pem
- removing /etc/pki/vdsm/libvirt-spice/server-key.pem
? /etc/pki/CA/cacert.pem already missing
? /etc/pki/libvirt/*.pem already missing
? /etc/pki/libvirt/private/*.pem already missing
? /etc/pki/ovirt-vmconsole/*.pem already missing
- removing /var/cache/libvirt/qemu
- removing /var/run/ovirt-hosted-engine-ha/vm.conf
- removing /var/run/ovirt-hosted-engine-ha/vm.conf.20170206145727

4)Clean the shared storage or choose a different one:
rm -rf /mnt/nsednev_he_4/*

5)Try deploying again without manually cleaning the host and without rebooting it:
.
.
.
.
[ INFO  ] Hosted Engine successfully deployed


Works for me on these components on hosts:
rhvm-appliance-4.1.20170126.0-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
vdsm-4.19.4-1.el7ev.x86_64
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)


Note You need to log in before you can comment on or make changes to this bug.