Bug 1576451

Summary:	ovirt-hosted-engine-cleanup must undefine the HostedEngine VM
Product:	[oVirt] ovirt-hosted-engine-setup	Reporter:	Polina <pagranat>
Component:	Tools	Assignee:	Simone Tiraboschi <stirabos>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Nikolai Sednev <nsednev>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.2.19	CC:	bugs, stirabos
Target Milestone:	ovirt-4.2.4	Keywords:	Triaged
Target Release:	2.2.21	Flags:	rule-engine: ovirt-4.2+
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	ovirt-hosted-engine-setup-2.2.21	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-26 08:38:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1581783

Description Polina 2018-05-09 13:33:16 UTC

Description of problem:
If the previous deploy was failed with some error, and the cleanup procedure was run, the deploy fails with error 
"ERROR The MAC address '...' is in use by another virtual machine."
Running the  <virsh -r list --all> we find that HostedEngine machine is still defined with State=" shut off"

Version-Release number of selected component (if applicable):
rhv-release-4.2.3-6-001.noarch

How reproducible:100%

Steps to Reproduce:
1. Run hosted-engine deploy(let's say it fails with error).
2. Run /usr/sbin/ovirt-hosted-engine-cleanup.
3. Run hosted-engine deploy after the error is fixed.

Actual results:
The second deploy fails with error "ERROR The MAC address '...' is in use by another virtual machine" Run on host <virsh -r list --all> returns  HostedEngine shut off.

Expected results:
HostedEngine must be undefined after cleanup.

Additional info: workaround: manually run on host<virsh undefine HostedEngine>

Comment 1 Nikolai Sednev 2018-06-04 14:37:03 UTC

[ INFO  ] changed: [localhost]
[ INFO  ] TASK [Wait for the host to be up]
^C[ ERROR ] Failed to execute stage 'Closing up': SIG2
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
^C[ ERROR ] Failed to execute stage 'Clean up': SIG2
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180604173237.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180604172012-vb0flx.log
[root@alma04 ~]# virsh -r list --all
 Id    Name                           State
----------------------------------------------------
 1     HostedEngineLocal              running

[root@alma04 ~]# ovirt-hosted-engine-cleanup
 This will de-configure the host to run ovirt-hosted-engine-setup from scratch. 
Caution, this operation should be used with care.

Are you sure you want to proceed? [y/n]
y
  -=== Destroy hosted-engine VM ===- 
You must run deploy first
  -=== Killing left-behind HostedEngine processes ===- 
error: failed to get domain 'HostedEngine'
error: Domain not found: no domain with matching name 'HostedEngine'

  -=== Stop HA services ===- 
  -=== Shutdown sanlock ===- 
shutdown force 1 wait 0
shutdown done 0
  -=== Disconnecting the hosted-engine storage domain ===- 
You must run deploy first
  -=== De-configure VDSM networks ===- 
  -=== Stop other services ===- 
  -=== De-configure external daemons ===- 
  -=== Removing configuration files ===- 
? /etc/init/libvirtd.conf already missing
- removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml
? /etc/ovirt-hosted-engine/answers.conf already missing
? /etc/ovirt-hosted-engine/hosted-engine.conf already missing
- removing /etc/vdsm/vdsm.conf
- removing /etc/pki/vdsm/certs/cacert.pem
- removing /etc/pki/vdsm/certs/vdsmcert.pem
- removing /etc/pki/vdsm/keys/vdsmkey.pem
- removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem
- removing /etc/pki/vdsm/libvirt-spice/server-cert.pem
- removing /etc/pki/vdsm/libvirt-spice/server-key.pem
- removing /etc/pki/CA/cacert.pem
- removing /etc/pki/libvirt/clientcert.pem
- removing /etc/pki/libvirt/private/clientkey.pem
? /etc/pki/ovirt-vmconsole/*.pem already missing
- removing /var/cache/libvirt/qemu
? /var/run/ovirt-hosted-engine-ha/* already missing

alma04 ~]# virsh -r list --all
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

[root@alma04 ~]# hosted-engine --deploy 

successfully deployed.

Works for me on these components:
ovirt-hosted-engine-ha-2.2.13-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch
rhvm-appliance-4.2-20180601.0.el7.noarch
Red Hat Enterprise Linux Server release 7.5 (Maipo)
Linux 3.10.0-862.3.2.el7.x86_64 #1 SMP Tue May 15 18:22:15 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Moving to verified.

Comment 2 Nikolai Sednev 2018-06-04 15:01:34 UTC

During redeployment I've failed with another error, hence opened separate bug https://bugzilla.redhat.com/show_bug.cgi?id=1585739.

Comment 3 Sandro Bonazzola 2018-06-26 08:38:10 UTC

This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.