Bug 1535791 - Upgrading node brings back previous hosted-engine configuration
Summary: Upgrading node brings back previous hosted-engine configuration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: imgbased
Version: 4.1.8
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-4.2.1
: 4.2.0
Assignee: Ryan Barry
QA Contact: Yihui Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-18 06:12 UTC by Germano Veit Michel
Modified: 2021-03-11 19:50 UTC (History)
12 users (show)

Fixed In Version: imgbased-1.0.7
Doc Type: Bug Fix
Doc Text:
Red Hat Virtualization Host now ensures that deleted configuration files are no longer restored from previous layers when upgrading.
Clone Of:
Environment:
Last Closed: 2018-05-15 17:57:44 UTC
oVirt Team: Node
Target Upstream Version:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)
/var/log/* (691.81 KB, application/x-bzip)
2018-01-22 08:35 UTC, Yihui Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3325321 0 None None None 2018-01-18 06:23:12 UTC
Red Hat Product Errata RHSA-2018:1524 0 None None None 2018-05-15 17:58:49 UTC
oVirt gerrit 86567 0 master MERGED osupdater: don't copy extraneous files 2018-01-24 21:32:58 UTC
oVirt gerrit 86773 0 ovirt-4.2 MERGED osupdater: don't copy extraneous files 2018-01-24 21:33:11 UTC
oVirt gerrit 86774 0 ovirt-4.1 MERGED osupdater: don't copy extraneous files 2018-01-24 21:35:32 UTC

Description Germano Veit Michel 2018-01-18 06:12:37 UTC
Description of problem:

Upgrading RHV-H restores some hosted-engine configuration from previous layers. A host that was undeployed gets its deployment configuration back after upgrade, reverting the undeployment and joining the HA cluster again.

Version-Release number of selected component (if applicable):
Several, including 4.1.8 Async, see reproduction for specifics.

How reproducible:
Always

Steps to Reproduce:
1) Install host with 4.1.7 image, deploying Hosted-Engine

Results:
  IMAGE: rhvh-4.1-0.20171101.0:
  # ll /etc/ovirt-hosted-engine/hosted-engine.conf
    -rw-r--r--. 1 root root 1089 Jan 18 14:50 /etc/ovirt-hosted-engine/hosted-engine.conf
  # systemctl status ovirt-ha-agent | grep Active
    Active: active (running) since Thu 2018-01-18 14:55:49 AEST; 1min 58s ago

2) Switch host to maintenance and upgrade to 4.1.8 (not async yet), reboot.
# yum install redhat-virtualization-host-image-update-4.1-20171207.0.el7_4

Results:
  IMAGE: rhvh-4.1-0.20171207.0+1
  # ll /etc/ovirt-hosted-engine/hosted-engine.conf
    -rw-r--r--. 1 root root 1089 Jan 18 14:50 /etc/ovirt-hosted-engine/hosted-engine.conf
  # systemctl status ovirt-ha-agent | grep Active
    Active: active (running) since Thu 2018-01-18 15:27:03 AEST; 1min 38s ago

3) Undeploy Hosted Engine (Maintenance -> Re-install -> Hosted-Engine -> Undeploy)

Results:
  # ls /etc/ovirt-hosted-engine/
    hosted-engine.conf.20180118152956  virsh_auth.conf
  # systemctl status ovirt-ha-agent | grep Loaded
    Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; disabled; vendor preset: disabled)

So all good until this point. The file was moved away and the ha daemon was disabled and stopped.

Now cames the problem.

4) Upgrade to latest (4.1.8 async as of today) and reboot

Results:
  IMAGE: rhvh-4.1-0.20180102.0+1
  # ls /etc/ovirt-hosted-engine
    hosted-engine.conf  hosted-engine.conf.20180118152956  virsh_auth.conf
  # systemctl status ovirt-ha-agent | egrep 'Loaded|Active'
    Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled)
    Active: active (running) since Thu 2018-01-18 16:00:43 AEST; 1min 41s ago

At step 4, the hosted-engine.conf file should not be there and the ha services should be disabled.
So the Hosted-Engine configuration came back from the dead.

imgbase logs:

First upgrade:
2018-01-18 15:13:49,180 [DEBUG] (migrate_etc) Calling binary: (['cp', '-a', '-r', u'/tmp/mnt.l56vW///etc/systemd/system/multi-user.target.wants/ovirt-ha-agent.service', u'/tmp/mnt.gERGc///etc/systemd/system/multi-user.target.wants/ovirt-ha-agent.service'],) {}
2018-01-18 15:13:49,707 [DEBUG] (migrate_etc) Calling binary: (['cp', '-a', '-r', u'/tmp/mnt.l56vW///etc/ovirt-hosted-engine/hosted-engine.conf', u'/tmp/mnt.gERGc///etc/ovirt-hosted-engine/hosted-engine.conf'],) {}

Second upgrade:
2018-01-18 15:46:36,547 [DEBUG] (migrate_etc) Calling binary: (['cp', '-a', '-r', u'/tmp/mnt.lbXZw///etc/systemd/system/multi-user.target.wants/ovirt-ha-agent.service', u'/tmp/mnt.2zgs8///etc/systemd/system/multi-user.target.wants/ovirt-ha-agent.service'],) {}
2018-01-18 15:46:37,610 [DEBUG] (migrate_etc) Calling binary: (['cp', '-a', '-r', u'/tmp/mnt.lbXZw///etc/ovirt-hosted-engine/hosted-engine.conf', u'/tmp/mnt.2zgs8///etc/ovirt-hosted-engine/hosted-engine.conf'],) {}
2018-01-18 15:46:37,614 [DEBUG] (migrate_etc) Calling binary: (['cp', '-a', '-r', u'/tmp/mnt.lbXZw///etc/ovirt-hosted-engine/hosted-engine.conf.20180118152956', u'/tmp/mnt.2zgs8///etc/ovirt-hosted-engine/hosted-engine.conf.20180118152956'],) {}

Actual results:
HE Deployment came back

Expected results:
HE Undeployed

Additional info:
This is similar to what was reported in BZ1501047, now for hosted engine.

Comment 10 Yihui Zhao 2018-01-22 08:32:36 UTC
Can reproduce.

Additional host to the hostedengine cluster:

1. Deploy HostedEngine on one host, then add the another host to the HostedEngine cluster as the hostedengine additional host
---------------------------------------------------------------------
[root@localhost ~]# imgbase w
You are on rhvh-4.1-0.20171101.0+1


ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 22 15:12 hosted-engine.conf
-rw-------. 1 root root  103 Nov  2 04:34 virsh_auth.conf

[root@localhost ~]# systemctl status ovirt-ha-agent | grep Active
   Active: active (running) since Mon 2018-01-22 15:12:45 CST; 32min ago
---------------------------------------------------------------------


2. Upgrade to 4.1.8 (not async yet), reboot.
---------------------------------------------------------------------
[root@hp-dl385pg8-11 ~]# imgbase w
You are on rhvh-4.1-0.20171207.0+1


[root@hp-dl385pg8-11 ~]# ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 22 15:12 hosted-engine.conf
-rw-------. 1 root root  103 Dec  8 04:16 virsh_auth.conf

[root@hp-dl385pg8-11 ~]# systemctl status ovirt-ha-agent | grep Active
   Active: active (running) since Mon 2018-01-22 16:00:24 CST; 1min 18s ago

----------------------------------------------------------------------------



3. Undeploy Hosted Engine (Maintenance -> Re-install -> Hosted-Engine -> Undeploy)
-----------------------------------------------------------------------
ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 22 15:12 hosted-engine.conf.20180122160414
-rw-------. 1 root root  103 Dec  8 04:16 virsh_auth.conf


# systemctl status ovirt-ha-agent | grep Active
   Active: failed (Result: exit-code) since Mon 2018-01-22 16:05:41 CST; 1min 21s ago


# systemctl status ovirt-ha-agent | grep Loaded
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; disabled; vendor preset: disabled)
---------------------------------------------------------------------------

4. Upgrade to latest (4.1.8 async as of today) and reboot
--------------------------------------------------------------------------
[root@hp-dl385pg8-11 ~]# imgbase w
You are on rhvh-4.1-0.20180102.0+1

[root@hp-dl385pg8-11 ~]# ll /etc/ovirt-hosted-engine
total 12
-rw-r--r--. 1 root root 1039 Jan 22 16:17 hosted-engine.conf
-rw-r--r--. 1 root root 1039 Jan 22 15:12 hosted-engine.conf.20180122160414
-rw-------. 1 root root  103 Jan  3 07:17 virsh_auth.conf


[root@hp-dl385pg8-11 ~]# systemctl status ovirt-ha-agent | egrep 'Loaded|Active'
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-01-22 16:29:35 CST; 29s ago
------------------------------------------------------------------------------

Comment 11 Yihui Zhao 2018-01-22 08:35:37 UTC
Created attachment 1384248 [details]
/var/log/*

Comment 12 Yihui Zhao 2018-01-30 07:12:13 UTC
The bug was fixed.  Here is the verified process.

Additional host to the hostedengine cluster:

1. Deploy HostedEngine on one host, then add the another host to the HostedEngine cluster as the hostedengine additional host
---------------------------------------------------------------------
[root@hp-dl385pg8-11 ~]# imgbase w
You are on rhvh-4.1-0.20171101.0+1


ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 30 14:00 hosted-engine.conf
-rw-------. 1 root root  103 Nov  2 04:34 virsh_auth.conf


[root@hp-dl385pg8-11 ~]# systemctl status ovirt-ha-agent | grep Active
   Active: active (running) since Tue 2018-01-30 14:00:46 CST; 2min 19s ago

---------------------------------------------------------------------



2. Upgrade to 4.1.8 (not async yet), reboot.
---------------------------------------------------------------------
[root@hp-dl385pg8-11 ~]# imgbase w
You are on rhvh-4.1-0.20171207.0+1

[root@hp-dl385pg8-11 ~]# ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 30 14:00 hosted-engine.conf
-rw-------. 1 root root  103 Dec  8 04:16 virsh_auth.conf


[root@hp-dl385pg8-11 ~]#  systemctl status ovirt-ha-agent | grep Active
   Active: active (running) since Tue 2018-01-30 14:22:36 CST; 2min 56s ago

----------------------------------------------------------------------------



3. Undeploy Hosted Engine (Maintenance -> Re-install -> Hosted-Engine -> Undeploy)
-----------------------------------------------------------------------
[root@hp-dl385pg8-11 ~]# ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 30 14:00 hosted-engine.conf.20180130142716
-rw-------. 1 root root  103 Dec  8 04:16 virsh_auth.conf



[root@hp-dl385pg8-11 ~]# systemctl status ovirt-ha-agent | grep Active
   Active: failed (Result: exit-code) since Tue 2018-01-30 14:28:07 CST; 1min 18s ago



[root@hp-dl385pg8-11 ~]# systemctl status ovirt-ha-agent | grep Loaded
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; disabled; vendor preset: disabled)

---------------------------------------------------------------------------

4. Upgrade to latest (latest 4.2.1) and reboot

--------------------------------------------------------------------------
[root@hp-dl385pg8-11 ~]# imgbase w
You are on rhvh-4.2.1.2-0.20180126.0+1


[root@hp-dl385pg8-11 ~]# ll /etc/ovirt-hosted-engine
total 8
-rw-r--r--. 1 root root 1039 Jan 30 14:00 hosted-engine.conf.20180130142716
-rw-------. 1 root root  103 Jan 26 20:16 virsh_auth.conf



[root@hp-dl385pg8-11 ~]# systemctl status ovirt-ha-agent | egrep 'Loaded|Active'
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

-----------------------------------------------------------------------------





So, change ths bug's status to verified.

Comment 15 errata-xmlrpc 2018-05-15 17:57:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1524

Comment 16 Franta Kust 2019-05-16 13:09:26 UTC
BZ<2>Jira Resync

Comment 17 Daniel Gur 2019-08-28 13:15:12 UTC
sync2jira

Comment 18 Daniel Gur 2019-08-28 13:20:14 UTC
sync2jira


Note You need to log in before you can comment on or make changes to this bug.