Bug 1311027 - Prevent upgrade of engine to 3.6 if it's a hosted-engine on el6 hosts
Prevent upgrade of engine to 3.6 if it's a hosted-engine on el6 hosts
Status: CLOSED CURRENTRELEASE
Product: ovirt-engine
Classification: oVirt
Component: Setup.Engine (Show other bugs)
3.6.1
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-3.6.3
: 3.6.3.3
Assigned To: Yedidyah Bar David
Artyom
: Reopened, Triaged
Depends On:
Blocks: 1395357
  Show dependency treegraph
 
Reported: 2016-02-23 03:23 EST by Yedidyah Bar David
Modified: 2017-05-11 05:26 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: 3.6 supports only el7 hosts. Cross-cluster-migration from el6 hosts to el7 hosts is supported only on 3.5. Consequence: It's not possible (easily, at least) to upgrade hosted-engine hosts from el6 to el7 after starting upgrading to 3.6. Fix: engine-setup of 3.6 now checks if it's running as a hosted-engine on el6 hosts, and if so, suggests to first upgrade them to el7. Result: The upgrade process can be finished successfully. Note that this is also tracked in bug 1188679. Feel free to drop current from RN and/or write a very short text only mentioning engine-setup (mainly so that people that stumble upon this when running it will be less surprized).
Story Points: ---
Clone Of:
: 1395357 (view as bug list)
Environment:
Last Closed: 2016-05-06 01:20:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.6.z+
rule-engine: exception+
rule-engine: planning_ack+
sbonazzo: devel_ack+
gklein: testing_ack+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 53871 master NEW packaging: setup: check if self-hosted-engine on el6 2016-02-23 03:27 EST
oVirt gerrit 53876 ovirt-engine-3.6 MERGED packaging: setup: check if self-hosted-engine on el6 2016-02-23 04:59 EST
oVirt gerrit 53878 ovirt-engine-3.6.3 MERGED packaging: setup: check if self-hosted-engine on el6 2016-02-23 05:13 EST
oVirt gerrit 66775 ovirt-engine-3.6 POST packaging: setup: more severe el6 hosts check 2016-11-21 05:21 EST

  None (edit)
Description Yedidyah Bar David 2016-02-23 03:23:40 EST
Description of problem:

Upgrading hosted-engine to 3.6 in parallel with OS upgrade from el6 to el7 is problematic.

Prevent that by checking in engine-setup if we are a hosted-engine on el6 hosts and suggest to abort setup.
Comment 1 Artyom 2016-02-28 10:38:26 EST
Verified on rhevm-setup-3.6.3.3-0.1.el6.noarch
1) Start with two hosts one RHEL6.7 and second RHEV-H 6.7
2) Deploy 3.5 HE on two hosts
3) Set global maintenance and try to update engine:
[ INFO  ] Cleaning stale zombie tasks and commands
          It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade the hosts to el7 before upgrading the engine to 3.6.
          Please check the log file for more details.
          Replying "No" will abort Setup
4) Reprovision RHEL6.7 host to RHEL7.2 and redeploy it
5) Set global maintenance and try to update engine:
[ INFO  ] Cleaning stale zombie tasks and commands
          It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade the hosts to el7 before upgrading the engine to 3.6.
          Please check the log file for more details.
          Replying "No" will abort Setup
6) Reprovision RHEV-H6.7 host to RHEV-H7.2 and redeploy it
7) Set global maintenance and try to update engine no message about el6 hosts
PASS
Comment 2 Marina 2016-05-05 11:03:03 EDT
So .. the docs were not really clear about that and we had a customer (at least one that I know off, but I am sure there are more), that have upgraded to 3.6 without upgrading the hosts to RHEL7.
How bad is that?

In the docs:
While [1] says you must upgrade to rhel7, the link it links to [2], says it can stay in compat mode 3.5.



[1]
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine.html

[2]
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine_from_6_to_7.html
Comment 3 Marina 2016-05-05 18:25:14 EDT
IMO, it does not matter when you reinstall your host to RHEL 7 from RHEL6.
Due to the fact that the packages are available in the same channel and once on RHEL7, there is no way to separate 3.5 and 3.6 HA packages. A special procedure is needed. And this check during the setup is useless. We may just change the text to point to this kcs. (or maybe you guys would fix the repos or find another creative solution).

https://access.redhat.com/solutions/2300331
Specifically:
3.6 Hosted Engine Storage structure has changed from 3.5 to 3.6. If on RHEL7 host, updating ovirt-hosted-engine-ha package would initiate HE storage migration to a new structure. However, if this is a pre-3.6 HE and the hosts are RHEL6 based, this upgrade would not happen, since rhel6 channel does not contain those packages. However, RHEL7 would get the latest packages and will build a new storage structure. And this would not be compatible with the existing setup.
This is why a special workaround needed to install 3.5 ovirt-hosted-engine-ha packages on RHEL7, add it to the existing HE setup and then upgrade and trigger the storage structure change.

Reopening this bug.
Comment 4 Red Hat Bugzilla Rules Engine 2016-05-05 18:25:19 EDT
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 5 Sandro Bonazzola 2016-05-06 01:20:04 EDT
(In reply to Marina from comment #2)
> So .. the docs were not really clear about that and we had a customer (at
> least one that I know off, but I am sure there are more), that have upgraded
> to 3.6 without upgrading the hosts to RHEL7.
> How bad is that?

Simone can you detail here?




(In reply to Marina from comment #3)
> IMO, it does not matter when you reinstall your host to RHEL 7 from RHEL6.
> Due to the fact that the packages are available in the same channel and once
> on RHEL7, there is no way to separate 3.5 and 3.6 HA packages.

You can use yum version locking for preventing the upgrade.

> A special
> procedure is needed. And this check during the setup is useless. We may just
> change the text to point to this kcs. (or maybe you guys would fix the repos
> or find another creative solution).

Please open a separate bug for changing the solution.
The current solution was pm approved, developed and QE tested.
If something different is suggested here, it deserves the same treatment.

> 
> https://access.redhat.com/solutions/2300331
> Specifically:
> 3.6 Hosted Engine Storage structure has changed from 3.5 to 3.6. If on RHEL7
> host, updating ovirt-hosted-engine-ha package would initiate HE storage
> migration to a new structure. However, if this is a pre-3.6 HE and the hosts
> are RHEL6 based, this upgrade would not happen, since rhel6 channel does not
> contain those packages. However, RHEL7 would get the latest packages and
> will build a new storage structure. And this would not be compatible with
> the existing setup.
> This is why a special workaround needed to install 3.5
> ovirt-hosted-engine-ha packages on RHEL7, add it to the existing HE setup
> and then upgrade and trigger the storage structure change.
> 
> Reopening this bug.
Comment 6 Simone Tiraboschi 2016-05-06 05:26:16 EDT
(In reply to Sandro Bonazzola from comment #5)
> (In reply to Marina from comment #2)
> > So .. the docs were not really clear about that and we had a customer (at
> > least one that I know off, but I am sure there are more), that have upgraded
> > to 3.6 without upgrading the hosts to RHEL7.
> > How bad is that?
> 
> Simone can you detail here?

I don't think that updating the engine before the host is that bad. It's not the recommended and tested path but AFAIK it should work if the use take care to set 3.5 compatibility mode for the new cluster used for el6->el7 migration.

The issue is another.
moving an hosted-engine host from el6 -> el7 means:
1. reinstalling it with a new OS (el7)
2. redeploying it with hosted-engine-setup

In 3.5 all the configurations were local to the host, in 3.6 we moved to the shared storage to enable additional features and we an automatic upgrade procedure that copies them to the shared storage on the first upgraded host.

But...
the upgrade procedure need to find the configuration on the local storage in order to know what to copy to the shared storage.

So the recommended way to upgrade an hosted-engine host from el6 to el7 is:
1. initial status: host with hosted-engine-ha from 3.5 (1.2) on el6
2. reinstall the host with el7
3. redeploy it with hosted-engine-setup from 3.5 on el7 (1.2); this will create the local config
4. upgrade hosted-engine-ha with rpm from 3.6 (1.3): this will move the configuration to the shared storage

What we cannot do is:
directly redeploying/deploying an additional el7 host with hosted-engine-setup from 3.6 (1.2) in a cluster that is still at 3.5 since it will not found the hosted-engine configuration on the shared storage and the deploy procedure will fail.
This conditions is also enforced since this patch: https://gerrit.ovirt.org/#/c/54062/
hosted-engine-setup 1.3.z will refuse do deploy an additional host if the hosted-engine storage is still at 3.5 shape.

So the first hosted-engine host that upgrades from el6 to el7 should necessarily do:
1. hosted-engine-setup/ha 1.2 (3.5/el6)
2. hosted-engine-setup/ha 1.2 (3.5/el7)
3. hosted-engine-setup/ha 1.3 (3.6/el7)

if the upgrade went fine for the first upgrade host, other host can directly do:
1. hosted-engine-setup/ha 1.2.z (3.5/el6)
2. hosted-engine-setup/ha 1.3 (3.6/el7)

Rebuilding hosted-engine-ha 1.3.z for el6 is quite an effort since it will probably also require to rebuild vdsm 4.17.z and all its dependencies for el6.

What we can simply do, since hosted-engine-setup is already clearly detecting the issue, is printing an additional hint there suggesting to downgrade hosted-engine-setup/ha to 1.2.z before trying again to redeploy the first el7 host.
Other hosts will be fine as is.

Note You need to log in before you can comment on or make changes to this bug.