Description of problem: Upgrading hosted-engine to 3.6 in parallel with OS upgrade from el6 to el7 is problematic. Prevent that by checking in engine-setup if we are a hosted-engine on el6 hosts and suggest to abort setup.
Verified on rhevm-setup-3.6.3.3-0.1.el6.noarch 1) Start with two hosts one RHEL6.7 and second RHEV-H 6.7 2) Deploy 3.5 HE on two hosts 3) Set global maintenance and try to update engine: [ INFO ] Cleaning stale zombie tasks and commands It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade the hosts to el7 before upgrading the engine to 3.6. Please check the log file for more details. Replying "No" will abort Setup 4) Reprovision RHEL6.7 host to RHEL7.2 and redeploy it 5) Set global maintenance and try to update engine: [ INFO ] Cleaning stale zombie tasks and commands It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade the hosts to el7 before upgrading the engine to 3.6. Please check the log file for more details. Replying "No" will abort Setup 6) Reprovision RHEV-H6.7 host to RHEV-H7.2 and redeploy it 7) Set global maintenance and try to update engine no message about el6 hosts PASS
So .. the docs were not really clear about that and we had a customer (at least one that I know off, but I am sure there are more), that have upgraded to 3.6 without upgrading the hosts to RHEL7. How bad is that? In the docs: While [1] says you must upgrade to rhel7, the link it links to [2], says it can stay in compat mode 3.5. [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine.html [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine_from_6_to_7.html
IMO, it does not matter when you reinstall your host to RHEL 7 from RHEL6. Due to the fact that the packages are available in the same channel and once on RHEL7, there is no way to separate 3.5 and 3.6 HA packages. A special procedure is needed. And this check during the setup is useless. We may just change the text to point to this kcs. (or maybe you guys would fix the repos or find another creative solution). https://access.redhat.com/solutions/2300331 Specifically: 3.6 Hosted Engine Storage structure has changed from 3.5 to 3.6. If on RHEL7 host, updating ovirt-hosted-engine-ha package would initiate HE storage migration to a new structure. However, if this is a pre-3.6 HE and the hosts are RHEL6 based, this upgrade would not happen, since rhel6 channel does not contain those packages. However, RHEL7 would get the latest packages and will build a new storage structure. And this would not be compatible with the existing setup. This is why a special workaround needed to install 3.5 ovirt-hosted-engine-ha packages on RHEL7, add it to the existing HE setup and then upgrade and trigger the storage structure change. Reopening this bug.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
(In reply to Marina from comment #2) > So .. the docs were not really clear about that and we had a customer (at > least one that I know off, but I am sure there are more), that have upgraded > to 3.6 without upgrading the hosts to RHEL7. > How bad is that? Simone can you detail here? (In reply to Marina from comment #3) > IMO, it does not matter when you reinstall your host to RHEL 7 from RHEL6. > Due to the fact that the packages are available in the same channel and once > on RHEL7, there is no way to separate 3.5 and 3.6 HA packages. You can use yum version locking for preventing the upgrade. > A special > procedure is needed. And this check during the setup is useless. We may just > change the text to point to this kcs. (or maybe you guys would fix the repos > or find another creative solution). Please open a separate bug for changing the solution. The current solution was pm approved, developed and QE tested. If something different is suggested here, it deserves the same treatment. > > https://access.redhat.com/solutions/2300331 > Specifically: > 3.6 Hosted Engine Storage structure has changed from 3.5 to 3.6. If on RHEL7 > host, updating ovirt-hosted-engine-ha package would initiate HE storage > migration to a new structure. However, if this is a pre-3.6 HE and the hosts > are RHEL6 based, this upgrade would not happen, since rhel6 channel does not > contain those packages. However, RHEL7 would get the latest packages and > will build a new storage structure. And this would not be compatible with > the existing setup. > This is why a special workaround needed to install 3.5 > ovirt-hosted-engine-ha packages on RHEL7, add it to the existing HE setup > and then upgrade and trigger the storage structure change. > > Reopening this bug.
(In reply to Sandro Bonazzola from comment #5) > (In reply to Marina from comment #2) > > So .. the docs were not really clear about that and we had a customer (at > > least one that I know off, but I am sure there are more), that have upgraded > > to 3.6 without upgrading the hosts to RHEL7. > > How bad is that? > > Simone can you detail here? I don't think that updating the engine before the host is that bad. It's not the recommended and tested path but AFAIK it should work if the use take care to set 3.5 compatibility mode for the new cluster used for el6->el7 migration. The issue is another. moving an hosted-engine host from el6 -> el7 means: 1. reinstalling it with a new OS (el7) 2. redeploying it with hosted-engine-setup In 3.5 all the configurations were local to the host, in 3.6 we moved to the shared storage to enable additional features and we an automatic upgrade procedure that copies them to the shared storage on the first upgraded host. But... the upgrade procedure need to find the configuration on the local storage in order to know what to copy to the shared storage. So the recommended way to upgrade an hosted-engine host from el6 to el7 is: 1. initial status: host with hosted-engine-ha from 3.5 (1.2) on el6 2. reinstall the host with el7 3. redeploy it with hosted-engine-setup from 3.5 on el7 (1.2); this will create the local config 4. upgrade hosted-engine-ha with rpm from 3.6 (1.3): this will move the configuration to the shared storage What we cannot do is: directly redeploying/deploying an additional el7 host with hosted-engine-setup from 3.6 (1.2) in a cluster that is still at 3.5 since it will not found the hosted-engine configuration on the shared storage and the deploy procedure will fail. This conditions is also enforced since this patch: https://gerrit.ovirt.org/#/c/54062/ hosted-engine-setup 1.3.z will refuse do deploy an additional host if the hosted-engine storage is still at 3.5 shape. So the first hosted-engine host that upgrades from el6 to el7 should necessarily do: 1. hosted-engine-setup/ha 1.2 (3.5/el6) 2. hosted-engine-setup/ha 1.2 (3.5/el7) 3. hosted-engine-setup/ha 1.3 (3.6/el7) if the upgrade went fine for the first upgrade host, other host can directly do: 1. hosted-engine-setup/ha 1.2.z (3.5/el6) 2. hosted-engine-setup/ha 1.3 (3.6/el7) Rebuilding hosted-engine-ha 1.3.z for el6 is quite an effort since it will probably also require to rebuild vdsm 4.17.z and all its dependencies for el6. What we can simply do, since hosted-engine-setup is already clearly detecting the issue, is printing an additional hint there suggesting to downgrade hosted-engine-setup/ha to 1.2.z before trying again to redeploy the first el7 host. Other hosts will be fine as is.