Description of problem: As per this https://bugzilla.redhat.com/show_bug.cgi?id=1319595#c4 , direct upgrade from 3.5 el6 to 3.6 el7 is not supported . The flow is 3.5 el6 to 3.5 el7 and then upgrade to el7. This need to be mentioned in the documentation https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Self-Hosted_Engine_Guide/index.html#Upgrading_the_Self-Hosted_Engine_from_6_to_7 Version-Release number of selected component (if applicable): Red Hat Enterprise Virtualization 3.6
Great. Nijin, thanks for opening this bug. I was about to open something similar. See my comment here: https://bugzilla.redhat.com/show_bug.cgi?id=1311027#c2 We had an extended conversation about this with Simone and here is what we got: - It does not really matter when you reinstall your hosts to RHEL7, while on 3.5 or on 3.6. - If you have RHEL6 host on Hosted Engine, and especially, if you have RHEV-H host, you would have a problem, described in the root cause section here: https://access.redhat.com/solutions/2300331[1] - P.S. Our current guide suggest this https://access.redhat.com/solutions/637583 for upgrading RHEL 6 hosts to 7 instead of reinstall. After digging in this issue, I do not think we should recommend this procedure anymore for SHE. Action Items: - Review, text and fix the solution: [Eng/GSS/ QE] https://access.redhat.com/solutions/2300331 - Update docs with the correct procedure. - Ideally have Engineering fix this issue by providing el6 version of HA 1.3, to allow storage migration in a more convenient way. [1] The problem: 3.6 Hosted Engine Storage structure has changed from 3.5 to 3.6. If on RHEL7 host, updating ovirt-hosted-engine-ha package would initiate HE storage migration to a new structure. However, if this is a pre-3.6 HE and the hosts are RHEL6 based, this upgrade would not happen, since rhel6 channel does not contain those packages. However, RHEL7 would get the latest packages and will build a new storage structure. And this would not be compatible with the existing setup.
Assigning to Byron for review.
https://bugzilla.redhat.com/show_bug.cgi?id=1337641
*** Bug 1339001 has been marked as a duplicate of this bug. ***
Current best reference for upgrading 3.5 HE setup is the upgrade helper: https://access.redhat.com/labs/rhevupgradehelper/ It covers all possible options.
*** Bug 1337641 has been marked as a duplicate of this bug. ***
The decided flow for RHEL-H updates is: 3.5 RHEL-H 6 Cluster -> Single host upgrade to 3.5 RHEV-H 7 -> Reinstall cluster to 3.6 RHEL-H 7 The decided flow for RHEL-H updates is: 3.5 RHEV-H 6 Cluster -> Single host upgrade to 3.5 RHEV-H 7 -> Reinstall cluster to 3.6 RHEV-H 7 Didi, can you please provide the flow to cover this using in cluster upgrade?
I think we should not say - "Single host upgrade to 3.5 RHEV-H 7". It would never be an upgrade, but clean install in both cases. But maybe it is only semantics. :-)
*** Bug 1319595 has been marked as a duplicate of this bug. ***
I now finished testing the following flow: 1. Installed and deployed hosted-engine 3.5 on two rhel6 hosts with nfs storage 2. Moved host1 to maintenance in the web admin 3. Removed it from the engine 4. Installed on it RHEV-H 3.5 rhel7 (from a usb stick) 5. From the tui, chose hosted-engine -> Additional host setup 6. Input host ID 3 - in 3.5, can't reuse 1. If you have N hosts, input N+1. 7. Eventually it failed as expected because can't add an el7 host to an el6 cluster, prompting to retry 8. Created new cluster el7, moved host to maintenance, edited it to be in cluster el7 9. Accepted "Retry", it finished successfully 10. Upgraded it to RHEV-H 3.6: Installed on the engine rhev-hypervisor7-7.2-20160711.0.el6ev.noarch.rpm (latest QE build, most likely works also with current released build), moved to maintenance, chose Upgrade from web admin 11. (Optional) verified that it upgraded the conf to be on the shared storage with: egrep 'Upgrading|Successfully upgraded' /var/log/ovirt-hosted-engine-ha/agent.log 12. Moved host2 to maintenance and removed from engine 13. Reinstalled with rhel7.2 and hosted-engine 3.6 14. Stopped NetworkManager, ran 'hosted-engine --deploy' as additional host 15. Accepted default host ID 2, but for more hosts, reuse previous ID that this host had 16. Since it was the last host, it was successfully added to cluster Default. For more hosts, need to add to cluster el7 and Retry 17. In principle, repeat steps 12 to 16 for the rest of the hosts. I didn't have any. 17. Since the requirement is for "RHEL-H" only hosts, moved host1 to maintenance, reinstalled it with rhel7.2 and hosted-engine 3.6, ran deploy, and used the opportunity to now reuse host ID 1. 18. Upgraded the engine: set global maintenance, added 3.6 repos, yum update rhevm\*setup\*, engine-setup 19. At that point I already had the two hosts in cluster Default. In principle, move those that are not to it. 20. Set compatibility level of cluster Default to 3.6. Got a message that I need to reboot all VMs (new for 3.6.7 I think), rebooted them (set global maintenance for engine vm prior to rebooting it and none after it was up). 21. At this point OVF_STORE was created, but score was still 2400 on both hosts. Moved host1 to maintenance and rebooted it, set maintenance to none. It got score 3400, and as expected, the engine vm was shutdown, and started on this host. 22. Moved also the other host to maintenance and reboot (actually just restarted ovirt-ha-agent and ovirt-ha-broker this time, to see that it's enough). 23. Repeat step 22 for all other hosts - I didn't have any. Then verify that all settled down on score 3400. 24. To get rid of the now unused host ID 3 in 'hosted-engine --vm-status', moved one host to maintenance, stopped ovirt-ha-agent, then ran hosted-engine --clean-metadata --host-id=3, started ha-agent, and activated. Replace '3' with N+1. Verified that now 'hosted-engine --vm-status' is clean. This is very similar to the current procedure, with one important difference (except for using RHEV-H): I do all "at once" - do not upgrade hosts to 3.5/el7 (except for the RHEV-H one) and then to 3.6, but directly to 3.6/el7. This requires restarting the agent (steps 21-23), which are not needed in current procedure. This also has a side-effect of upgrading the engine only after upgrading the hosts. Can't see any real issue with this, just in case setting needinfo on Doron in case he wants to comment/review or wants me to try some other flow. BTW, not sure what Yaniv meant by "in cluster upgrade". It can't be "in cluster", because we do not allow a single cluster with both el6 and el7 hosts, no matter if rhev-h or rhel-h.
In general, I do like it more, as we discussed - much better then installing specific packages. I think this solution is much cleaner. Couple of questions on the process: (In reply to Yedidyah Bar David from comment #26) > I now finished testing the following flow: ... > 14. Stopped NetworkManager, ran 'hosted-engine --deploy' as additional host Why do we need to stop NM? IS it also in the regular instructions? .. > 20. Set compatibility level of cluster Default to 3.6. Got a message that I need to reboot all VMs (new for 3.6.7 I think), rebooted them (set global maintenance for engine vm prior to rebooting it and none after it was up). Starting 3.6.8 you would not need to restart the VMs in order to change compatibility mode. You would be able doing it in run-time. So, all the running VMs would have to get a note they should be restarted at some point (bz#1356194). .. > This is very similar to the current procedure, with one important difference > (except for using RHEV-H): I do all "at once" - do not upgrade hosts to > 3.5/el7 (except for the RHEV-H one) and then to 3.6, but directly to > 3.6/el7. This requires restarting the agent (steps 21-23), which are not > needed in current procedure. What will happen if we do not restart the engine and continue running with 2400 score until they are restarted? > This also has a side-effect of upgrading the > engine only after upgrading the hosts. Can't see any real issue with this, > just in case setting needinfo on Doron in case he wants to comment/review or > wants me to try some other flow. Why couldn't you upgrade the engine first? Answer: because you would not be able adding the 3.5 host to the environment. I see. The only challenge here is what if a customer would upgrade the manager on its own first. There is no rollback for this. I guess, if that will be the case, we can use the flow suggested in the kcs. https://access.redhat.com/solutions/2351141 > > BTW, not sure what Yaniv meant by "in cluster upgrade". It can't be "in > cluster", because we do not allow a single cluster with both el6 and el7 > hosts, no matter if rhev-h or rhel-h. Indeed, I don't know why you couldn't add it to the original cluster. I do not see anywhere where we specify host OS in 3.5 on the cluster level. However, as long as this procedure is tested and documented, I think it is ok. So, this would cover both RHEL and RHEVH upgrade flows, right? For RHEV-H it would be same, just get RHEV-H hosts instead of RHELs.
*** Bug 1358599 has been marked as a duplicate of this bug. ***
Created attachment 1182930 [details] Adding 3.5 host warning
Didi, please see the screenshot. So, the customer would see this message and will have to choose Yes to continue, right?
(In reply to Marina from comment #30) > Didi, please see the screenshot. > So, the customer would see this message and will have to choose Yes to > continue, right? No. At least I didn't see it during comment 26. The message in the screenshot is "It seems like your existing HE infrastructure was deployed with version 3.5 (or before) and never upgraded to current release.". The flow I used was specifically designed to not enter this state. It does this: 1. Start with all 3.5/el6 hosts - all good 2. Replace one with a 3.5/el7 RHEV-H - still all good 3. _Upgrade_ that one to 3.6 - Not _add_ it as a new one, but upgrade using the RHEV-H upgrade feature - This will upgrade the hosted_storage to 3.6 4. For all the others, remove and reinstall 3.6.
(In reply to Marina from comment #27) > In general, I do like it more, as we discussed - much better then installing > specific packages. I think this solution is much cleaner. > Couple of questions on the process: > > (In reply to Yedidyah Bar David from comment #26) > > I now finished testing the following flow: > ... > > 14. Stopped NetworkManager, ran 'hosted-engine --deploy' as additional host > Why do we need to stop NM? IS it also in the regular instructions? > .. Didn't check, but if you don't, it will tell you to stop it and abort. > > 20. Set compatibility level of cluster Default to 3.6. Got a message that I need to reboot all VMs (new for 3.6.7 I think), rebooted them (set global maintenance for engine vm prior to rebooting it and none after it was up). > > Starting 3.6.8 you would not need to restart the VMs in order to change > compatibility mode. You would be able doing it in run-time. So, all the > running VMs would have to get a note they should be restarted at some point > (bz#1356194). > .. Right. > > This is very similar to the current procedure, with one important difference > > (except for using RHEV-H): I do all "at once" - do not upgrade hosts to > > 3.5/el7 (except for the RHEV-H one) and then to 3.6, but directly to > > 3.6/el7. This requires restarting the agent (steps 21-23), which are not > > needed in current procedure. > What will happen if we do not restart the engine and continue running with > 2400 score until they are restarted? I didn't try that. Most likely everything will work well. Most likely some 3.6 features won't, not sure. I'd personally not consider a system fully-upgraded until all hosts have score 3400. For any other case, users should have good reasons and know what they are doing. > > > This also has a side-effect of upgrading the > > engine only after upgrading the hosts. Can't see any real issue with this, > > just in case setting needinfo on Doron in case he wants to comment/review or > > wants me to try some other flow. > Why couldn't you upgrade the engine first? Because engine-setup will tell you you can't, see bug 1311027. > Answer: because you would not be > able adding the 3.5 host to the environment. I see. > The only challenge here is what if a customer would upgrade the manager on > its own first. There is no rollback for this. > I guess, if that will be the case, we can use the flow suggested in the kcs. > https://access.redhat.com/solutions/2351141 > > > > BTW, not sure what Yaniv meant by "in cluster upgrade". It can't be "in > > cluster", because we do not allow a single cluster with both el6 and el7 > > hosts, no matter if rhev-h or rhel-h. > Indeed, I don't know why you couldn't add it to the original cluster. I do > not see anywhere where we specify host OS in 3.5 on the cluster level. We do not specify. It's just that the engine checks the OS of one host in the cluster, if it's not empty. Try this (in 3.5): 1. Create a new cluster 2. Add an el6 host - works 3. Add an el7 host - fails 4. Remove the el6 host - works 5. Add an el7 host - works 6. Add an el6 host - fails > However, as long as this procedure is tested and documented, I think it is > ok. > > So, this would cover both RHEL and RHEVH upgrade flows, right? > For RHEV-H it would be same, just get RHEV-H hosts instead of RHELs. Indeed. Didn't try with RHEV-H hosts, but can't see why it won't work.
*** Bug 1364568 has been marked as a duplicate of this bug. ***
*** Bug 1364543 has been marked as a duplicate of this bug. ***