## Description of problem: I've recently upgrade from 3.6 to 4.0 (hosted engine). I'm at the part where i have one new rhvh4.0 hypervisor, and all the rest are from 3.6. I removed the old 3.6 one via the gui, re-install with rhvh4.0, then try and add it via the gui (with hosted engine) and it says: "Error while executing action: Cannot add Host. The action deploy hosted engine host is supported from cluster level 3.6" ## Version-Release number of selected component (if applicable): RHV4.0 ovirt-hosted-engine-ha Cluster and DC at 3.6 (still completing upgrade) ## How reproducible: ## Steps to Reproduce: 1. 2. 3. ## Actual results: To complete the upgrade, it seems I need to: - remove *all* hosts (except the 4.0 one), then upgrade Cluster/DC to 4.0, then upgrade hosts and add back - but what if I can't run my all my VM's on just one hypervisor? or - rolling remove/upgrade/add of hosts to 4.0, then upgrade Cluster/DC to 4.0, then remove hosts again and re-add again as 'hosted engine' hosts - it seems unnecessary to remove hosts and add them twice, just to enable hosted engine deploy. ## Expected results: Be able to add a new hosts with hosted engine enabled to a 3.6 cluster. Or, be able to enable hosted engine after a host has been added (not have to remove and add it).
Marcus, I do not understand why is this RFE? If you cannot add a 4.0 Host to an upgraded environment, then it is a bug. OR I misunderstand something in the flow. Can you please provide simple steps to reproduce?
I have tried again - but this time instead of using the RHEV gui, I tried via the cockpit interface (on the hypervisor), and it worked! Do we still support adding hosts via the RHEV gui? If not, may need to clarify this in the documentation: https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/paged/self-hosted-engine-guide/chapter-7-installing-additional-hosts-to-a-self-hosted-environment
Marcus, is your cluster compatibility level set to 3.6? That might be the issue here. We do support deploying of 4.0 hosted engine hosts using the UI, but not when 4.0 features are disabled by the compatibility mode. We will investigate if that limitation is really necessary and either remove it or update the UI to reflect it.
Hey Martin, it is set to 3.6 as I am still in the process of upgrading (from 3.6 to 4.0) I can't change it to 4.0 until I upgrade all the hosts. Only after I have upgraded all hosts (so I can upgrade cluster and DC to 4.0), then I can deploy another hosted engine host. If I only have two or thee hosts, it's not a lot of time without a second HE host. But for a large environment (40 hosts, etc), thats a lot of time with only one host capable of running HostedEngine. It just seems a bit strange that HE is already running on an upgraded hypervisor, but I can't add another upgraded HE capable hosts. Also we can do it via the cockpit gui (c#5), just not via the rhevm gui it seems. If we can't remove this restriction, perhaps we should recommend that all hypervisors be upgraded first? (before upgrading rhevm from 3.6 to 4.0)
(In reply to Marcus West from comment #9) > Hey Martin, it is set to 3.6 as I am still in the process of upgrading (from > 3.6 to 4.0) I can't change it to 4.0 until I upgrade all the hosts. Only > after I have upgraded all hosts (so I can upgrade cluster and DC to 4.0), > then I can deploy another hosted engine host. > > If I only have two or thee hosts, it's not a lot of time without a second HE > host. But for a large environment (40 hosts, etc), thats a lot of time with > only one host capable of running HostedEngine. > > It just seems a bit strange that HE is already running on an upgraded > hypervisor, but I can't add another upgraded HE capable hosts. Also we can > do it via the cockpit gui (c#5), just not via the rhevm gui it seems. > > If we can't remove this restriction, perhaps we should recommend that all > hypervisors be upgraded first? (before upgrading rhevm from 3.6 to 4.0) Have you followed the "5.4. Upgrading a RHEV-H-Based Self-Hosted Engine Environment." from https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#Upgrading_a_RHEV-H-Based_Self-Hosted_Engine_Environment?
Hello Nikolai, Yes I have. Specifically, in the outline: " - Place the high-availability agents that manage the Manager virtual machine into the global maintenance mode. - Add a new RHVH 4.0 host to your environment. - Migrate the Manager virtual machine to the new host and set as SPM. - Run the upgrade utility to upgrade the Manager. - Update the hosts. - After the Manager virtual machine and all hosts in the cluster have been updated, change the cluster compatibility version to 4.0. " For the 2nd last step "Update the hosts", this could take quite a while for a large environment (40+hosts, etc). That's long time to run without HostedEngine redundancy. Cheers, Marcus
(In reply to Marcus West from comment #11) > Hello Nikolai, > > Yes I have. Specifically, in the outline: > > " > - Place the high-availability agents that manage the Manager virtual > machine into the global maintenance mode. > - Add a new RHVH 4.0 host to your environment. > - Migrate the Manager virtual machine to the new host and set as SPM. > - Run the upgrade utility to upgrade the Manager. > - Update the hosts. > - After the Manager virtual machine and all hosts in the cluster have been > updated, change the cluster compatibility version to 4.0. > " > > For the 2nd last step "Update the hosts", this could take quite a while for > a large environment (40+hosts, etc). That's long time to run without > HostedEngine redundancy. > > > > Cheers, > > Marcus So I understand that your only issue is redundancy with at least two HE-hosts. I've also followed the same steps, upgraded OK, I agree with your point that redundancy is important for HE, hence for step "Add a new RHVH 4.0 host to your environment.", I've added two hosts, so I would have the redundancy available on my environment regardless all other hosts, which being upgraded.
This bug looks like the same as we're testing in https://bugzilla.redhat.com/show_bug.cgi?id=1420283.
(In reply to Nikolai Sednev from comment #14) > This bug looks like the same as we're testing in > https://bugzilla.redhat.com/show_bug.cgi?id=1420283. No, this is about the ability to deploy HE from UI during update to 4.0 on 3.6 compatibility on 4.0 manager to make it easier. It's not about the manager upgrade flow to 4.0.
1)Deployed clean 3.6 HE environment over NFS storage and added 2 NFS data storage domains. 2)auto-import of hosted storage was successful. Components on engine: rhevm-setup-plugin-ovirt-engine-common-3.6.10.2-0.2.el6.noarch rhevm-websocket-proxy-3.6.10.2-0.2.el6.noarch rhevm-dependencies-3.6.1-1.el6ev.noarch rhevm-spice-client-x64-cab-3.6-7.el6.noarch rhevm-setup-plugins-3.6.5-1.el6ev.noarch rhevm-sdk-python-3.6.9.1-1.el6ev.noarch rhevm-userportal-3.6.10.2-0.2.el6.noarch rhevm-image-uploader-3.6.1-2.el6ev.noarch rhevm-branding-rhev-3.6.0-10.el6ev.noarch rhevm-lib-3.6.10.2-0.2.el6.noarch rhevm-dwh-3.6.8-1.el6ev.noarch rhevm-restapi-3.6.10.2-0.2.el6.noarch rhevm-setup-base-3.6.10.2-0.2.el6.noarch rhevm-spice-client-x86-cab-3.6-7.el6.noarch rhevm-guest-agent-common-1.0.11-6.el6ev.noarch rhevm-reports-setup-3.6.5.1-1.el6ev.noarch rhevm-dbscripts-3.6.10.2-0.2.el6.noarch rhevm-3.6.10.2-0.2.el6.noarch rhevm-setup-plugin-ovirt-engine-3.6.10.2-0.2.el6.noarch rhevm-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch rhevm-extensions-api-impl-3.6.10.2-0.2.el6.noarch rhevm-log-collector-3.6.1-1.el6ev.noarch rhevm-spice-client-x86-msi-3.6-7.el6.noarch rhevm-reports-3.6.5.1-1.el6ev.noarch rhevm-webadmin-portal-3.6.10.2-0.2.el6.noarch rhevm-tools-3.6.10.2-0.2.el6.noarch rhevm-setup-plugin-websocket-proxy-3.6.10.2-0.2.el6.noarch rhevm-doc-3.6.10-1.el6ev.noarch rhevm-cli-3.6.9.0-1.el6ev.noarch rhevm-dwh-setup-3.6.8-1.el6ev.noarch rhevm-tools-backup-3.6.10.2-0.2.el6.noarch rhev-guest-tools-iso-3.6-6.el6ev.noarch rhevm-setup-plugin-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch rhevm-setup-3.6.10.2-0.2.el6.noarch rhevm-spice-client-x64-msi-3.6-7.el6.noarch rhevm-iso-uploader-3.6.0-1.el6ev.noarch rhevm-backend-3.6.10.2-0.2.el6.noarch Linux version 2.6.32-642.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Wed Apr 13 00:51:26 EDT 2016 Linux 2.6.32-642.el6.x86_64 #1 SMP Wed Apr 13 00:51:26 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.8 (Santiago) Components on hosts: rhevm-appliance-20160620.0-1.el7ev.noarch ovirt-host-deploy-1.4.1-1.el7ev.noarch rhev-release-3.6.5-3-001.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch libvirt-client-2.0.0-10.el7_3.5.x86_64 qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64 mom-0.5.6-1.el7ev.noarch vdsm-4.17.37-1.el7ev.noarch ovirt-hosted-engine-setup-1.3.7.3-1.el7ev.noarch ovirt-setup-lib-1.0.1-1.el7ev.noarch rhevm-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch ovirt-hosted-engine-ha-1.3.5.9-2.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017 Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 3)Upgraded one of my hosts to 4.1 components and then migrated HE-VM to it and made it SPM. 4.1 host's components: ovirt-setup-lib-1.1.0-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64 mom-0.5.9-1.el7ev.noarch ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-hosted-engine-ha-2.1.0.4-1.el7ev.noarch vdsm-4.19.7-1.el7ev.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 rhevm-appliance-20161130.0-1.el7ev.noarch ovirt-host-deploy-1.6.3-1.el7ev.noarch rhev-release-3.6.5-3-001.noarch ovirt-hosted-engine-setup-2.1.0.4-1.el7ev.noarch Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017 Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 4)Set to maintenance remaining 3.6 host. 5)Backed up the engine db and copied it to 4.1 host. 6)Set 4.1 host to global maintenance. 7)Installed 4.0 appliance on 4.1 host and upgraded the engine to 4.0, using upgrade-appliance tool, then updated the 4.0 engine to latest bits e.g. "yum update -y" and then "engine-setup". Components on engine: rhevm-dependencies-4.0.0-1.el7ev.noarch rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch rhevm-guest-agent-common-1.0.12-4.el7ev.noarch rhevm-4.0.7.4-0.1.el7ev.noarch rhev-guest-tools-iso-4.0-6.el7ev.noarch rhevm-setup-plugins-4.0.0.3-1.el7ev.noarch rhevm-doc-4.0.7-1.el7ev.noarch rhevm-branding-rhev-4.0.0-7.el7ev.noarch Linux version 3.10.0-514.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:24:13 EDT 2016 Linux 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 8)Set global maintenance to none on 4.1 host. 9)Added additional third hosted engine 4.0 host to verify this bug, while host cluster was still in 3.6 compatibility mode. At hosted-engine tab "The cluster compatibility version is less than 4.0. The hosted engine deployment will still work on a host with support for 4.0 and above, but might fail when only older (3.x) packages are available." was shown as expected. 4.0 host's components: ovirt-hosted-engine-setup-2.0.4.3-3.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-host-deploy-1.5.5-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch mom-0.5.9-1.el7ev.noarch vdsm-4.18.24-3.el7ev.x86_64 ovirt-hosted-engine-ha-2.0.7-2.el7ev.noarch ovirt-setup-lib-1.1.0-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch libvirt-client-2.0.0-10.el7_3.5.x86_64 ovirt-vmconsole-host-1.0.4-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64 ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016 Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 10)Successfully added fourth hosted engine 4.1 host to verify this bug, while host cluster was still in 3.6 compatibility mode. Moving to verified.
Created attachment 1263755 [details] screencast-2017-03-16_18.56.20.mkv
Hi Andrej, I can see that there was doc text on this bug and it was removed when the bug was verified. I wanted to confirm whether this bug still needed doc text? If so, could you please add the doc text back. If not, please change the doc text flag. Thanks, Megan
Hi, it was probably removed by mistake. It is back now.
Thanks Andrej. I thought that might be the case but didn't want to assume that.