Bug 1383156

Summary: Enable HE deploy option in 3.6 cluster compatibility and add note in hosted engine tab that it will only work in host that are 4.0 and above.
Product: Red Hat Enterprise Virtualization Manager Reporter: Marcus West <mwest>
Component: ovirt-hosted-engine-haAssignee: Andrej Krejcir <akrejcir>
Status: CLOSED ERRATA QA Contact: Nikolai Sednev <nsednev>
Severity: medium Docs Contact:
Priority: high    
Version: 4.0.3CC: akrejcir, dfediuck, lsurette, melewis, mgoldboi, mkalinin, msivak, mwest, ykaul, ylavi
Target Milestone: ovirt-4.1.2Keywords: Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, deploying self-hosted engine hosts through Administration Portal was not allowed if the cluster compatibility level was earlier than Red Hat Virtualization 4.0. This was because the self-hosted engine agent for Red Hat Enterprise Virtualization 3.6 did not support it. It was also not possible to upgrade a self-hosted engine cluster from Red Hat Enterprise Virtualization 3.6 to Red Hat Virtualization 4.0. During upgrade the cluster compatibility level was required to be set as Red Hat Enterprise Virtualization 3.6 because the cluster contained some Red Hat Enterprise Virtualization 3.6 hosts. This meant that deploying the self-hosted engine on an upgraded Red Hat Virtualization 4.0 host was only possible through the Administration Portal but at the same time the deployment was disabled due to compatibility issues. Now, the deployment of self-hosted engine hosts is allowed for hosts with a cluster compatibility level earlier than Red Hat Virtualization 4.0 but a warning is displayed advising the user that the deployment should only be performed on hosts with repositories for Red Hat Virtualization 4.0 or later. This means that it is now possible to upgrade a self-hosted engine cluster to Red Hat Virtualization 4.0.
 Story Points: ---
Clone Of:
: 1425892 (view as bug list) Environment:
Last Closed: 2017-05-25 06:48:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1425892    
Attachments:
Description Flags
screencast-2017-03-16_18.56.20.mkv none

Description Marcus West 2016-10-10 05:08:46 UTC 
## Description of problem:

I've recently upgrade from 3.6 to 4.0 (hosted engine).  I'm at the part where i have one new rhvh4.0 hypervisor, and all the rest are from 3.6.  I removed the old 3.6 one via the gui, re-install with rhvh4.0, then try and add it via the gui (with hosted engine) and it says:

"Error while executing action: Cannot add Host.  The action deploy hosted engine host is supported from cluster level 3.6"

## Version-Release number of selected component (if applicable):

RHV4.0
ovirt-hosted-engine-ha
Cluster and DC at 3.6 (still completing upgrade)

## How reproducible:


## Steps to Reproduce:
1.
2.
3.

## Actual results:

To complete the upgrade, it seems I need to:

- remove *all* hosts (except the 4.0 one), then upgrade Cluster/DC to 4.0, then upgrade hosts and add back - but what if I can't run my all my VM's on just one hypervisor?

or 

- rolling remove/upgrade/add of hosts to 4.0, then upgrade Cluster/DC to 4.0, then remove hosts again and re-add again as 'hosted engine' hosts - it seems unnecessary to remove hosts and add them twice, just to enable hosted engine deploy.


## Expected results:

Be able to add a new hosts with hosted engine enabled to a 3.6 cluster.  Or, be able to enable hosted engine after a host has been added (not have to remove and add it).
Comment 2 Marina Kalinin 2016-11-07 04:31:47 UTC 
Marcus, I do not understand why is this RFE?
If you cannot add a 4.0 Host to an upgraded environment, then it is a bug. OR I misunderstand something in the flow.

Can you please provide simple steps to reproduce?
Comment 5 Marcus West 2016-11-09 06:51:52 UTC 
I have tried again - but this time instead of using the RHEV gui, I tried via the cockpit interface (on the hypervisor), and it worked!

Do we still support adding hosts via the RHEV gui?  If not, may need to clarify this in the documentation:

https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/paged/self-hosted-engine-guide/chapter-7-installing-additional-hosts-to-a-self-hosted-environment
Comment 8 Martin Sivák 2016-11-23 11:47:16 UTC 
Marcus, is your cluster compatibility level set to 3.6? That might be the issue here. We do support deploying of 4.0 hosted engine hosts using the UI, but not when 4.0 features are disabled by the compatibility mode.

We will investigate if that limitation is really necessary and either remove it or update the UI to reflect it.
Comment 9 Marcus West 2017-01-09 23:25:27 UTC 
Hey Martin, it is set to 3.6 as I am still in the process of upgrading (from 3.6 to 4.0)  I can't change it to 4.0 until I upgrade all the hosts.  Only after I have upgraded all hosts (so I can upgrade cluster and DC to 4.0), then I can deploy another hosted engine host.

If I only have two or thee hosts, it's not a lot of time without a second HE host.  But for a large environment (40 hosts, etc), thats a lot of time with only one host capable of running HostedEngine.

It just seems a bit strange that HE is already running on an upgraded hypervisor, but I can't add another upgraded HE capable hosts.  Also we can do it via the cockpit gui (c#5), just not via the rhevm gui it seems.

If we can't remove this restriction, perhaps we should recommend that all hypervisors be upgraded first? (before upgrading rhevm from 3.6 to 4.0)
Comment 10 Nikolai Sednev 2017-01-15 13:53:10 UTC 
(In reply to Marcus West from comment #9)
> Hey Martin, it is set to 3.6 as I am still in the process of upgrading (from
> 3.6 to 4.0)  I can't change it to 4.0 until I upgrade all the hosts.  Only
> after I have upgraded all hosts (so I can upgrade cluster and DC to 4.0),
> then I can deploy another hosted engine host.
> 
> If I only have two or thee hosts, it's not a lot of time without a second HE
> host.  But for a large environment (40 hosts, etc), thats a lot of time with
> only one host capable of running HostedEngine.
> 
> It just seems a bit strange that HE is already running on an upgraded
> hypervisor, but I can't add another upgraded HE capable hosts.  Also we can
> do it via the cockpit gui (c#5), just not via the rhevm gui it seems.
> 
> If we can't remove this restriction, perhaps we should recommend that all
> hypervisors be upgraded first? (before upgrading rhevm from 3.6 to 4.0)

Have you followed the "5.4. Upgrading a RHEV-H-Based Self-Hosted Engine Environment." from https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#Upgrading_a_RHEV-H-Based_Self-Hosted_Engine_Environment?
Comment 11 Marcus West 2017-01-16 05:32:33 UTC 
Hello Nikolai,

Yes I have.  Specifically, in the outline:

"
 - Place the high-availability agents that manage the Manager virtual machine into the global maintenance mode.
 - Add a new RHVH 4.0 host to your environment.
 - Migrate the Manager virtual machine to the new host and set as SPM.
 - Run the upgrade utility to upgrade the Manager.
 - Update the hosts.
 - After the Manager virtual machine and all hosts in the cluster have been updated, change the cluster compatibility version to 4.0.
"

For the 2nd last step "Update the hosts", this could take quite a while for a large environment (40+hosts, etc).  That's  long time to run without HostedEngine redundancy.



Cheers,

Marcus
Comment 12 Nikolai Sednev 2017-01-16 10:07:26 UTC 
(In reply to Marcus West from comment #11)
> Hello Nikolai,
> 
> Yes I have.  Specifically, in the outline:
> 
> "
>  - Place the high-availability agents that manage the Manager virtual
> machine into the global maintenance mode.
>  - Add a new RHVH 4.0 host to your environment.
>  - Migrate the Manager virtual machine to the new host and set as SPM.
>  - Run the upgrade utility to upgrade the Manager.
>  - Update the hosts.
>  - After the Manager virtual machine and all hosts in the cluster have been
> updated, change the cluster compatibility version to 4.0.
> "
> 
> For the 2nd last step "Update the hosts", this could take quite a while for
> a large environment (40+hosts, etc).  That's  long time to run without
> HostedEngine redundancy.
> 
> 
> 
> Cheers,
> 
> Marcus
So I understand that your only issue is redundancy with at least two HE-hosts.
 
I've also followed the same steps, upgraded OK, I agree with your point that redundancy is important for HE, hence for step "Add a new RHVH 4.0 host to your environment.", I've added two hosts, so I would have the redundancy available on my environment regardless all other hosts, which being upgraded.
Comment 14 Nikolai Sednev 2017-02-15 13:59:10 UTC 
This bug looks like the same as we're testing in https://bugzilla.redhat.com/show_bug.cgi?id=1420283.
Comment 16 Yaniv Lavi 2017-02-27 13:59:20 UTC 
(In reply to Nikolai Sednev from comment #14)
> This bug looks like the same as we're testing in
> https://bugzilla.redhat.com/show_bug.cgi?id=1420283.

No, this is about the ability to deploy HE from UI during update to 4.0 on 3.6 compatibility on 4.0 manager to make it easier. It's not about the manager upgrade flow to 4.0.
Comment 18 Nikolai Sednev 2017-03-16 17:03:27 UTC 
1)Deployed clean 3.6 HE environment over NFS storage and added 2 NFS data storage domains.
2)auto-import of hosted storage was successful.
Components on engine:
rhevm-setup-plugin-ovirt-engine-common-3.6.10.2-0.2.el6.noarch
rhevm-websocket-proxy-3.6.10.2-0.2.el6.noarch
rhevm-dependencies-3.6.1-1.el6ev.noarch
rhevm-spice-client-x64-cab-3.6-7.el6.noarch
rhevm-setup-plugins-3.6.5-1.el6ev.noarch
rhevm-sdk-python-3.6.9.1-1.el6ev.noarch
rhevm-userportal-3.6.10.2-0.2.el6.noarch
rhevm-image-uploader-3.6.1-2.el6ev.noarch
rhevm-branding-rhev-3.6.0-10.el6ev.noarch
rhevm-lib-3.6.10.2-0.2.el6.noarch
rhevm-dwh-3.6.8-1.el6ev.noarch
rhevm-restapi-3.6.10.2-0.2.el6.noarch
rhevm-setup-base-3.6.10.2-0.2.el6.noarch
rhevm-spice-client-x86-cab-3.6-7.el6.noarch
rhevm-guest-agent-common-1.0.11-6.el6ev.noarch
rhevm-reports-setup-3.6.5.1-1.el6ev.noarch
rhevm-dbscripts-3.6.10.2-0.2.el6.noarch
rhevm-3.6.10.2-0.2.el6.noarch
rhevm-setup-plugin-ovirt-engine-3.6.10.2-0.2.el6.noarch
rhevm-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch
rhevm-extensions-api-impl-3.6.10.2-0.2.el6.noarch
rhevm-log-collector-3.6.1-1.el6ev.noarch
rhevm-spice-client-x86-msi-3.6-7.el6.noarch
rhevm-reports-3.6.5.1-1.el6ev.noarch
rhevm-webadmin-portal-3.6.10.2-0.2.el6.noarch
rhevm-tools-3.6.10.2-0.2.el6.noarch
rhevm-setup-plugin-websocket-proxy-3.6.10.2-0.2.el6.noarch
rhevm-doc-3.6.10-1.el6ev.noarch
rhevm-cli-3.6.9.0-1.el6ev.noarch
rhevm-dwh-setup-3.6.8-1.el6ev.noarch
rhevm-tools-backup-3.6.10.2-0.2.el6.noarch
rhev-guest-tools-iso-3.6-6.el6ev.noarch
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch
rhevm-setup-3.6.10.2-0.2.el6.noarch
rhevm-spice-client-x64-msi-3.6-7.el6.noarch
rhevm-iso-uploader-3.6.0-1.el6ev.noarch
rhevm-backend-3.6.10.2-0.2.el6.noarch
Linux version 2.6.32-642.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Wed Apr 13 00:51:26 EDT 2016
Linux 2.6.32-642.el6.x86_64 #1 SMP Wed Apr 13 00:51:26 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Components on hosts:
rhevm-appliance-20160620.0-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
rhev-release-3.6.5-3-001.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
mom-0.5.6-1.el7ev.noarch
vdsm-4.17.37-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.3-1.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.9-2.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017
Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

3)Upgraded one of my hosts to 4.1 components and then migrated HE-VM to it and made it SPM.
4.1 host's components:
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
mom-0.5.9-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-hosted-engine-ha-2.1.0.4-1.el7ev.noarch
vdsm-4.19.7-1.el7ev.x86_64
libvirt-client-2.0.0-10.el7_3.5.x86_64
rhevm-appliance-20161130.0-1.el7ev.noarch
ovirt-host-deploy-1.6.3-1.el7ev.noarch
rhev-release-3.6.5-3-001.noarch
ovirt-hosted-engine-setup-2.1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017
Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

4)Set to maintenance remaining 3.6 host.
5)Backed up the engine db and copied it to 4.1 host.
6)Set 4.1 host to global maintenance.
7)Installed 4.0 appliance on 4.1 host and upgraded the engine to 4.0, using upgrade-appliance tool, then updated the 4.0 engine to latest bits e.g. "yum update -y" and then "engine-setup".
Components on engine:
rhevm-dependencies-4.0.0-1.el7ev.noarch
rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch
rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch
rhevm-guest-agent-common-1.0.12-4.el7ev.noarch
rhevm-4.0.7.4-0.1.el7ev.noarch
rhev-guest-tools-iso-4.0-6.el7ev.noarch
rhevm-setup-plugins-4.0.0.3-1.el7ev.noarch
rhevm-doc-4.0.7-1.el7ev.noarch
rhevm-branding-rhev-4.0.0-7.el7ev.noarch
Linux version 3.10.0-514.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:24:13 EDT 2016
Linux 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

8)Set global maintenance to none on 4.1 host.
9)Added additional third hosted engine 4.0 host to verify this bug, while host cluster was still in 3.6 compatibility mode. At hosted-engine tab "The cluster compatibility version is less than 4.0. The hosted engine deployment will still work on a host with support for 4.0 and above, but might fail when only older (3.x) packages are available." was shown as expected.
4.0 host's components:
ovirt-hosted-engine-setup-2.0.4.3-3.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-host-deploy-1.5.5-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
mom-0.5.9-1.el7ev.noarch
vdsm-4.18.24-3.el7ev.x86_64
ovirt-hosted-engine-ha-2.0.7-2.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

10)Successfully added fourth hosted engine 4.1 host to verify this bug, while host cluster was still in 3.6 compatibility mode.

Moving to verified.
Comment 19 Nikolai Sednev 2017-03-16 17:05:00 UTC 
Created attachment 1263755 [details]
screencast-2017-03-16_18.56.20.mkv
Comment 20 Megan Lewis 2017-04-05 06:09:31 UTC 
Hi Andrej, 

I can see that there was doc text on this bug and it was removed when the bug was verified. I wanted to confirm whether this bug still needed doc text? If so, could you please add the doc text back. If not, please change the doc text flag. 

Thanks, 
Megan
Comment 21 Andrej Krejcir 2017-04-05 08:32:52 UTC 
Hi, 

it was probably removed by mistake.
It is back now.
Comment 22 Megan Lewis 2017-04-10 02:18:26 UTC 
Thanks Andrej. I thought that might be the case but didn't want to assume that.