Bug 1351533 - Cant upgrade to new cluster version when HE VM is running in it
Summary: Cant upgrade to new cluster version when HE VM is running in it
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.HostedEngine
Version: 4.0.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.0.2
: 4.0.2.1
Assignee: Roy Golan
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks: 1356127
TreeView+ depends on / blocked
 
Reported: 2016-06-30 09:05 UTC by Roy Golan
Modified: 2017-05-11 09:24 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1356127 (view as bug list)
Environment:
Last Closed: 2016-08-12 14:25:10 UTC
oVirt Team: SLA
Embargoed:
nsednev: needinfo-
rule-engine: ovirt-4.0.z+
mgoldboi: blocker+
rule-engine: planning_ack+
rgolan: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 60912 0 master MERGED core: Support migrating the HE VM out of cluster 2016-07-26 09:47:25 UTC
oVirt gerrit 61359 0 ovirt-engine-4.0 MERGED core: Support migrating the HE VM out of cluster 2016-07-26 09:58:56 UTC
oVirt gerrit 61360 0 ovirt-engine-4.0.2 MERGED core: Support migrating the HE VM out of cluster 2016-07-26 09:49:41 UTC
oVirt gerrit 62425 0 ovirt-engine-3.6 MERGED core: Support migrating the HE VM out of cluster 2016-09-06 07:17:13 UTC

Description Roy Golan 2016-06-30 09:05:56 UTC
Description of problem:
The cluster of the HE VM can't have its compat level updated from 3.6 to 4.0, because modifying the HE VM cluster is a blocked operation. So eventhough technically the HE VM could be started or migrated out-of-cluster (right click, migrate, advanced, choose your 4.0 cluster), the underlying ChangeVmCluster action is blocked specifically

Version-Release number of selected component (if applicable):
all engine versions

How reproducible:
100%

Steps to Reproduce:
1. have 3.6 cluster which the new 4.0 HE VM is running in
2. create new 4.0 cluster, deploy and additinonal HE host there
3. right click HE-VM, migrate -> advanced -> choose new 4.0 cluster

Actual results:
VM migrates, but its cluster will show its still the old one
in the logs it will fail with:

```log
2016-06-28 21:55:20,734 WARN  [org.ovirt.engine.core.bll.ChangeVMClusterCommand] (ForkJoinPool-1-worker-0) [3ce2a3ad] Validation of action 'ChangeVMCluster' failed for user SYSTEM. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__VM__CLUSTER,ACTION_TYPE_FAILED_CANNOT_RUN_ACTION_ON_NON_MANAGED_VM
```

Expected results:
Allow to change VM cluster on HE VM to complete the cluster compat upgrade

Additional info:

Probably we should allow to 'force' operations using REST API

Comment 1 Roy Golan 2016-07-03 08:32:51 UTC
To make this action safe, it will prevent the ChangeVmCluster action if some HE hosts are active on other clusters which are not the destination cluster. 

All the HE hosts should be running on the destination cluster to prevent the HA cluster to span across ovirt clusters.

Comment 2 Jiri Belka 2016-07-21 07:50:55 UTC
(In reply to Roy Golan from comment #0)
> Description of problem:
> The cluster of the HE VM can't have its compat level updated from 3.6 to
> 4.0, because modifying the HE VM cluster is a blocked operation. So
> eventhough technically the HE VM could be started or migrated out-of-cluster
> (right click, migrate, advanced, choose your 4.0 cluster) [...]

This didn't work for me because there's a check that destination host is not HE host. Can the migration be done from a HE host to non-HE host? Does VM settings need to be changed before this "forced" migration?

> [...]

> Steps to Reproduce:
> 1. have 3.6 cluster which the new 4.0 HE VM is running in
> 2. create new 4.0 cluster, deploy and additinonal HE host there

What does it mean 'deploy an additional HE host'? Should it be done via hosted-engine --deploy (wouldn't this end in a way the host would be part of _same_ cluster as the former HE host?) or should we only and another host with ovirt-hosted-engine-setup rpm?

Thanks for clarification, the steps are not clear at all.

Comment 3 Roy Golan 2016-07-28 10:05:32 UTC
(In reply to Jiri Belka from comment #2)
> (In reply to Roy Golan from comment #0)
> > Description of problem:
> > The cluster of the HE VM can't have its compat level updated from 3.6 to
> > 4.0, because modifying the HE VM cluster is a blocked operation. So
> > eventhough technically the HE VM could be started or migrated out-of-cluster
> > (right click, migrate, advanced, choose your 4.0 cluster) [...]
> 
> This didn't work for me because there's a check that destination host is not
> HE host. Can the migration be done from a HE host to non-HE host? Does VM
> settings need to be changed before this "forced" migration?
> 
> > [...]
> 
> > Steps to Reproduce:
> > 1. have 3.6 cluster which the new 4.0 HE VM is running in
> > 2. create new 4.0 cluster, deploy and additinonal HE host there
> 
> What does it mean 'deploy an additional HE host'? Should it be done via
> hosted-engine --deploy (wouldn't this end in a way the host would be part of
> _same_ cluster as the former HE host?) or should we only and another host
> with ovirt-hosted-engine-setup rpm?
> 
> Thanks for clarification, the steps are not clear at all.

in 4.0 the supported way of adding the additional HE host is via the engine. This means you add a new host and choose to 'Deploy' under the Hosted Engine section in the dialog

After that, you HE VM will have a valid host to migrate to in this new cluster

Comment 4 Nikolai Sednev 2016-07-31 14:29:20 UTC
Works for me on these components on engine:
ovirt-engine-lib-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-setup-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-tools-4.0.2.2-0.1.el7ev.noarch
ovirt-iso-uploader-4.0.0-1.el7ev.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-webadmin-portal-debuginfo-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-dbscripts-4.0.2.2-0.1.el7ev.noarch
ovirt-vmconsole-proxy-1.0.4-1.el7ev.noarch
ovirt-engine-dwh-setup-4.0.2-1.el7ev.noarch
ovirt-engine-dwh-4.0.2-1.el7ev.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-websocket-proxy-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-webadmin-portal-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-cli-3.6.7.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-engine-dashboard-1.0.1-0.el7ev.x86_64
ovirt-engine-setup-base-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-extensions-api-impl-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-tools-backup-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-backend-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-restapi-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
ovirt-log-collector-4.0.0-1.el7ev.noarch
ovirt-host-deploy-java-1.5.1-1.el7ev.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-userportal-debuginfo-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-4.0.2.2-0.1.el7ev.noarch
python-ovirt-engine-sdk4-4.0.0-0.5.a5.el7ev.x86_64
ovirt-engine-vmconsole-proxy-helper-4.0.2.2-0.1.el7ev.noarch
ovirt-engine-userportal-4.0.2.2-0.1.el7ev.noarch
ovirt-image-uploader-4.0.0-1.el7ev.noarch
ovirt-host-deploy-1.5.1-1.el7ev.noarch
ovirt-engine-extension-aaa-jdbc-1.1.0-1.el7ev.noarch
rhev-guest-tools-iso-4.0-5.el7ev.noarch
rhevm-setup-plugins-4.0.0.2-1.el7ev.noarch
rhevm-4.0.2.2-0.1.el7ev.noarch
rhevm-doc-4.0.0-3.el7ev.noarch
rhevm-branding-rhev-4.0.0-3.el7ev.noarch
rhevm-guest-agent-common-1.0.12-3.el7ev.noarch
rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch
rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch
rhevm-dependencies-4.0.0-1.el7ev.noarch
rhev-release-4.0.2-4-001.noarch
rhev-release-4.0.1-2-001.noarch
Linux version 3.10.0-327.30.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Jul 13 22:09:46 EDT 2016
Linux 3.10.0-327.30.1.el7.x86_64 #1 SMP Wed Jul 13 22:09:46 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

Host:
qemu-kvm-rhev-2.3.0-31.el7_2.20.x86_64
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.5.x86_64
ovirt-vmconsole-1.0.4-1.el7ev.noarch
rhev-release-4.0.2-4-001.noarch
vdsm-4.18.9-1.el7ev.x86_64
ovirt-host-deploy-1.5.1-1.el7ev.noarch
ovirt-hosted-engine-ha-2.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.0.1.3-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
mom-0.5.5-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
sanlock-3.2.4-3.el7_2.x86_64
Linux version 3.10.0-327.30.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Jul 13 22:09:46 EDT 2016
Linux 3.10.0-327.30.1.el7.x86_64 #1 SMP Wed Jul 13 22:09:46 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

Comment 5 Nikolai Sednev 2016-07-31 15:34:59 UTC
I've made the whole migration flow using CLI and it worked well between different host clusters within the same data center and even between different data centers. This did not worked via WEBUI, as there was no host in drop down menu available, although it's HA score was 3400 in both WEBUI and CLI.

Host cluster compatibility mode changed successfully via WEBUI 3.6->4.0.

Please consider reopening in case that you must have the ability to migrate HE-VM using WEBUI for cross host cluster migrations while in the same data center.

Comment 6 Nikolai Sednev 2016-07-31 15:49:42 UTC
It was specific host related issue, I was trying to migrate HE-VM to NGN host with relatively old components, when I tried to do the same to el7.2 with latest components, then it worked well also via WEBUI.

NGN's components:
sanlock-3.2.4-2.el7_2.x86_64
ovirt-hosted-engine-ha-2.0.1-1.el7ev.noarch
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch
ovirt-host-deploy-1.5.1-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.16.x86_64
mom-0.5.5-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.5.x86_64
vdsm-4.18.6-1.el7ev.x86_64
ovirt-hosted-engine-setup-2.0.1-1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016
Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 7.2

Comment 7 Jiri Belka 2016-08-01 11:33:49 UTC
> > > Steps to Reproduce:
> > > 1. have 3.6 cluster which the new 4.0 HE VM is running in
> > > 2. create new 4.0 cluster, deploy and additinonal HE host there
> > 
> > What does it mean 'deploy an additional HE host'? Should it be done via
> > hosted-engine --deploy (wouldn't this end in a way the host would be part of
> > _same_ cluster as the former HE host?) or should we only and another host
> > with ovirt-hosted-engine-setup rpm?
> > 
> > Thanks for clarification, the steps are not clear at all.
> 
> in 4.0 the supported way of adding the additional HE host is via the engine.
> This means you add a new host and choose to 'Deploy' under the Hosted Engine
> section in the dialog
> 
> After that, you HE VM will have a valid host to migrate to in this new
> cluster

The above answer is confusing. To raise level of DC/CL from 3.6 to 4.0 where HE VM is running, one just need to change cluster/DC level directly (hosts should of course support 4.0 compat). There is no need to "create new 4.0 cluster, deploy and acditinonal HE host there".

Let's hope doctext would be clear enough.

Comment 8 Nikolai Sednev 2016-08-01 13:00:56 UTC
(In reply to Jiri Belka from comment #7)
> > > > Steps to Reproduce:
> > > > 1. have 3.6 cluster which the new 4.0 HE VM is running in
> > > > 2. create new 4.0 cluster, deploy and additinonal HE host there
> > > 
> > > What does it mean 'deploy an additional HE host'? Should it be done via
> > > hosted-engine --deploy (wouldn't this end in a way the host would be part of
> > > _same_ cluster as the former HE host?) or should we only and another host
> > > with ovirt-hosted-engine-setup rpm?
> > > 
> > > Thanks for clarification, the steps are not clear at all.
> > 
> > in 4.0 the supported way of adding the additional HE host is via the engine.
> > This means you add a new host and choose to 'Deploy' under the Hosted Engine
> > section in the dialog
> > 
> > After that, you HE VM will have a valid host to migrate to in this new
> > cluster
> 
> The above answer is confusing. To raise level of DC/CL from 3.6 to 4.0 where
> HE VM is running, one just need to change cluster/DC level directly (hosts
> should of course support 4.0 compat). There is no need to "create new 4.0
> cluster, deploy and acditinonal HE host there".
> 
> Let's hope doctext would be clear enough.

I totally agree on this, and only wanted to add that comment as part of reproduction steps, which were described in https://bugzilla.redhat.com/show_bug.cgi?id=1351533#c0

Comment 9 Roman Mohr 2016-08-08 09:30:35 UTC
Sorry wrong bug, removing the doc text


Note You need to log in before you can comment on or make changes to this bug.