Bug 1154631

Summary: Inhibit migrations RHEL7.0 -> RHEL 6.5 (or equivalent: CentOS)
Product: Red Hat Enterprise Virtualization Manager Reporter: Michal Skrivanek <michal.skrivanek>
Component: ovirt-engineAssignee: Tomas Jelinek <tjelinek>
Status: CLOSED CURRENTRELEASE QA Contact: Ilanit Stein <istein>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.5.0CC: bugs, ecohen, fromani, gklein, iheim, lpeer, lsurette, mavital, michal.skrivanek, nsednev, pdwyer, pstehlik, rbalakri, Rhev-m-bugs, sherold, s.kieske, tjelinek, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: vt10 Doc Type: Known Issue
Doc Text:
Migrations from RHEL 7.x to RHEL 6.x are not supported. Therefore we don't allow mixed RHEL version inside a same cluster. For manual migration from RHEL 6.x to 7.0 (which is supported) there is a new advanced option in the Migrate dialog to be able to migrate to a different cluster within the same data center. However, the suitability of the destination cluster is left up to the user, so one needs to use extra caution.
 Story Points: ---
Clone Of: 1150191 Environment:
Last Closed: 2015-02-17 08:29:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1150191    
Bug Blocks: 1142879    

Description Michal Skrivanek 2014-10-20 11:28:10 UTC 
+++ This bug was initially created as a clone of Bug #1150191 +++

Description of problem:
Migration backward from RHEL/CentOS 7.0 to RHEL/CentOS 6.5 is not supported
(https://bugzilla.redhat.com/show_bug.cgi?id=1150163), so it is prohibited by engine.

Either migration should not be allowed on mixed clusters, or RHEL 7.0 should not be able to join RHEL 6.5 clusters.

Version-Release number of selected component (if applicable):
ovirt-engine-3.5.0-0.0.master.20140923231850.git42065cc.fc19.noarch


How reproducible:
100%

Steps to Reproduce:
1. have a cluster with RHEL 6.5 hosts
2. add one RHEL 7.0 host to cluster
3. start a VM on the RHEL 7.0 host
4. migrate the VM to any RHEL 6.5 host

Actual results:
RHEL 7.0 host can join a RHEL 6.5 cluster; a VM can be migrated from 7.0 to 6.5 host. after that, random misbehaviour (may) occur(s).

Expected results:
Either migration should not be allowed on mixed clusters, or RHEL 7.0 should not be able to join RHEL 6.5 clusters.

--- Additional comment from Francesco Romani on 2014-10-08 09:15:15 CEST ---

Just to share a thought, I believe Engine should never allow to migrate to a QEMU with lower version, except maybe for micro version.

--- Additional comment from Sven Kieske on 2014-10-08 13:11:07 CEST ---

(In reply to Francesco Romani from comment #1)
> Just to share a thought, I believe Engine should never allow to migrate to a
> QEMU with lower version, except maybe for micro version.

Then you should really check the qemu version and not the rhel version.
In general: features should be tested, not version numbers ;)

--- Additional comment from Michal Skrivanek on 2014-10-20 13:25:34 CEST ---

proposed implementation changes for separating RHEL 6 and RHEL 7 clusters and a special cross-cluster migration:

1) Scheduler changes - even when the specific host is selected we still go to the scheduler (IMHO we should not even in a common case); We can do a small change tricking scheduler to run against the target cluster instead of the one where the VM is, but that may uncover some hidden bugs when we actually pull information from the VM instead of the cluster passed in. Still probably not that difficult.
Another option is to bypass the scheduler as a special case, that's a fairly contained change and I would favor it.

2) on host coming up we check whether the newly joining host has the same set of machine types. If it differs we move it to non-operational with a message. This should be simple enough change, not requiring any db fields, the code has been recently cleaned up/reviewed by Roy for similar procesing of PPC vs x86 archs, so he should still have it in mind and relatively a low risk change.
The behavior would be
- if it's a first host, set the supported emulated machines (current behavior)
- if it's 2nd+ host check if any other host already in the cluster is the same RHEL (as reported in VDS caps already). We can limit this to the specific EL 6 and 7 case

3) In UI in the manual migration dialog we would add "Advanced option" (like in extra nfs options) to specify target cluster. We would simply show all clusters and leave it to the user to select the right one. Simple.

The upgrade workflow:
create a second cluster for migration
move to maintenance/install RHEL 7 hosts into the second cluster ,and bring them up while manually special-migrate the VMs
you end up with an empty old cluster
…so downside is the cluster settings need to be the same, but there aren't that many things to care about
Comment 1 Nikolai Sednev 2014-10-30 07:04:47 UTC 
Will migration 7.0->6.6 blocked as well? I see troubles with VM's being migrated 7.0->6.6 too.
Comment 2 Tomas Jelinek 2014-10-30 07:37:32 UTC 
well, you will not be able to have 6x and 7x hosts in one cluster, so it will not be possible to migrate between them. You will be able to migrate between clusters, but only on your own risk, so you can face problems there.
Comment 3 Michal Skrivanek 2014-10-30 15:44:05 UTC 
upstream is in
Comment 4 Ilanit Stein 2014-11-24 16:13:04 UTC 
On vt10.1 tested:
1. core: disable the mixing of RHEL6 and RHEL7 in one cluster
Tested by: adding a rhel7 host to cluster containing rhel6.6, and vice versa.
Result:
host became non operational, and there was event on that.
The reason for this host status, "Not possible to mix RHEL 6x and 7x hosts in one cluster." is under general tab, at the bottom, and it is not too much noticeable.
This reason should be displayed in event log, as well. 
Tomas, 
can you please consider adding this event?

2.core: enable migration to a different cluster 
Tested by migrating between 2 clusters, from rhel6.6 to rhel7 - successful.

3. webadmin: allow migration to different cluster
Tested Migration from rhel6.6 to rhel7 (different clusters), and from rhel6.6 to rhel6.6 (same cluster) - successful.

4. restapi: added support for optionally select the target cluster - Not tested yet.
Tomas: I guess it is just by adding cluster field to migrate vm object?

Michal,
Is this cross cluster migration additional option documented?
Comment 5 Tomas Jelinek 2014-11-25 07:48:17 UTC 
Hi Ilanit,

1: yes, I'll look at it today.
4: yes, it is only one optional parameter: <cluster id="6be85660-ef6c-42e0-a567-d6921bdb2a22"/>
Comment 6 Ilanit Stein 2014-11-25 08:08:03 UTC 
In continue to comment 4,
4. restapi: added support for optionally select the target cluster - 
Added cluster field to migrate vm object - migration from rhel6.6 to rhel7 cluster was successful. 

Please just add the "rhel 6x and 7x mix is not allowed" event. 

2. Documentation: 
I think we should also explain that manual VM migration has now new option,
of migration to another cluster, regardless to which OS is installed on the source/destination hosts.
Comment 7 Tomas Jelinek 2014-11-25 10:50:30 UTC 
@Ilanit: created a patch which adds the event log: http://gerrit.ovirt.org/#/c/35497
Comment 8 Ilanit Stein 2014-11-25 13:34:28 UTC 
Opened a separate bug for the event added in comment 7: Bug #1167827
Comment 9 Ilanit Stein 2014-11-26 09:22:45 UTC 
testcase is required only to the extent that the dialog works and when you have 2 same clusters you can do the migration and it succeeds.
Comment 10 Ilanit Stein 2014-12-10 11:50:26 UTC 
Verification addition to comment #4:
 
On vt13.1, 
Have 2 clusters, one with rhel6.6, and 2nd with rhel7.
1 VM running on rhel6.6.
Try to put rhel6.6 in maintenance resulted with operation cancelled:

"Error while executing action: The following Hosts have running VMs and cannot be switched to maintenance mode: <hostname>.
Please ensure that the following Clusters have at least one Host in UP state: <rhel6.6 clustername>."
Comment 11 Ilanit Stein 2014-12-25 11:48:46 UTC 
Verified on vt13.5, 
with rhel6.6, and rhel 7.1 Beta (Maipo):
=======================================

1. core: disable the mixing of RHEL6 and RHEL7 in one cluster
Tested by: adding a rhel7.1 host to cluster containing rhel6.6, and vice versa.
Result:
host became non operational, and there was event on that.
The reason for this host status, "Not possible to mix RHEL 6x and 7x hosts in one cluster." is under general tab, at the bottom, and it is not too much noticeable.
Reason is displayed in event log. 

2.core: enable migration to a different cluster 
Tested by migrating between 2 clusters, from rhel6.6 to rhel7.1 - successful.

3. webadmin: allow migration to different cluster
Tested Migration from rhel6.6 to rhel7.1 (different clusters), and from rhel7.1 to rhel6.6 (same cluster) - Failed, as expected.

4. restapi: added support for optionally select the target cluster - 
migration from rhel6.6 to rhel7.1 cluster was successful, migrate action completed and response OK. - as expected.
 
migration from rhel7.1 to rhel6.6 cluster:
The migration itself was not successful, as expected, and there was event on that. But the migrate action completed and response OK. - which is NOT as expected.I'll file a separate bug on that.

5. Have 2 clusters, one with rhel6.6, and 2nd with rhel7.1.
1 VM running on rhel6.6.
Try to put rhel6.6 host in maintenance resulted with operation cancelled:
"Error while executing action: The following Hosts have running VMs and cannot be switched to maintenance mode: <rhel6.6 hostname>.
Please ensure that the following Clusters have at least one Host in UP state: <rhel6.6 clustername>."
Comment 12 Michal Skrivanek 2014-12-25 12:35:31 UTC 
(In reply to Ilanit Stein from comment #11 
> migration from rhel7.1 to rhel6.6 cluster:
> The migration itself was not successful, as expected, and there was event on
> that. But the migrate action completed and response OK. - which is NOT as
> expected.I'll file a separate bug on that.
> 
That's normal, initiating the action should succeed. Then you're supposed to monitor the migration progress, which should be going on for some time and then fail, like in the UI
Comment 13 Omer Frenkel 2015-02-17 08:29:04 UTC 
RHEV-M 3.5.0 has been released