Bug 1341023

verified on : 
Red Hat Enterprise Virtualization Manager Version: 3.6.7.2-0.1.el6

Verification steps: 
1. Create a DC with 3.5 compatibility version
2. Add a host with 3.5 compatibility version
3. Create and run vm 
4. Try to edit the  compatibility version in cluster from 3.5 to 3.6

Result : 
'shut down all vm before changing the cluster level'

Does this fix break the Hosted Engine case? I have a 3.6 cluster I've upgraded to 4.0RC2 and I can't figure out how to how to change the Compatibility Version of the cluster to 4.0 now. When I attempt to change it by editing the cluster properties I get the message "Shut down all VMs before changing the cluster version." The only VM running is the Hosted Engine.

I tried simply creating a new cluster at version 4.0 but there doesn't seem to be any way of moving the Hosted Engine to the new cluster nor does there seem to be a way to migrate my 100+ VMs to the new cluster without having to manually edit the properties of each one. Is there a way to do this?

I'm new to oVirt so perhaps I'm missing something...

Thank you,
Carl Thompson

Didi, can you update on teh right way how to execute the upgrade patch?

patch->path. Sorry.

the point is there needs to be a HE shutdown somewhere, and then it should start up in the right 4.0 cluster

Hello, Michal. Thanks for responding.

I don't understand what you're suggesting; I've restarted the HE and it comes back up in the old cluster for which I can't change the Compatibility Version. Are you saying there's a way to restart it so that it comes up in a different cluster? Can you elaborate on how that's done?

Thanks,
Carl

(In reply to Carl Thompson from comment #6)
> Hello, Michal. Thanks for responding.
> 
> I don't understand what you're suggesting; I've restarted the HE and it
> comes back up in the old cluster for which I can't change the Compatibility
> Version. Are you saying there's a way to restart it so that it comes up in a
> different cluster? Can you elaborate on how that's done?
> 
> Thanks,
> Carl

Hi Carl,
this should be solved by bug 1341145 in 4.0.0.5 build, so please try to update to the latest available and try to change the cluster again

(In reply to Carl Thompson from comment #3)
 
> I tried simply creating a new cluster at version 4.0 but there doesn't seem
> to be any way of moving the Hosted Engine to the new cluster nor does there
> seem to be a way to migrate my 100+ VMs to the new cluster without having to
> manually edit the properties of each one. Is there a way to do this?

There might be a real issue with HE, still investigating, but to answer this specific point about 100 VMs - they need to be shut down if you want to move the cluster to newer cluster level or they need to have the per-VM cluster level override set.
This should be improved. I guess it needs to be done soon

(In reply to Michal Skrivanek from comment #7)

> Hi Carl,
> this should be solved by bug 1341145 in 4.0.0.5 build, so please try to
> update to the latest available and try to change the cluster again

This did not fix the problem for me. Still can't change the Compatibility Version and get the error message "Shut down all VMs before changing the cluster version." Bug 1341145 does not seem to be related to this at all... Are you sure about that bug number?

[root@2sexi ~]# rpm -qa | grep ovirt-engine
ovirt-engine-setup-base-4.0.0.6-1.el7.centos.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.0.0.6-1.el7.centos.noarch
ovirt-engine-vmconsole-proxy-helper-4.0.0.6-1.el7.centos.noarch
ovirt-engine-restapi-4.0.0.6-1.el7.centos.noarch
ovirt-engine-tools-4.0.0.6-1.el7.centos.noarch
ovirt-engine-dwh-4.0.0-2.gita09d329.el7.centos.noarch
ovirt-engine-extension-aaa-jdbc-1.1.0-1.el7.noarch
python-ovirt-engine-sdk4-4.0.0-0.3.a3.el7.centos.x86_64
ovirt-engine-dashboard-1.0.0-0.2.20160610git5d210ea.el7.centos.noarch
ovirt-engine-lib-4.0.0.6-1.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.0.0.6-1.el7.centos.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.0.6-1.el7.centos.noarch
ovirt-engine-extension-aaa-ldap-1.2.0-1.el7.noarch
ovirt-engine-websocket-proxy-4.0.0.6-1.el7.centos.noarch
ovirt-engine-setup-4.0.0.6-1.el7.centos.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7.centos.noarch
ovirt-engine-tools-backup-4.0.0.6-1.el7.centos.noarch
ovirt-engine-dbscripts-4.0.0.6-1.el7.centos.noarch
ovirt-engine-backend-4.0.0.6-1.el7.centos.noarch
ovirt-engine-4.0.0.6-1.el7.centos.noarch
ovirt-engine-cli-3.6.2.0-1.el7.centos.noarch
ovirt-engine-dwh-setup-4.0.0-2.gita09d329.el7.centos.noarch
ovirt-engine-wildfly-overlay-10.0.0-1.el7.noarch
ovirt-engine-wildfly-10.0.0-1.el7.x86_64
ovirt-engine-setup-plugin-ovirt-engine-4.0.0.6-1.el7.centos.noarch
ovirt-engine-extension-aaa-ldap-setup-1.2.0-1.el7.noarch
ovirt-engine-extensions-api-impl-4.0.0.6-1.el7.centos.noarch
ovirt-engine-webadmin-portal-4.0.0.6-1.el7.centos.noarch
ovirt-engine-userportal-4.0.0.6-1.el7.centos.noarch

yeah, the number is correct, but there were others still preventing this. We looked at that in more detail today and AFAICT it's not possible to do it even in the latest version. We're working on a fix
As a workaround for now you'd need to shut down the engine and make a direct database modification upgrading the number in the table

(In reply to Michal Skrivanek from comment #8)

> There might be a real issue with HE, still investigating, but to answer this
> specific point about 100 VMs - they need to be shut down if you want to move
> the cluster to newer cluster level or they need to have the per-VM cluster
> level override set.
> This should be improved. I guess it needs to be done soon

Thanks. However the issue I was alluding to there is that there doesn't seem to be a way to move VMs _en masse_ from one cluster to another. So I have these 100+ VMs (which are turned off currently) and the only way to move them from cluster 1 to cluster 2 is to click into the settings of each one individually. There doesn't seem to be a way to select them all and switch them to a different cluster at once. (I suppose I could try exporting them to the export domain and re-importing them into the new cluster but that seems like a really long workaround to solve a simple problem.)

BTW, this is a test cluster to evaluate whether we want to use oVirt in production so I can make changes and test as necessary to help resolve these issues.

Thanks,
Carl

(In reply to Michal Skrivanek from comment #8)

> There might be a real issue with HE, still investigating, but to answer this
> specific point about 100 VMs - they need to be shut down if you want to move
> the cluster to newer cluster level or they need to have the per-VM cluster
> level override set.
> This should be improved. I guess it needs to be done soon

Thanks. However the issue I was alluding to there is that there doesn't seem to be a way to move VMs _en masse_ from one cluster to another. So I have these 100+ VMs (which are turned off currently) and the only way to move them from cluster 1 to cluster 2 is to click into the settings of each one individually. There doesn't seem to be a way to select them all and switch them to a different cluster at once. (I suppose I could try exporting them to the export domain and re-importing them into the new cluster but that seems like a really long workaround to solve a simple problem.)

BTW, this is a test cluster to evaluate whether we want to use oVirt in production so I can make changes and test as necessary to help resolve these issues.

Thanks,
Carl

(In reply to Carl Thompson from comment #12)
> Thanks. However the issue I was alluding to there is that there doesn't seem
> to be a way to move VMs _en masse_ from one cluster to another.

If indeed impossible from the UI, you can open an RFE to allow this.

Should probably be not-too-difficult with the python-sdk (or even ovirt-shell), but I never tried this so can't provide details.

(In reply to Yedidyah Bar David from comment #13)
> (In reply to Carl Thompson from comment #12)
> > Thanks. However the issue I was alluding to there is that there doesn't seem
> > to be a way to move VMs _en masse_ from one cluster to another.
> 
> If indeed impossible from the UI, you can open an RFE to allow this.
> 
> Should probably be not-too-difficult with the python-sdk (or even
> ovirt-shell), but I never tried this so can't provide details.

we have a bug on mass edit VMs somewhere around...not going to happen any time soon though, not within the current UI infra:/

(In reply to Carl Thompson from comment #12)
> Thanks. However the issue I was alluding to there is that there doesn't seem
> to be a way to move VMs _en masse_ from one cluster to another. So I have
> these 100+ VMs (which are turned off currently) and the only way to move
> them from cluster 1 to cluster 2 is to click into the settings of each one
> individually. There doesn't seem to be a way to select them all and switch
> them to a different cluster at once. (I suppose I could try exporting them
> to the export domain and re-importing them into the new cluster but that
> seems like a really long workaround to solve a simple problem.)


sure. Currently if the only offending VM is the HE VM you can perhaps move that one using the cross-cluster migration to a newly created 3.6 cluster to clear the one with 100 VMs so it lets you update it. There should be not so many drawbacks running a 4.0 engine in a HE VM running in 3.6 cluster - it's still a 4.0 engine.

(In reply to Michal Skrivanek from comment #15)

> sure. Currently if the only offending VM is the HE VM you can perhaps move
> that one using the cross-cluster migration to a newly created 3.6 cluster to
> clear the one with 100 VMs so it lets you update it. There should be not so
> many drawbacks running a 4.0 engine in a HE VM running in 3.6 cluster - it's
> still a 4.0 engine.

It does not work to cross-cluster migrate the HE VM. The VM migrates to the new host in a different cluster fine but the HE VM is still associated with the previous cluster. In other words, the VM is in one cluster but ends up running on a  host that's in a different cluster. That itself might be considered a bug. Because the HE VM still is associated with the first cluster oVirt won't let me update the Compatibility Version of the first cluster (even though the HE VM is actually running on a host in a different cluster)!

Thanks,
Carl

(In reply to Michal Skrivanek from comment #10)

> As a workaround for now you'd need to shut down the engine and make a direct
> database modification upgrading the number in the table

Is this all I need to do:

    engine=# UPDATE cluster SET compatibility_version='4.0' WHERE name='ClusterOne';

Thanks,
Carl

(In reply to Carl Thompson from comment #17)
> (In reply to Michal Skrivanek from comment #10)
> 
> > As a workaround for now you'd need to shut down the engine and make a direct
> > database modification upgrading the number in the table
> 
> Is this all I need to do:
> 
>     engine=# UPDATE cluster SET compatibility_version='4.0' WHERE
> name='ClusterOne';
> 
> Thanks,
> Carl

Yes, that sounds about right. Should work. 

Didi, can you see comment #16 re HE behavior, maybe it needs to be changed in ovf or whatever is used for HE VM...

(In reply to Michal Skrivanek from comment #18)
> (In reply to Carl Thompson from comment #17)
> > Is this all I need to do:
> > 
> >     engine=# UPDATE cluster SET compatibility_version='4.0' WHERE
> > name='ClusterOne';
> > 
> > Thanks,
> > Carl
> 
> Yes, that sounds about right. Should work. 

This worked, thanks. Obviously not ideal but my cluster's Compatibility Version is updated.

Out of curiosity why is being able to change the Compatibility Version while VMs a bug? I can't imagine that changing the version would affect running VMs until they're restarted...

(In reply to Michal Skrivanek from comment #18)
> Didi, can you see comment #16 re HE behavior, maybe it needs to be changed
> in ovf or whatever is used for HE VM...

No idea about the engine's behavior re the hosted-engine vm. Roy - can you please have a look?

I'd just like to emphasize that there are two different, mostly-unrelated meanings, to the word "cluster" here:

1. A cluster in the engine (in its database)
2. A hosted-engine cluster - that is, all the hosts on which 'hosted-engine --deploy' was ran against a single hosted-storage.

Generally speaking, they should be identical in practice. But the code handling migrations inside the latter (2.), ovirt-hosted-engine-ha, does not check how the host it's running on is configured in the engine - all it needs is access to the hosted-storage. 'hosted-engine --deploy' should put them all in the same (1.)-cluster, but I am not sure if anything prevents moving hosts later between (1.)-clusters. If that's allowed (intentionally or not), the (2.)-cluster is not affected.

For comment #16 and my other comments I was speaking only of clusters as they are configured in the engine's database. I was not speaking about HE clusters at all. The HE itself appears to be special cased in the engine's database which causes these issues.

I would really appreciate if someone can clarify the upgrade procedure and whther it does or can work as of today (looking rgolan's way;)

Regardless, for the issue with "100 VMs needs to be shut down" I opened a separate bug 1348907 to allow for incremental upgrade while VMs are running

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1364

I opened documentation bug for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1356198

And here is the d/s bug to allow this change with running vms:
https://bugzilla.redhat.com/show_bug.cgi?id=1356194