Bug 1894576

Summary: Warning mentioning engine being at version 4.5 leads to a nightmare path upgrading the cluster ending with a dead datacenter
Product: [oVirt] ovirt-engine Reporter: Sandro Bonazzola <sbonazzo>
Component: BLL.InfraAssignee: Martin Perina <mperina>
Status: CLOSED CURRENTRELEASE QA Contact: Lucie Leistnerova <lleistne>
Severity: medium Docs Contact:
Priority: high    
Version: 4.4.3.10CC: bugs, michal.skrivanek, mperina
Target Milestone: ovirt-4.4.3-1Flags: pm-rhel: ovirt-4.4+
Target Release: 4.4.3.11   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.4.3.11 Doc Type: Bug Fix
Doc Text:
Upgrading cluster compatibility level will be possible only when there are no hosts in the cluster or when there is at least one host which supports new cluster compatibility level
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-10 11:31:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot
none
second error screenshot none

Description Sandro Bonazzola 2020-11-04 14:31:36 UTC
Installing an oVirt 4.4.3.10 engine, attaching a centos/RHEL 8.2 a warning message appears:

"Data Center Default compatibility version is 4.4, which is lower than latest engine version 4.5. Please upgrade your Data Center to latest version to successfully finish upgrade of your setup."

engine is not version 4.5, it's 4.4 and this is going to generate confusion at several levels including GSS

Comment 1 Sandro Bonazzola 2020-11-04 14:33:51 UTC
Created attachment 1726576 [details]
screenshot

Comment 2 Sandro Bonazzola 2020-11-04 14:38:55 UTC
Above mentioned message asks to update data center to 4.5 (assuming we fix the wording about it being engine version) but then, when you try to upgrade datacenter to latest version you get:


"Error while executing action: Cannot update Data Center compatibility version to a value that is greater than its cluster's version. The following clusters should be upgraded: Default."

Comment 3 Sandro Bonazzola 2020-11-04 14:40:18 UTC
Created attachment 1726577 [details]
second error screenshot

Comment 4 Sandro Bonazzola 2020-11-04 14:41:38 UTC
Then when you try to update the cluster level to 4.5 you get:

"Error while executing action: Cannot change Cluster Compatibility Version to higher version when there are active Hosts with lower version.
-Please move Host node0 with lower version to maintenance first."

Comment 5 Sandro Bonazzola 2020-11-04 14:44:32 UTC
Then, putting node to maintenance, successfully updated cluster to 4.5, successfully updated datacenter to 4.5, activating the hosts, leads to having host not operational with message:

"Host node0 is compatible with versions (4.2,4.3,4.4) and cannot join Cluster Default which is set to version 4.5."

Comment 6 Sandro Bonazzola 2020-11-04 14:48:12 UTC
And finally: node is a CentOS 8.2 and since 8.3 doesn't exists yet you can't really upgrade it reaching 4.5 compatibility. On RHEL side, there's no indication that for reaching 4.5 compatibility you need to upgrade to RHEL 8.3.

Comment 7 Sandro Bonazzola 2020-11-04 14:53:15 UTC
Last, once update cluster and datacenter to 4.5 there's no way to move back to 4.4. So you end up with a non working environment.

Comment 8 Martin Perina 2020-11-04 15:07:34 UTC
So why not creating a new datacenter and cluster with 4.4 version and move the there? I don't see any easy way how to fix that other thane mentioning in release notes, that cluster level 4.5 requires hosts with CentOS/RHEL 8.3, because with host in Maintenance we don't have any way how to find out the version installed the host

Comment 9 Sandro Bonazzola 2020-11-04 15:19:32 UTC
(In reply to Martin Perina from comment #8)
> So why not creating a new datacenter and cluster with 4.4 version and move
> the there? I don't see any easy way how to fix that other thane mentioning
> in release notes, that cluster level 4.5 requires hosts with CentOS/RHEL
> 8.3, because with host in Maintenance we don't have any way how to find out
> the version installed the host

it's not just moving the host, creating a new datacenter you need to move everything there, hosts, storage, VMs.
it's going to become a nightmare.

Comment 10 Sandro Bonazzola 2020-11-04 15:32:34 UTC
Can we at least change 
"Data Center Default compatibility version is 4.4, which is lower than latest engine version 4.5. Please upgrade your Data Center to latest version to successfully finish upgrade of your setup."

to 

"Data Center Default compatibility version is 4.4, which is lower than latest compatibility version 4.5. Please upgrade your hosts to Enterprise Linux 8.3 and then upgrade your Data Center to latest version to successfully finish upgrade of your setup." ?

Comment 11 Sandro Bonazzola 2020-11-04 15:34:25 UTC
Or better:
"Data Center Default compatibility version is 4.4, which is lower than latest compatibility version 4.5. Please upgrade your hosts to Enterprise Linux 8.3 and then upgrade in order your cluster and your datacenter to latest version to successfully finish upgrade of your setup." ?

Comment 14 Michal Skrivanek 2020-11-05 07:49:40 UTC
This is not a blocker by any means, not even a serious issue. This is the same behavior we have for years.
Adjusting severity

If you believe it needs to get in, ask for an exception, not a blocker

Comment 15 Sandro Bonazzola 2020-11-10 11:09:31 UTC
Leaving this on QA, but verified with engine 4.4.3.11.
New message is "Data Center Default compatibility version is 4.4, which is lower than latest available version 4.5. Please upgrade your Data Center to latest version to successfully finish upgrade of your setup." and when trying to upgrade the cluster to 4.5 using a CentOS 8.2 on hosts it fails with "Error while executing action: Cannot change Cluster compatibility version where there are no hosts in the Cluster which support that version." avoiding to reach a not working state.

Comment 17 Sandro Bonazzola 2020-11-11 06:45:47 UTC
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.