Bug 1854393 - Unable to update cluster compatibility, post upgrading the self-hosted-engine setup to RHV 4.4
Summary: Unable to update cluster compatibility, post upgrading the self-hosted-engine...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Gluster
Version: 4.4.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ovirt-4.4.1
: ---
Assignee: Gobinda Das
QA Contact: SATHEESARAN
URL:
Whiteboard:
Depends On:
Blocks: 1854401
TreeView+ depends on / blocked
 
Reported: 2020-07-07 11:57 UTC by SATHEESARAN
Modified: 2020-08-05 06:24 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1854401 (view as bug list)
Environment:
Last Closed: 2020-08-05 06:24:38 UTC
oVirt Team: Gluster
Embargoed:
sasundar: ovirt-4.4?
sasundar: blocker?
sasundar: planning_ack?
pm-rhel: devel_ack+
pm-rhel: testing_ack+


Attachments (Terms of Use)
engine.log (13.25 MB, application/octet-stream)
2020-07-07 12:01 UTC, SATHEESARAN
no flags Details
Error dialog, when updating cluster compatibility post upgrading to 4.4 (48.93 KB, image/png)
2020-07-07 12:14 UTC, SATHEESARAN
no flags Details

Description SATHEESARAN 2020-07-07 11:57:20 UTC
Description of problem:
------------------------
I had a 3 node self-hosted engine setup running RHV 4.3.10-async. This is a hyperconverged environment, gluster storage domains are used with brick created on the local disks

Upgrade was smooth, and post upgrade, edited the cluster to 4.4 compatibility, with all the VMs up and running. There was a general warning thrown out, but I have seen null pointer exception.

When turning off all the VMs except HostedEngine VM (4.4 VM), I have seen new set of error, 'CLUSTER_CANNOT_DISABLE_GLUSTER_WHEN_CLUSTER_CONTAINS_VOLUMES'

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
RHV 4.4.1
As there are no new appliance for QE, used rhvm-appliance-4.4-20200604.0.el8ev

How reproducible:
-----------------
Always

Steps to Reproduce:
--------------------
1. Create RHHI-V deployment with 3 node self-hosted-engine setup
2. Upgrade the environment to RHV 4.4
3. Update the cluster compatibility to 4.4 after all the hosts are upgraded

Actual results:
---------------
Unable to update the cluster to 4.4 as the error mentions that there are VM/Templates and asks to change that specifically. Then after turning off all the VMs, I see error - 'CLUSTER_CANNOT_DISABLE_GLUSTER_WHEN_CLUSTER_CONTAINS_VOLUMES'

Expected results:
-----------------
1. When VMs are up, updating cluster compatibility to 4.4 should succeed, with VMs needing to reboot later
2. When all the VMs are shutdown, able to update cluster compatibility

Additional info:
-----------------
I have started upgrade from RHV 4.3.8 to RHV 4.3.10-async, then continued to upgrade to RHV 4.4

Comment 1 RHEL Program Management 2020-07-07 11:57:26 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 2 SATHEESARAN 2020-07-07 12:01:48 UTC
Created attachment 1700142 [details]
engine.log

Comment 3 SATHEESARAN 2020-07-07 12:11:52 UTC
For your information, with RHHI-V environment hosts are deployed in to 'Default' cluster.

I see 2 problems here:

1. I am seeing Null Pointer Exception, when updating the compatibility version of the 'Default' cluster, when all the VMs are up and running.

2. When all the VMs are powered down, I see a different problem - 'CLUSTER_CANNOT_DISABLE_GLUSTER_WHEN_CLUSTER_CONTAINS_VOLUMES'
<snip>
2020-07-07 09:46:24,661Z WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-23) [bf38294d-faef-422a-b68a-22870e57e632] EVENT_ID: CANNOT_HIBERNATE_RUNNING_VMS_AFTER_CLUSTER_CPU_UPGRADE(9,006), Hibernation of VMs after CPU upgrade of Cluster Default is not supported. Please stop and restart those VMs in case you wish to hibernate them
2020-07-07 09:46:24,671Z WARN  [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-23) [bf38294d-faef-422a-b68a-22870e57e632] Validation of action 'UpdateCluster' failed for user admin@internal-authz. Reasons: VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,CLUSTER_CANNOT_DISABLE_GLUSTER_WHEN_CLUSTER_CONTAINS_VOLUMES
</snip>

Comment 4 SATHEESARAN 2020-07-07 12:14:41 UTC
Created attachment 1700144 [details]
Error dialog, when updating cluster compatibility post upgrading to 4.4

Here is the screenshot of the error dialog of when editing RHV default cluster, to update cluster compatibility to 4.4

Comment 5 Michal Skrivanek 2020-07-08 04:57:08 UTC
This was fixed in bug 1847513. Make sure you’re running up to date version please. You’re supposed to upgrade appliance before using it

Comment 6 SATHEESARAN 2020-07-09 07:41:36 UTC
(In reply to Michal Skrivanek from comment #5)
> This was fixed in bug 1847513. Make sure you’re running up to date version
> please. You’re supposed to upgrade appliance before using it

I am redoing the upgrade test and will update the results.

Comment 7 Gobinda Das 2020-07-09 13:29:32 UTC
Based on comment#5 moving this to ON_QA

Comment 8 SATHEESARAN 2020-07-13 08:00:11 UTC
Verified with RHV manager 4.4.1.8-0.7.el8ev

Post upgrading all the hosts with RHVH 4.4.1, with the latest appliance (rhvm-appliance-4.4-20200707.0.el8ev.x86_64),
the cluster could be updated with 4.4 without any errors

Comment 9 Sandro Bonazzola 2020-08-05 06:24:38 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.