Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1871740

Summary: [Upgrade] RHEL 8 host is deployed with 4.3 HE storage domain
Product: Red Hat Enterprise Virtualization Manager Reporter: Roman Hodain <rhodain>
Component: ovirt-engineAssignee: Steve Goodman <sgoodman>
Status: CLOSED CURRENTRELEASE QA Contact: Nikolai Sednev <nsednev>
Severity: high Docs Contact:
Priority: high    
Version: 4.4.1CC: emarcus, mavital, michal.skrivanek, nsednev, pelauter, sbonazzo, sgoodman
Target Milestone: ovirt-4.4.3-1Keywords: Documentation, NoDocsQEReview, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 10:59:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roman Hodain 2020-08-24 07:31:45 UTC
Description of problem:
When deploying the second hypervisor after upgrade to RHV 4.4, the host is deployed with the old hosted storage from 4.3 environment.

Version-Release number of selected component (if applicable):
4.4.1

How reproducible:
100%

Steps to Reproduce:
1. Proceed with the upgrade to 4.4 with the first HE host.
2. Migrate hosted engine to one of the old hosts
3. Start deploying another HE hosts.

Actual results:
The hosts are connected to the old HE SD

Expected results:
Hosts are correctly deployed

Additional info:
This most probably comes from the confusing situation after the initial upgrade of the RHV-m. In that stage, the upgraded host is running RHEL8 with new hosted storage and hosted engine. But there are still RHEL7 hosts that are marked as HE hosts as well and the new hosted engine can be migrated on them. When the HE is on the old host and a new deployment is triggered the deployment is done for the old HE storage.
This situation is also confusing as if you use the hosted-engine --vm-status on the RHEL8 host you will see that the HE is down and it cannot be started as the disk is already locked form another host which is not reported in the status as it operates with different SD.

Comment 2 Sandro Bonazzola 2020-08-27 07:28:41 UTC
Why is step 2 done? During this phase hosted engine should be in global maintenance, nothing should migrate it till all the HE hosts are upgraded.

Comment 3 Steve Goodman 2020-08-27 07:38:52 UTC
Per Sandro: We need to state clearly in the HE section of the upgrade guide that you need to detach the 4.3 storage domain before upgrading the engine to 4.4, and use a dedicated storage domain for the 4.4 hosted-engine deployment.

Two reasons:
- In an HA environment, other 4.3 HA-hosts are still connected to the 4.3 storage domain.
- In HA and non-HA environments, keeping the 4.3 storage domain is necessary for disaster recovery.

Comment 4 Steve Goodman 2020-10-27 15:46:17 UTC
Here are the current steps to upgrade SHE from 4.3 to 4.4 [1]:

Upgrading a self-hosted engine environment from version 4.3 to 4.4 involves the following steps:

====

...

3   Migrate any virtual machines that are running on the same host as the Manager virtual machine to another host in the same cluster.
4   Place the environment in global maintenance mode
5   Update the 4.3 Manager to the latest version of 4.3
6   Upgrade the Manager from 4.3 to 4.4
7   Upgrade the self-hosted engine nodes, and any standard hosts, while reducing virtual machine downtime.
8   (Optional) Upgrade RHVH while preserving local storage.
9   Update the compatibility version of the clusters
10  Reboot any running or suspended virtual machines to update their configuration
11  Update the compatibility version of the data centers 

====


So I will add:

4.1 Detach the 4.3 storage domain before upgrading the engine to 4.4, and use a dedicated storage domain for the 4.4 hosted-engine deployment.


[1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/upgrade_guide/index#SHE_Upgrading_from_4-3

Comment 10 Steve Goodman 2020-11-11 10:52:15 UTC
Adding back NoDocsQEReview keyword after speaking with Sandro.

Comment 20 Steve Goodman 2020-11-16 10:21:16 UTC
Moving to verified after both Nikolai and Roman signed off.

Comment 21 Steve Goodman 2020-11-16 15:05:16 UTC
Eli, please do peer review. See comment 5 for the merge request.

Comment 22 Eli Marcus 2020-11-16 16:33:15 UTC
Peer reviewed,
merged into 4.4 branch - 
RHV Upgrade Guide

Comment 23 Nikolai Sednev 2020-11-16 18:25:40 UTC
For any questions regarding working steps, I've verified this flow following https://bugzilla.redhat.com/show_bug.cgi?id=1893385#c5.