Bug 1420402 - [Docs][Upgrade] Update 4.0 upgrade flow documentation for RHEV-H/RHVH hosts in a SHE environment
Summary: [Docs][Upgrade] Update 4.0 upgrade flow documentation for RHEV-H/RHVH hosts i...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: Documentation
Version: 4.1.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.1.3
: ---
Assignee: Tahlia Richardson
QA Contact: Byron Gravenorst
URL:
Whiteboard:
Depends On:
Blocks: 1420283
TreeView+ depends on / blocked
 
Reported: 2017-02-08 15:00 UTC by Nikolai Sednev
Modified: 2019-05-07 12:55 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-22 01:55:39 UTC
oVirt Team: Docs
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1420283 0 urgent CLOSED Ensure that upgrading the engine vm from 3.6/el6 to 4.0/el7 is properly working once we release 4.1 2021-02-22 00:41:40 UTC

Internal Links: 1420283

Description Nikolai Sednev 2017-02-08 15:00:47 UTC
Description of problem:
The flow for RHV-H needs to go through 4.0 RHV-H upgrade appliance and then reinstall hosts and deploy via the UI\API.

See also https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#Upgrading_a_RHEV-H-Based_Self-Hosted_Engine_Environment for supported upgrade flow for 3.6->4.0.

Addition of 4.1 hosted engine host is not allowed any more from CLI, hence I can't add additional hosted engine hosts to 3.6 environment via CLI.
Adding additional hosted engine hosts to 3.6 supported only using CLI, as addition of hosted engine hosts from UI was introduced in 4.0.
alma04 ~]# hosted-engine --deploy
[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          It has been detected that this program is executed through an SSH connection without using screen.
          Continuing with the installation may lead to broken installation if the network connection fails.
          It is highly recommended to abort the installation and run it inside a screen session using command "screen".
          Do you want to continue anyway? (Yes, No)[No]: yes
[ INFO  ] Hardware supports virtualization
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170208155021-1bgfos.log
          Version: otopi-1.6.0 (otopi-1.6.0-1.el7ev)
[ INFO  ] Detecting available oVirt engine appliances
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Generating libvirt-spice certificates
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: 
          Please specify the full shared storage connection path to use (example: host:/path): 10.35.110.11:/Compute_NFS/nsednev_he_1
[ ERROR ] The selected device already contains a storage domain.
[ ERROR ] Setup of additional hosts using this software is not allowed anymore. Please use the engine web interface to deploy any additional hosts.
[ ERROR ] Failed to execute stage 'Environment customization': Setup of additional hosts using this software is not allowed anymore. Please use the engine web interface to deploy any additional hosts.
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170208155108.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170208155021-1bgfos.log

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.Deploy 3.6HE environment with one 3.6RHEVH HE-host or more.
2.Try adding 4.1RHEVH to 3.6 HE-environment.
3.

Actual results:
4.1RHEVH does not supporting addition of it as additional HE-host via CLI, while addition of additional HE-hosts via UI was introduced only from 4.0.
Only 4.0RHEVH might be added as additional HE host to 3.6Environment.

Comment 1 Yaniv Lavi 2017-02-08 15:06:55 UTC
Any workaround to not require customers to do this stepped RHV-H upgrade to 4.1?

Comment 2 Simone Tiraboschi 2017-02-08 15:16:50 UTC
(In reply to Yaniv Dary from comment #1)
> Any workaround to not require customers to do this stepped RHV-H upgrade to
> 4.1?

This just impacts customers who have just RHEL-H 3.y hosts.

Two options here:
A. In 4.0 the capability to add an additional hosted-engine host via CLI was deprecated but it was still there so simply add (or upgrade) a 4.0 RHEV-H host from CLI as an hosted-engine host and perform the upgrade of the engine VM from 3.6/el6 to 4.0/el7 from there.
B. Use RHEL on one additional host: deploy the host as a 3.6 one HE one via CLI getting the rpms from 7Server-RHEV-Agents-7 channel; then add 7Server-RHEV-4-Agents-7 channel on that host in order to get ovirt-hosted-engine-setup-2.1.z (and it's dependencies) and upgrade the host. At this point the user can run 'hosted-engine --upgrade-appliance' to upgrade the engine appliance from 3.6/el6 to 4.0/el7.

Comment 3 Lucy Bopf 2017-02-21 07:24:26 UTC
Adding additional context for the docs:

The instructions for upgrading a SHE installation from 3.6 to 4.0 must be updated when RHV 4.1 becomes generally available, to account for a scenario in which a user wants to update 3.6 RHEV-H self-hosted engine hosts to 4.1 RHVH.

Although we only support upgrading/migrating from one version to the next version (rather than 3.6 to 4.1), we do support 3.6 compatibility level for clusters. Users who have a self-hosted engine running on 3.6-compatible RHEV-H hosts may want to upgrade to 4.1, and then at some point migrate those hosts to the new RHVH (which requires reinstallation rather than a true upgrade/migration). They will encounter a scenario where they are unable to add the RHVH host to the 3.6 self-hosted engine, because the 3.6 SHE only supports addition of self-hosted engine hosts via the CLI, and this functionality is removed in 4.1.

Suggested flow from Yaniv Dary (with additional info added by lbopf):

If you are running a 3.6 cluster, do not install the 4.1 release of RHVH straight away. Because 4.1 removes the addition of hosts to the SHE cluster via CLI (instead UI/API only), you won't be able to add the host back to the cluster, because the 3.6 manager only supports addition of hosts via CLI.

You should first upgrade one RHEV-H host in the self-hosted engine cluster (that is, the hosts that the SHE runs on) from the legacy RHEV-H to the 4.0 release of RHVH. This is a reinstall of the host and is not considered a upgrade, so the host will not be part of the SHE cluster after reinstall.

To work around this, add the host to the SHE cluster via CLI (still supported in 4.0, although deprecated) and run the --upgrade-appliance command so the manager will migrate from 3.6 to 4.0.

After that, other legacy RHEV-H hosts can be upgraded to 4.1 directly (as 4.1 has 3.6 compatibility) and can be re-added to SHE cluster via API/UI.

Comment 4 Lucy Bopf 2017-04-10 01:52:44 UTC
Assigning to Tahlia for review.

Tahlia, the requirement for this bug is chiefly to clearly outline the workflow a user has to take in moving from 3.6 RHEV-H self-hosted engine hosts to (eventually) 4.1 RHVH self-hosted engine hosts. Comment 3 attempts to explain this as clearly as possible.

Because most of the workflow is in the 3.6->4.0 upgrade, I imagine that this content would be best placed in the 4.0 Upgrade Guide, and referenced from the 4.1 Upgrade Guide.

Comment 5 Tahlia Richardson 2017-05-17 07:18:20 UTC
Let me see if I understand the start and end goal here:
* You start with a 3.6 SHE env.
* You end with a 4.1 SHE env. The Manager and one host have a brief stop at 4.0 along the way, but the rest of the hosts can jump straight to 4.1.
* This only applies if all your hosts are RHEV-H, as RHEL hosts don't have this problem. 

The basic steps, IIUC: 

1. Install a new 4.0 RHVH (or remove a 3.6 RHEV-H and "upgrade" it to 4.0 RHVH)

2. Add it to the 3.6 SHE env using "hosted-engine --deploy" (the steps are here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Self-Hosted_Engine_Guide/index.html#chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment but it may not be worth linking to as it's not quite correct for 4.0 RHVH. Better to just write some new steps. Actually, now that I think about it, the 4.0 upgrade section, linked in the next step, should probably specify to add the new RHVH host using the CLI?)

3. Upgrade the Manager from 3.6 to 4.0 using these steps: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html-single/self-hosted_engine_guide/#Upgrading_a_RHEV-H-Based_Self-Hosted_Engine_Environment (from step 3 onwards)

4. Remove each 3.6 RHEV-H and reinstall as 4.1 RHVH, and add to the 4.0 SHE using these steps: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html-single/self-hosted_engine_guide/#chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment

5. Upgrade the Manager to 4.1 (the steps are here: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/self-hosted_engine_guide/upgrading_the_self-hosted_engine but should they use "hosted-engine --upgrade-appliance" instead of manually changing the repos and everything? might need to be updated)

6. Update the remaining 4.0 host to 4.1 with the Upgrade button in the UI or "yum update".

I think this would be best placed as a new section after the regular upgrade section in the 4.1 Self-Hosted Engine Guide, with links to the appropriate 4.0 sections.

Please let me know if I've misunderstood anything.

Comment 6 Tahlia Richardson 2017-05-19 07:03:26 UTC
Yaniv, am I on the right track in my comment above?

Comment 7 Yaniv Lavi 2017-05-21 08:29:37 UTC
(In reply to Tahlia Richardson from comment #5)
> Let me see if I understand the start and end goal here:
> * You start with a 3.6 SHE env.
> * You end with a 4.1 SHE env. The Manager and one host have a brief stop at
> 4.0 along the way, but the rest of the hosts can jump straight to 4.1.
> * This only applies if all your hosts are RHEV-H, as RHEL hosts don't have
> this problem. 
> 
> The basic steps, IIUC: 
> 
> 1. Install a new 4.0 RHVH (or remove a 3.6 RHEV-H and "upgrade" it to 4.0
> RHVH)

Ack

> 
> 2. Add it to the 3.6 SHE env using "hosted-engine --deploy" (the steps are
> here:
> https://access.redhat.com/documentation/en-US/
> Red_Hat_Enterprise_Virtualization/3.6/html-single/Self-Hosted_Engine_Guide/
> index.html#chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment but
> it may not be worth linking to as it's not quite correct for 4.0 RHVH.
> Better to just write some new steps. Actually, now that I think about it,
> the 4.0 upgrade section, linked in the next step, should probably specify to
> add the new RHVH host using the CLI?)

Ack

> 
> 3. Upgrade the Manager from 3.6 to 4.0 using these steps:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/
> html-single/self-hosted_engine_guide/#Upgrading_a_RHEV-H-Based_Self-
> Hosted_Engine_Environment (from step 3 onwards)

This is the flow for 3.6->4.0 migration:

https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html-single/self-hosted_engine_guide/#Upgrading_the_Self-Hosted_Engine

> 
> 4. Remove each 3.6 RHEV-H and reinstall as 4.1 RHVH, and add to the 4.0 SHE
> using these steps:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/
> html-single/self-hosted_engine_guide/#chap-
> Installing_Additional_Hosts_to_a_Self-Hosted_Environment

Ack

> 
> 5. Upgrade the Manager to 4.1 (the steps are here:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/
> html/self-hosted_engine_guide/upgrading_the_self-hosted_engine but should
> they use "hosted-engine --upgrade-appliance" instead of manually changing
> the repos and everything? might need to be updated)

To update the manager they need to enable channels, yum update and engine-setup just like any minor.

> 
> 6. Update the remaining 4.0 host to 4.1 with the Upgrade button in the UI or
> "yum update".
> 
> I think this would be best placed as a new section after the regular upgrade
> section in the 4.1 Self-Hosted Engine Guide, with links to the appropriate
> 4.0 sections.
> 
> Please let me know if I've misunderstood anything.

The flow is also affected by:
https://bugzilla.redhat.com/show_bug.cgi?id=1445161
As this adds a NGN RHV-H for 3.6 and the steps are like RHEL which don't need these extra step. We should recommend to use that everywhere we can.

Comment 8 Yaniv Lavi 2017-05-21 08:32:05 UTC
Simone, please keep me honest.

Comment 9 Tahlia Richardson 2017-05-22 06:44:03 UTC
(In reply to Yaniv Dary from comment #7)
> > 
> > 3. Upgrade the Manager from 3.6 to 4.0 using these steps:
> > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/
> > html-single/self-hosted_engine_guide/#Upgrading_a_RHEV-H-Based_Self-
> > Hosted_Engine_Environment (from step 3 onwards)
> 
> This is the flow for 3.6->4.0 migration:
> 
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/
> html-single/self-hosted_engine_guide/#Upgrading_the_Self-Hosted_Engine
> 

If it's a newly-installed RHVH 4.0, then we don't need to enable the rhv-4-mgmt-agent repo, or install/upgrade ovirt-hosted-engine-setup, so https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html-single/self-hosted_engine_guide/#Upgrading_the_Self-Hosted_Engine isn't quite right.

https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html-single/self-hosted_engine_guide/#Upgrading_a_RHEV-H-Based_Self-Hosted_Engine_Environment describes the exact scenario we're talking about here, i.e. installing a new RHVH 4.0 and adding it to a 3.6 SHE env, then upgrading the Manager to 4.0. As far as I can tell, all that needs to be changed is that step 2 should specify to use the CLI instead of linking to the 4.0 instructions for adding an additional SHE host, and step 4 can (I hope?) be replaced with yum install rhvm-appliance.

Comment 15 Byron Gravenorst 2017-06-21 03:44:11 UTC
Reviewed and verified.


Note You need to log in before you can comment on or make changes to this bug.