Bug 1420283 - Ensure that upgrading the engine vm from 3.6/el6 to 4.0/el7 is properly working once we release 4.1
Summary: Ensure that upgrading the engine vm from 3.6/el6 to 4.0/el7 is properly worki...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ovirt-4.1.1
: ---
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
URL:
Whiteboard: integration
Depends On: 1403846 1416011 1420402
Blocks: 1235200 1370041 1403903
TreeView+ depends on / blocked
 
Reported: 2017-02-08 11:02 UTC by Simone Tiraboschi
Modified: 2017-04-25 01:11 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
oVirt Team: Integration
Target Upstream Version:


Attachments (Terms of Use)
sosreport-alma04.qa.lab.tlv.redhat.com-20170208213942.tar.xz (11.21 MB, application/x-xz)
2017-02-08 19:43 UTC, Nikolai Sednev
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1403846 0 unspecified CLOSED keep 3.6 in the supportedEngines reported by VDSM 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1420402 0 high CLOSED [Docs][Upgrade] Update 4.0 upgrade flow documentation for RHEV-H/RHVH hosts in a SHE environment 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2017:1002 0 normal SHIPPED_LIVE ovirt-hosted-engine-setup bug fix and enhancement update 2017-04-18 20:14:37 UTC
oVirt gerrit 71965 0 None None None 2017-02-14 11:28:37 UTC
oVirt gerrit 71991 0 None None None 2017-02-14 11:29:05 UTC

Internal Links: 1403846 1420402

Description Simone Tiraboschi 2017-02-08 11:02:58 UTC
Description of problem:
The only supported flow for upgrading the engine from 3.6/el6 to 4.1/el7 is
1. 3.6/el6 -> 4.0/el7 with 'hosted-engine --upgrade-appliance' to replace the engine VM with a new one based on el7.
2. Only at that point, 4.0/el7 -> 4.1/el7 upgrading the engine rpms on the engine VM as for a regular engine.

ovirt-hosted-engine-setup-2.0.z is from 4.0 time while ovirt-hosted-engine-setup-2.1.z is from 4.1; both of them are going to be distributed on 7Server-RHEV-4-Agents-7 channel so once we will release 4.1, all the customer will get ovirt-hosted-engine-setup-2.1.z.

We need to ensure that 3.6/el6 -> 4.0/el7 is still properly and flawless working with ovirt-hosted-engine-setup-2.1.z
ovirt-hosted-engine-setup-2.1.z should instead prevent the user from trying a direct 3.6/el6 -> 4.1/el7 via 'hosted-engine --upgrade-appliance'.

The 4.0 appliance is distributed as rhevm-appliance rpm on 7Server-RHEV-4-Agents-7 while the 4.1 appliance is distributed as rhvm-appliance rpm on 7Server-RHEV-4-Agents-7 so the user has to install rhevm-appliance for the 3.6/el6 -> 4.0/el7 upgrade and rhvm-appliance for a fresh 4.1 install.

Comment 2 Nikolai Sednev 2017-02-08 13:57:22 UTC
Addition of 4.1 hosted engine host is not allowed any more from CLI, hence I can't add additional hosted engine hosts to 3.6 environment via CLI.
Adding additional hosted engine hosts to 3.6 supported only using CLI, as addition of hosted engine hosts from UI was introduced in 4.0.
alma04 ~]# hosted-engine --deploy
[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          It has been detected that this program is executed through an SSH connection without using screen.
          Continuing with the installation may lead to broken installation if the network connection fails.
          It is highly recommended to abort the installation and run it inside a screen session using command "screen".
          Do you want to continue anyway? (Yes, No)[No]: yes
[ INFO  ] Hardware supports virtualization
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170208155021-1bgfos.log
          Version: otopi-1.6.0 (otopi-1.6.0-1.el7ev)
[ INFO  ] Detecting available oVirt engine appliances
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Generating libvirt-spice certificates
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: 
          Please specify the full shared storage connection path to use (example: host:/path): 10.35.110.11:/Compute_NFS/nsednev_he_1
[ ERROR ] The selected device already contains a storage domain.
[ ERROR ] Setup of additional hosts using this software is not allowed anymore. Please use the engine web interface to deploy any additional hosts.
[ ERROR ] Failed to execute stage 'Environment customization': Setup of additional hosts using this software is not allowed anymore. Please use the engine web interface to deploy any additional hosts.
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170208155108.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170208155021-1bgfos.log

Comment 3 Nikolai Sednev 2017-02-08 14:00:27 UTC
I've deployed clean 3.6 environment with one 3.6 hosted engine host and tried to add 4.1 RHEVH as additional hosted engine host and failed.
So this flow is not possible for 4.1RHEVH based hosts.

Comment 4 Simone Tiraboschi 2017-02-08 14:59:34 UTC
(In reply to Nikolai Sednev from comment #3)
> I've deployed clean 3.6 environment with one 3.6 hosted engine host and
> tried to add 4.1 RHEVH as additional hosted engine host and failed.
> So this flow is not possible for 4.1RHEVH based hosts.

You can just add the additional host from 3.y channel, then add the 4.y one and upgrade it.
Once released, you should get hosted-engine-setup-2.1.z from there.

Comment 5 Nikolai Sednev 2017-02-08 15:07:06 UTC
Please see also  https://bugzilla.redhat.com/show_bug.cgi?id=1420402 regarding RHEVH upgrade flow.
Please see also https://bugzilla.redhat.com/show_bug.cgi?id=1403846, till its fixed, the only way of upgrading 3.6->4.1 for RHEL based hosts will be direct upgrade-appliance for 4.1RHEL host in 3.6environment directly to 4.1 engine, using upgrade-appliance.

Comment 6 Simone Tiraboschi 2017-02-08 15:34:25 UTC
(In reply to Nikolai Sednev from comment #5)
> Please see also https://bugzilla.redhat.com/show_bug.cgi?id=1403846, till
> its fixed, the only way of upgrading 3.6->4.1 for RHEL based hosts will be
> direct upgrade-appliance for 4.1RHEL host in 3.6environment directly to 4.1
> engine, using upgrade-appliance.

Sorry, why cannot you use a 4.1 RHEL host in 3.6 environment to upgrade the engine appliance to 4.0 and than upgrade the engine to 4.1 via yum inside the VM?

Comment 7 Nikolai Sednev 2017-02-08 16:01:02 UTC
RPM upgrading 3.6RHEL7.3 from 3.6 to 4.1 rpms not possible:
	
Feb 8, 2017 5:55:35 PM
	
	
Host alma04.qa.lab.tlv.redhat.com is installed with VDSM version (4.19) and cannot join cluster Default which is compatible with VDSM versions [4.9, 4.10, 4.11, 4.12, 4.13, 4.14, 4.15, 4.16, 4.17].

Comment 8 Nikolai Sednev 2017-02-08 19:39:35 UTC
1)Manually changed /usr/lib/python2.7/site-packages/vdsm/dsaversion.py
from 'supportedENGINEs': ['4.0', '4.1'], to 'supportedENGINEs': ['3.6', '4.0', '4.1'], then successfully activated back 4.1RHEL7.3.
2)Manually localinstalled (4.0 appliance on 4.1RHEL7.3 host) rhevm-appliance-20161130.0-1.el7ev.noarch.
3)Migrated HE-VM to 4.1RHEL7.3 from 3.6RHEVH7.3.
4)Made 4.1RHEL7.3 as SPM.
5)Moved hosts to global maintenance.
6)Backed up the db from 3.6 engine and copied it to host.
7)Ran hosted-engine --upgrade-appliance on 4.1RHEL7.3.
8)Upgrade failed.
[ INFO  ] Stage: Closing up
[ ERROR ] Failed to execute stage 'Closing up': refresh_local_conf_file() got an unexpected keyword argument 'localcopy_filename'
[ INFO  ] Stage: Clean up
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine upgrade failed: this system is not reliable, you can use --rollback-upgrade option to recover the engine VM disk from a backup
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170208183742-aas1e8.log

Sosreport being attached.

Comment 9 Nikolai Sednev 2017-02-08 19:41:21 UTC
Components from 4.1RHEL7.3 host:
rhevm-appliance-20161130.0-1.el7ev.noarch
vdsm-4.19.4-1.el7ev.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
mom-0.5.8-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.10.1.el7.x86_64 (mockbuild@x86-019.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Jan 30 11:07:00 EST 2017
Linux alma04.qa.lab.tlv.redhat.com 3.10.0-514.10.1.el7.x86_64 #1 SMP Mon Jan 30 11:07:00 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Comment 10 Nikolai Sednev 2017-02-08 19:43:58 UTC
Created attachment 1248657 [details]
sosreport-alma04.qa.lab.tlv.redhat.com-20170208213942.tar.xz

Comment 11 Simone Tiraboschi 2017-02-08 20:27:03 UTC
It's due to an API change introduced here: https://gerrit.ovirt.org/#/c/67924
Fixing using the new API

Comment 12 Yaniv Lavi 2017-02-14 08:35:18 UTC
So this flow is only relevant to RHEL, since the upgrade for RHV-H will have to be stepped.

Comment 13 Nikolai Sednev 2017-03-05 14:50:28 UTC
1.Deployed clean HE over two 3.6RHEL7.3 hosts, over NFS and added 2 data storage domains over NFS, got auto-imported HE-VM and HE storage domain.
Components on engine:
rhevm-setup-plugin-websocket-proxy-3.6.10.2-0.2.el6.noarch
rhevm-doc-3.6.10-1.el6ev.noarch
rhevm-webadmin-portal-3.6.10.2-0.2.el6.noarch
rhevm-spice-client-x64-cab-3.6-7.el6.noarch
rhevm-setup-plugins-3.6.5-1.el6ev.noarch
rhevm-setup-base-3.6.10.2-0.2.el6.noarch
rhevm-setup-plugin-ovirt-engine-common-3.6.10.2-0.2.el6.noarch
rhevm-websocket-proxy-3.6.10.2-0.2.el6.noarch
rhevm-dbscripts-3.6.10.2-0.2.el6.noarch
rhevm-branding-rhev-3.6.0-10.el6ev.noarch
rhevm-sdk-python-3.6.9.1-1.el6ev.noarch
rhevm-dependencies-3.6.1-1.el6ev.noarch
rhevm-tools-backup-3.6.10.2-0.2.el6.noarch
rhevm-restapi-3.6.10.2-0.2.el6.noarch
rhevm-spice-client-x86-cab-3.6-7.el6.noarch
rhevm-guest-agent-common-1.0.11-6.el6ev.noarch
rhevm-setup-3.6.10.2-0.2.el6.noarch
rhevm-image-uploader-3.6.1-2.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.6.10.2-0.2.el6.noarch
rhevm-tools-3.6.10.2-0.2.el6.noarch
rhevm-log-collector-3.6.1-1.el6ev.noarch
rhevm-backend-3.6.10.2-0.2.el6.noarch
rhevm-spice-client-x86-msi-3.6-7.el6.noarch
rhevm-lib-3.6.10.2-0.2.el6.noarch
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch
rhevm-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch
rhevm-extensions-api-impl-3.6.10.2-0.2.el6.noarch
rhevm-3.6.10.2-0.2.el6.noarch
rhev-guest-tools-iso-3.6-6.el6ev.noarch
rhevm-cli-3.6.9.0-1.el6ev.noarch
rhevm-userportal-3.6.10.2-0.2.el6.noarch
rhevm-spice-client-x64-msi-3.6-7.el6.noarch
rhevm-iso-uploader-3.6.0-1.el6ev.noarch
Linux version 2.6.32-642.el6.x86_64 (mockbuild@x86-033.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Wed Apr 13 00:51:26 EDT 2016
Linux 2.6.32-642.el6.x86_64 #1 SMP Wed Apr 13 00:51:26 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Components on hosts:
ovirt-hosted-engine-ha-1.3.5.9-2.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
mom-0.5.6-1.el7ev.noarch
vdsm-4.17.37-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.3-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-setup-lib-1.0.1-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
ovirt-host-deploy-1.4.1-1.el7ev.noarch
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)
2)Created guest-VM with bootable disk.
3)Set puma18 to maintenance from WEBUI and updated it to 4.1 repos, then updated it to latest 4.1 components e.g. "yum update ovirt-hosted-engine-setup -y", then "yum update -y" and then rebooted the host.
6)Host got updated to these components:
mom-0.5.9-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.19.6-1.el7ev.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
ovirt-hosted-engine-ha-2.1.0.3-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.3-1.el7ev.noarch
Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild@x86-039.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017
Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)
4)Activated puma18 via WEBUI and waited until it's score became positive (3400).
5)Selected puma18 as SPM.
6)Migrated HE-VM from puma19 (3.6 host) to puma18 (4.1 host).
7)Installed on puma18 rhevm-appliance-4.0.20170227.0-1.el7ev.noarch.rpm.
8)Created backup on engine "engine-backup --mode=backup --file=nsednev_from_puma19_rhevm_3_6 --log=Log_nsednev_from_puma19_rhevm_3_6", while environment is not in global maintenance, to verify bug 1403903.
9)Copied backup files to puma18 (4.1 host on which HE-VM was running) to "/backup" directory.
10)Set puma18 in global maintenance, so step 11 could be accomplished.
11)Ran hosted-engine --upgrade-appliance on puma18.
12)Successfully upgraded the engine from 3.6 to 4.0:
puma18 ~]# hosted-engine --upgrade-appliance
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
         
          ==================================================================================
          Welcome to the oVirt Self Hosted Engine setup/Upgrade tool.
         
          Please refer to the oVirt install guide:
          https://www.ovirt.org/documentation/how-to/hosted-engine/#fresh-install
          Please refer to the oVirt upgrade guide:
          https://www.ovirt.org/documentation/how-to/hosted-engine/#upgrade-hosted-engine
          ==================================================================================
          Continuing will upgrade the engine VM running on this hosts deploying and configuring a new appliance.
          If your engine VM is already based on el7 you can also simply upgrade the engine there.
          This procedure will create a new disk on the hosted-engine storage domain and it will backup there the content of your current engine VM disk.
          The new el7 based appliance will be deployed over the existing disk destroying its content; at any time you will be able to rollback using the content of the backup disk.
          You will be asked to take a backup of the running engine and copy it to this host.
          The engine backup will be automatically injected and recovered on the new appliance.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170305144438-4cclp3.log
          Version: otopi-1.6.0 (otopi-1.6.0-1.el7ev)
[ INFO  ] Detecting available oVirt engine appliances
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Checking maintenance mode
[ INFO  ] The engine VM is running on this host
[ INFO  ] Stage: Environment customization
[ INFO  ] Answer file successfully loaded
          Enter engine admin password: 
          Confirm engine admin password: 
[ INFO  ] Acquiring internal CA cert from the engine
[ INFO  ] The following CA certificate is going to be used, please immediately interrupt if not correct:
[ INFO  ] Issuer: C=US, O=scl.lab.tlv.redhat.com, CN=nsednev-he-4.scl.lab.tlv.redhat.com.89576, Subject: C=US, O=scl.lab.tlv.redhat.com, CN=nsednev-he-4.scl.lab.tlv.redhat.com.89576, Fingerprint (SHA-1): D7E41C9B457311C6635CB27EA6F1B1DF4B7ED0D6
[ INFO  ] Bridge ovirtmgmt already created
[WARNING] Unable to uniquely detect the interface where Bridge ovirtmgmt has been created on, [u'vnet0', u'enp4s0f0'] appear to be valid alternatives
[ INFO  ] Checking SPM status on this host
[ INFO  ] Connecting to the Engine
[ INFO  ] This upgrade tool is running on the SPM host
          The following appliance have been found on your system:
                [1] - The RHEV-M Appliance image (OVA) - 4.0.20170227.0-1.el7ev
                [2] - Directly select an OVA file
          Please select an appliance (1, 2) [1]: 
[ INFO  ] Verifying its sha1sum
[ INFO  ] Checking OVF archive content (could take a few minutes depending on archive size)
[ INFO  ] Checking OVF XML content (could take a few minutes depending on archive size)
          Please specify the size of the VM disk in GB: [50]: 
[ INFO  ] Connecting to the Engine
[ INFO  ] The hosted-engine storage domain has enough free space to contain a new backup disk.
[ INFO  ] Checking version requirements
[ INFO  ] Connecting to the Engine
[ INFO  ] All the datacenters and clusters are at a compatible level
[WARNING] Unsupported upgrade path
          This procedure has been designed and tested only for upgrading the engine VM from ['3.6'] to ['4.0'].
          Any other usage is highly experimental and potentially dangerous:
            Current engine: 3.6
            Selected appliance: 4.0.20170227.0-1.el7ev
          Do you want to abort the upgrade process? (Yes, No) [Yes]: no
[WARNING] Proceeding on an unsupported and highly experimental path
[ INFO  ] Detecting host timezone.
          Enter root password that will be used for the engine appliance (leave it empty to skip): 
          Confirm appliance root password: 
          Enter ssh public key for the root user that will be used for the engine appliance (leave it empty to skip): 
[WARNING] Skipping appliance root ssh public key
          Do you want to enable ssh access for the root user (yes, no, without-password) [yes]: 
          Please take a backup of the current engine running this command on the engine VM:
           engine-backup --mode=backup --archive-compressor=gzip --file=engine_backup.tar.gz --log=engine_backup.log
          Then copy the backup archive to this host and input here its path when ready.
          Please specify path to engine backup archive you would like to restore on the new appliance: /backup/nsednev_from_puma19_rhevm_3_6
[ INFO  ] Validating backup file '/backup/nsednev_from_puma19_rhevm_3_6'
[ INFO  ] Validating authentication plugins
[ INFO  ] Connecting to the Engine
[ INFO  ] '/backup/nsednev_from_puma19_rhevm_3_6' is a sane backup file
[WARNING] Please take care that this will simply add an entry for this host under /etc/hosts on the engine VM. If in the past you added other entries there, recovering them is up to you.
[ INFO  ] Stage: Setup validation
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Connecting to the Engine
[ INFO  ] Waiting for the engine to complete disk creation. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] The engine VM backup disk is now ready
[ INFO  ] Shutting down the current engine VM
[ INFO  ] Creating a backup of the engine VM disk (could take a few minutes depending on archive size)

[ INFO  ] Successfully created
[ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
[ INFO  ] Validating pre-allocated volume size
[ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
[ INFO  ] Injecting engine backup
[ INFO  ] Backup successfully injected
[ INFO  ] Image successfully imported from OVF
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Reloading vm.conf from the shared storage domain
[ INFO  ] Trying to get a fresher copy of vm configuration from the OVF_STORE
[ INFO  ] Found an OVF for HE VM, trying to convert
[ INFO  ] Got vm.conf from OVF_STORE
[ INFO  ] Running engine-setup on the appliance
          |- Preparing to restore:
          |- - Unpacking file '/root/nsednev_from_puma19_rhevm_3_6'
          |- Restoring:
          |- - Files
          |- ------------------------------------------------------------------------------
          |- Please note:
          |- Operating system is different from the one used during backup.
          |- Current operating system: redhat7
          |- Operating system at backup: redhat6
          |- Apache httpd configuration will not be restored.
          |- You will be asked about it on the next engine-setup run.
          |- ------------------------------------------------------------------------------
          |- Provisioning PostgreSQL users/databases:
          |- - user 'engine', database 'engine'
          |- Restoring:
          |- - Engine database 'engine'
          |-   - Cleaning up temporary tables in engine database 'engine'
          |-   - Inserting DbJustRestored VdcOption in engine database
          |- ------------------------------------------------------------------------------
          |- Please note:
          |- The engine database was backed up at 2017-03-05 14:25:50.000000000 +0200 .
          |- Objects that were added, removed or changed after this date, such as virtual
          |- machines, disks, etc., are missing in the engine, and will probably require
          |- recovery or recreation.
          |- ------------------------------------------------------------------------------
          |- You should now run engine-setup.
          |- Done.
          |- HE_APPLIANCE_ENGINE_RESTORE_SUCCESS
[ INFO  ] Engine backup successfully restored
          |- [ INFO  ] Stage: Initializing
          |- [ INFO  ] Stage: Environment setup
          |-           Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']
          |-           Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170305150314-tznjmq.log
          |-           Version: otopi-1.5.2 (otopi-1.5.2-1.el7ev)
          |- [ INFO  ] The engine DB has been restored from a backup
          |- [ INFO  ] Stage: Environment packages setup
          |- [ INFO  ] Stage: Programs detection
          |- [ INFO  ] Stage: Environment setup
          |- [ INFO  ] Stage: Environment customization
          |-          
          |-           Welcome to the RHEV 4.0 setup/upgrade.
          |-           Please read the RHEV 4.0 install guide
          |-           https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/4.0/html/Installation_Guide/index.html.
          |-           Please refer to the RHEV Upgrade Helper application
          |-           https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.
          |-           Would you like to proceed? (Yes, No) [Yes]: 
          |-          
          |-           --== PRODUCT OPTIONS ==--
          |-          
          |-           Configure Image I/O Proxy on this host? (Yes, No) [Yes]: 
          |-           Please note: Data Warehouse is required for the engine. If you choose to not configure it on this host, you have to configure it on a remote host, and then configure the engine on this host so that it can access the database of the remote Data Warehouse host.
          |-           Configure Data Warehouse on this host (Yes, No) [Yes]: 
          |-          
          |-           --== PACKAGES ==--
          |-          
          |-          
          |-           --== NETWORK CONFIGURATION ==--
          |-          
          |- [ INFO  ] firewalld will be configured as firewall manager.
          |-          
          |-           --== DATABASE CONFIGURATION ==--
          |-          
          |-           Where is the DWH database located? (Local, Remote) [Local]: 
          |-           Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.
          |-           Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]: 
          |-          
          |-           --== OVIRT ENGINE CONFIGURATION ==--
          |-          
          |-          
          |-           --== STORAGE CONFIGURATION ==--
          |-          
          |-          
          |-           --== PKI CONFIGURATION ==--
          |-          
          |-          
          |-           --== APACHE CONFIGURATION ==--
          |-          
          |-          
          |-           --== SYSTEM CONFIGURATION ==--
          |-          
          |-          
          |-           --== MISC CONFIGURATION ==--
          |-          
          |-           Please choose Data Warehouse sampling scale:
          |-           (1) Basic
          |-           (2) Full
          |-           (1, 2)[1]: 
          |-          
          |-           --== END OF CONFIGURATION ==--
          |-          
          |- [ INFO  ] Stage: Setup validation
          |- [ INFO  ] Cleaning stale zombie tasks and commands
          |-          
          |-           --== CONFIGURATION PREVIEW ==--
          |-          
          |-           Application mode                        : both
          |-           Default SAN wipe after delete           : False
          |-           Firewall manager                        : firewalld
          |-           Update Firewall                         : True
          |-           Host FQDN                               : nsednev-he-4.scl.lab.tlv.redhat.com
          |-           Engine database secured connection      : False
          |-           Engine database user name               : engine
          |-           Engine database name                    : engine
          |-           Engine database host                    : localhost
          |-           Engine database port                    : 5432
          |-           Engine database host name validation    : False
          |-           DWH database secured connection         : False
          |-           DWH database host                       : localhost
          |-           DWH database user name                  : ovirt_engine_history
          |-           DWH database name                       : ovirt_engine_history
          |-           DWH database port                       : 5432
          |-           DWH database host name validation       : False
          |-           Engine installation                     : True
          |-           PKI organization                        : scl.lab.tlv.redhat.com
          |-           Configure local Engine database         : True
          |-           Set application as default page         : True
          |-           Configure Apache SSL                    : True
          |-           DWH installation                        : True
          |-           Configure local DWH database            : True
          |-           Engine Host FQDN                        : nsednev-he-4.scl.lab.tlv.redhat.com
          |-           Configure Image I/O Proxy               : True
          |-           Configure VMConsole Proxy               : True
          |-           Configure WebSocket Proxy               : True
          |- [ INFO  ] Cleaning async tasks and compensations
          |- [ INFO  ] Unlocking existing entities
          |- [ INFO  ] Checking the Engine database consistency
          |- [ INFO  ] Stage: Transaction setup
          |- [ INFO  ] Stopping engine service
          |- [ INFO  ] Stopping ovirt-fence-kdump-listener service
          |- [ INFO  ] Stopping dwh service
          |- [ INFO  ] Stopping Image I/O Proxy service
          |- [ INFO  ] Stopping websocket-proxy service
          |- [ INFO  ] Stage: Misc configuration
          |- [ INFO  ] Stage: Package installation
          |- [ INFO  ] Stage: Misc configuration
          |- [ INFO  ] Upgrading CA
          |- [ INFO  ] Creating PostgreSQL 'ovirt_engine_history' database
          |- [ INFO  ] Configuring PostgreSQL
          |- [ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20170305150324.CXePP1.dump'.
          |- [ INFO  ] Creating/refreshing Engine database schema
          |- [ INFO  ] Creating/refreshing DWH database schema
          |- [ INFO  ] Configuring Image I/O Proxy
          |- [ INFO  ] Configuring WebSocket Proxy
          |- [ INFO  ] Creating/refreshing Engine 'internal' domain database schema
          |- [ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
          |- [ INFO  ] Stage: Transaction commit
          |- [ INFO  ] Stage: Closing up
          |- [ INFO  ] Starting engine service
          |- [ INFO  ] Starting dwh service
          |- [ INFO  ] Restarting ovirt-vmconsole proxy service
          |-          
          |-           --== SUMMARY ==--
          |-          
          |- [ INFO  ] Restarting httpd
          |-           Web access is enabled at:
          |-               http://nsednev-he-4.scl.lab.tlv.redhat.com:80/ovirt-engine
          |-               https://nsednev-he-4.scl.lab.tlv.redhat.com:443/ovirt-engine
          |-           Internal CA D7:E4:1C:9B:45:73:11:C6:63:5C:B2:7E:A6:F1:B1:DF:4B:7E:D0:D6
          |-           SSH fingerprint: 3f:ae:99:ac:54:9c:f2:4f:23:ec:bc:be:aa:5a:bf:77
          |-          
          |-           --== END OF SUMMARY ==--
          |-          
          |- [ INFO  ] Stage: Clean up
          |-           Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170305150314-tznjmq.log
          |- [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170305150417-setup.conf'
          |- [ INFO  ] Stage: Pre-termination
          |- [ INFO  ] Stage: Termination
          |- [ INFO  ] Execution of setup completed successfully
          |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
[ INFO  ] Engine-setup successfully completed 
[ INFO  ] Engine is still unreachable
[ INFO  ] Engine is still not reachable, waiting...
[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
[ INFO  ] Connecting to the Engine
[ INFO  ] Connecting to the Engine
[ INFO  ] Registering the hosted-engine backup disk in the DB
[ INFO  ] Waiting for the engine to complete disk registration. This may take several minutes...
[ INFO  ] The engine VM backup disk is now ready
[ INFO  ] Shutting down the engine VM
[ INFO  ] Stage: Clean up
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully upgraded
[ INFO  ] Please exit global maintenance mode to restart the engine VM.
[ INFO  ] Please note that the engine VM ssh keys have changed. Please remove the engine VM entry in ssh known_hosts on your clients.
You have new mail in /var/spool/mail/root
13)Removed environment from global maintenance e.g. "hosted-engine --set-maintenance --mode=none".

Components on upgraded 4.0 engine:
rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch
rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch
rhev-guest-tools-iso-4.0-7.el7ev.noarch
rhevm-setup-plugins-4.0.0.3-1.el7ev.noarch
rhevm-doc-4.0.7-1.el7ev.noarch
rhevm-dependencies-4.0.0-1.el7ev.noarch
rhevm-guest-agent-common-1.0.12-4.el7ev.noarch
rhevm-branding-rhev-4.0.0-7.el7ev.noarch
rhevm-4.0.7.3-0.1.el7ev.noarch
Linux version 3.10.0-514.6.2.el7.x86_64 (mockbuild@x86-017.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Feb 17 19:21:31 EST 2017
Linux 3.10.0-514.6.2.el7.x86_64 #1 SMP Fri Feb 17 19:21:31 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

14)Set puma19 to maintenance from WEBUI and updated it to 4.1 repos, then updated it to latest 4.1 components e.g. "yum update ovirt-hosted-engine-setup -y", then "yum update -y" and then rebooted the host.
15)Host got updated to these components:
mom-0.5.9-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.19.6-1.el7ev.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
ovirt-hosted-engine-ha-2.1.0.3-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.3-1.el7ev.noarch
Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild@x86-039.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017
Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)
16)Activated puma19 via WEBUI and waited until it's score became positive (3400).
17)Bumped up host cluster compatibility mode from 3.6 to 4.0.
18)Bumped up data center compatibility mode from 3.6 to 4.0.
19)Upgraded engine only from 4.0 to latest 4.1, using global maintenance on hosts.

Moving to verified.


Note You need to log in before you can comment on or make changes to this bug.