Bug 1398374 - [Text] The 3.6 engine backup tool should point users at the upgrade helper
Summary: [Text] The 3.6 engine backup tool should point users at the upgrade helper
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.9
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ovirt-3.6.10
: ---
Assignee: Ido Rosenzwig
QA Contact: Nikolai Sednev
URL:
Whiteboard:
: 1397409 (view as bug list)
Depends On:
Blocks: 1397409
TreeView+ depends on / blocked
 
Reported: 2016-11-24 15:24 UTC by Yedidyah Bar David
Modified: 2017-01-17 18:07 UTC (History)
17 users (show)

Fixed In Version: rhv-3.6.10-3
Doc Type: No Doc Update
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
Clone Of: 1397409
Environment:
Last Closed: 2017-01-17 18:07:03 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0109 0 normal SHIPPED_LIVE vdsm 3.6.10 bug fix and enhancement update 2017-01-17 22:48:48 UTC
oVirt gerrit 67278 0 None None None 2017-01-05 14:42:02 UTC

Comment 2 Yaniv Kaul 2016-12-08 14:24:07 UTC
Can we get it to 3.6.10? I won't wait for it, but I don't see why it cannot get in today even?

Comment 3 Yedidyah Bar David 2016-12-08 14:44:50 UTC
(In reply to Yaniv Kaul from comment #2)
> Can we get it to 3.6.10? I won't wait for it, but I don't see why it cannot
> get in today even?

I think we can - there is only a single patch pending merge. Alexander - anything missing? Thanks.

Comment 11 Sandro Bonazzola 2017-01-02 15:06:35 UTC
Not included in latest 3.6.10 build, postponing to 3.6.11.

Comment 12 Sandro Bonazzola 2017-01-04 08:42:15 UTC
Included in rhv-3.6.10-3 build

Comment 15 Nikolai Sednev 2017-01-04 20:35:13 UTC
1-Deployed HE on pair of 3.6's RHEVH7.2, over NFS and added two data storage domains.
2-Got hosted-storage auto-imported.

These are host's components while still in 3.6:
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.5.x86_64
ovirt-node-lib-config-3.6.1-15.0.el7ev.noarch
ovirt-node-selinux-3.6.1-15.0.el7ev.noarch
ovirt-node-plugin-snmp-logic-3.6.1-15.0.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.21.x86_64
vdsm-4.17.35-1.el7ev.noarch
ovirt-node-plugin-rhn-3.6.1-15.0.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
sanlock-3.2.4-1.el7.x86_64
ovirt-setup-lib-1.0.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-node-lib-legacy-3.6.1-15.0.el7ev.noarch
ovirt-node-lib-3.6.1-15.0.el7ev.noarch
ovirt-node-3.6.1-15.0.el7ev.noarch
ovirt-node-plugin-cim-logic-3.6.1-15.0.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.8-1.el7ev.noarch
ovirt-node-plugin-hosted-engine-0.3.0-8.el7ev.noarch
ovirt-host-deploy-offline-1.4.0-1.el7ev.x86_64
ovirt-node-plugin-vdsm-0.6.1-11.el7ev.noarch
ovirt-node-plugin-snmp-3.6.1-15.0.el7ev.noarch
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.3-1.el7ev.noarch
ovirt-node-plugin-cim-3.6.1-15.0.el7ev.noarch
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-node-branding-rhev-3.6.1-15.0.el7ev.noarch
Linux version 3.10.0-327.36.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Aug 17 03:02:37 EDT 2016
Linux 3.10.0-327.36.1.el7.x86_64 #1 SMP Wed Aug 17 03:02:37 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160920.1.el7ev)

Engine:
rhv-release-3.6.10-3-001.noarch
rhevm-setup-base-3.6.10.2-0.2.el6.noarch
rhevm-extensions-api-impl-3.6.10.2-0.2.el6.noarch
rhevm-backend-3.6.10.2-0.2.el6.noarch
ovirt-vmconsole-1.0.4-1.el6ev.noarch
rhevm-doc-3.6.10-1.el6ev.noarch
rhev-guest-tools-iso-3.6-6.el6ev.noarch
rhevm-spice-client-x86-cab-3.6-7.el6.noarch
rhevm-setup-plugin-ovirt-engine-3.6.10.2-0.2.el6.noarch
rhevm-dbscripts-3.6.10.2-0.2.el6.noarch
rhevm-reports-setup-3.6.5.1-1.el6ev.noarch
rhevm-image-uploader-3.6.1-2.el6ev.noarch
rhevm-spice-client-x86-msi-3.6-7.el6.noarch
rhevm-setup-plugins-3.6.5-1.el6ev.noarch
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch
rhevm-setup-3.6.10.2-0.2.el6.noarch
rhevm-tools-3.6.10.2-0.2.el6.noarch
rhevm-restapi-3.6.10.2-0.2.el6.noarch
rhevm-reports-3.6.5.1-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.7-2.el6ev.noarch
rhevm-iso-uploader-3.6.0-1.el6ev.noarch
rhevm-dependencies-3.6.1-1.el6ev.noarch
rhevm-guest-agent-common-1.0.11-6.el6ev.noarch
ovirt-vmconsole-proxy-1.0.4-1.el6ev.noarch
rhevm-branding-rhev-3.6.0-10.el6ev.noarch
rhevm-spice-client-x64-cab-3.6-7.el6.noarch
rhevm-setup-plugin-ovirt-engine-common-3.6.10.2-0.2.el6.noarch
rhevm-websocket-proxy-3.6.10.2-0.2.el6.noarch
rhevm-tools-backup-3.6.10.2-0.2.el6.noarch
rhevm-userportal-3.6.10.2-0.2.el6.noarch
rhevm-dwh-3.6.8-1.el6ev.noarch
rhevm-sdk-python-3.6.9.1-1.el6ev.noarch
rhevm-log-collector-3.6.1-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
rhevm-spice-client-x64-msi-3.6-7.el6.noarch
rhevm-setup-plugin-websocket-proxy-3.6.10.2-0.2.el6.noarch
rhevm-webadmin-portal-3.6.10.2-0.2.el6.noarch
ovirt-setup-lib-1.0.1-1.el6ev.noarch
rhevm-dwh-setup-3.6.8-1.el6ev.noarch
rhevm-cli-3.6.9.0-1.el6ev.noarch
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
rhevm-lib-3.6.10.2-0.2.el6.noarch
rhevm-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch
rhevm-3.6.10.2-0.2.el6.noarch
Linux version 2.6.32-642.11.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Wed Oct 26 10:25:23 EDT 2016
Linux 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.8 (Santiago)

3-Installed on the engine the latest rhev-hypervisor7.noarch 0:7.3-20161028.1.el6ev
4-Updated both hosts from RHEV Hypervisor - 7.2 - 20160920.1.el7ev 
OS Version:
RHEV Hypervisor - 7.2 - 20160920.1.el7ev
Kernel Version:
3.10.0 - 327.36.1.el7.x86_64
KVM Version:
2.3.0 - 31.el7_2.21
LIBVIRT Version:
libvirt-1.2.17-13.el7_2.5
VDSM Version:
vdsm-4.17.35-1.el7ev
SPICE Version:
0.12.4 - 15.el7_2.2
GlusterFS Version:
[N/A]
CEPH Version:
librbd1-0.94.5-15.el7cp

to 
OS Version:
RHEV Hypervisor - 7.3 - 20161028.1.el7ev
Kernel Version:
3.10.0 - 514.el7.x86_64
KVM Version:
2.6.0 - 27.el7
LIBVIRT Version:
libvirt-2.0.0-10.el7
VDSM Version:
vdsm-4.17.35-1.el7ev
SPICE Version:
0.12.4 - 19.el7
GlusterFS Version:
[N/A]
CEPH Version:
librbd1-0.94.5-1.el7

ovirt-node-lib-3.6.1-34.0.el7ev.noarch
ovirt-node-plugin-cim-logic-3.6.1-34.0.el7ev.noarch
qemu-kvm-rhev-2.6.0-27.el7.x86_64
ovirt-node-plugin-hosted-engine-0.3.0-8.el7ev.noarch
ovirt-node-plugin-vdsm-0.6.1-11.el7ev.noarch
ovirt-node-plugin-cim-3.6.1-34.0.el7ev.noarch
ovirt-node-branding-rhev-3.6.1-34.0.el7ev.noarch
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
libvirt-client-2.0.0-10.el7.x86_64
ovirt-node-lib-legacy-3.6.1-34.0.el7ev.noarch
ovirt-node-selinux-3.6.1-34.0.el7ev.noarch
ovirt-node-plugin-snmp-logic-3.6.1-34.0.el7ev.noarch
mom-0.5.6-1.el7ev.noarch
vdsm-4.17.35-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.3-1.el7ev.noarch
ovirt-node-plugin-rhn-3.6.1-34.0.el7ev.noarch
ovirt-node-plugin-snmp-3.6.1-34.0.el7ev.noarch
sanlock-3.2.4-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
ovirt-node-lib-config-3.6.1-34.0.el7ev.noarch
ovirt-node-3.6.1-34.0.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.8-1.el7ev.noarch
ovirt-host-deploy-offline-1.4.0-1.el7ev.x86_64
ovirt-setup-lib-1.0.1-1.el7ev.noarch
Linux version 3.10.0-514.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:24:13 EDT 2016
Linux 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Virtualization Hypervisor release 7.3 (20161028.1.el7ev)

4-Redeployed second host as 4.0.6 RHEL7.3, to get mixed environment, components on RHEL host are:
rhevm-appliance-20161130.0-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
ovirt-hosted-engine-setup-2.0.4.1-2.el7ev.noarch
ovirt-host-deploy-1.5.3-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-imageio-daemon-0.4.0-0.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.2.x86_64
rhev-release-4.0.6-8-001.noarch
sanlock-3.4.0-1.el7.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64
ovirt-hosted-engine-ha-2.0.6-1.el7ev.noarch
rhevm-appliance-20161130.0-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
vdsm-4.18.21-1.el7ev.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.2.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Nov 16 13:15:13 EST 2016
Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Wed Nov 16 13:15:13 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

OS Version:
RHEL - 7.3 - 7.el7
Kernel Version:
3.10.0 - 514.2.2.el7.x86_64
KVM Version:
2.6.0 - 28.el7_3.2
LIBVIRT Version:
libvirt-2.0.0-10.el7_3.2
VDSM Version:
vdsm-4.18.21-1.el7ev
SPICE Version:
0.12.4 - 19.el7
GlusterFS Version:
[N/A]
CEPH Version:
librbd1-0.94.5-1.el7

5-Followed the https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#Upgrading_the_Self-Hosted_Engine to upgrade my HE mixed (RHEVH&RHEL) environment environment.
Engine was running on 4.0.6 RHEL7.3 host, while it was SPM.
Engine had disk at the size of 200Gig, while was still in 3.6.10
# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
vda    252:0    0   200G  0 disk 
├─vda1 252:1    0   200M  0 part /boot
├─vda2 252:2    0     2G  0 part [SWAP]
└─vda3 252:3    0 197.8G  0 part /
sr0     11:0    1  1024M  0 rom  


6-Failed to get engine upgraded:
 
alma04 /]# hosted-engine --upgrade-appliance
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will upgrade the engine VM running on this hosts deploying and configuring a new appliance.
          If your engine VM is already based on el7 you can also simply upgrade the engine there.
          This procedure will create a new disk on the hosted-engine storage domain and it will backup there the content of your current engine VM disk.
          The new el7 based appliance will be deployed over the existing disk destroying its content; at any time you will be able to rollback using the content of the backup disk.
          You will be asked to take a backup of the running engine and copy it to this host.
          The engine backup will be automatically injected and recovered on the new appliance.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170104203619-yd3320.log
          Version: otopi-1.5.2 (otopi-1.5.2-1.el7ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Checking maintenance mode
[ INFO  ] The engine VM is running on this host
[ INFO  ] Stage: Environment customization
[ INFO  ] Answer file successfully loaded
          Enter engine admin password: 
          Confirm engine admin password: 
[ INFO  ] Acquiring internal CA cert from the engine
[ INFO  ] The following CA certificate is going to be used, please immediately interrupt if not correct:
[ INFO  ] Issuer: C=US, O=qa.lab.tlv.redhat.com, CN=nsednev-he-1.qa.lab.tlv.redhat.com.28618, Subject: C=US, O=qa.lab.tlv.redhat.com, CN=nsednev-he-1.qa.lab.tlv.redhat.com.28618, Fingerprint (SHA-1): 8CACF3CA06BECBC9F4DCB2F4E74BF2EDED3EBEC5
[ INFO  ] Bridge ovirtmgmt already created
[ INFO  ] Checking SPM status on this host
[ INFO  ] Connecting to the Engine
[ INFO  ] This upgrade tool is running on the SPM host
[ INFO  ] Detecting available oVirt engine appliances
          The following appliance have been found on your system:
                [1] - The RHEV-M Appliance image (OVA) - 20161130.0-1.el7ev
                [2] - Directly select an OVA file
          Please select an appliance (1, 2) [1]: 
[ INFO  ] Verifying its sha1sum
[ INFO  ] Checking OVF archive content (could take a few minutes depending on archive size)
[ INFO  ] Checking OVF XML content (could take a few minutes depending on archive size)
[WARNING] OVF does not contain a valid image description, using default.
[ INFO  ] Connecting to the Engine
[ INFO  ] The hosted-engine storage domain has enough free space to contain a new backup disk.
          Please take a backup of the current engine running this command on the engine VM:
           engine-backup --mode=backup --archive-compressor=gzip --file=engine_backup.tar.gz --log=engine_backup.log
          Then copy the backup archive to this host and input here its path when ready.
          Please specify path to engine backup archive you would like to restore on the new appliance: /backup/backup.bck
[ INFO  ] Validating backup file '/backup/backup.bck'
[ INFO  ] The provided file contains also a DWH DB backup: it will be restored as well
[ INFO  ] The provided file contains also a Reports DB backup: it will be restored as well
[ INFO  ] Validating authentication plugins
[ INFO  ] Connecting to the Engine
[ INFO  ] '/backup/backup.bck' is a sane backup file
[ INFO  ] Checking version requirements
[ INFO  ] Connecting to the Engine
[ INFO  ] All the datacenters and clusters are at a compatible level
          Please provide the FQDN you would like to use for the engine appliance.
          Note: This will be the FQDN of the engine VM you are now going to launch,
          it should not point to the base host or to any other existing machine.
          Engine VM FQDN: (leave it empty to skip):  []: nsednev-he-1.qa.lab.tlv.redhat.com
          Please provide the domain name you would like to use for the engine appliance.
          Engine VM domain: [qa.lab.tlv.redhat.com]
          Enter root password that will be used for the engine appliance (leave it empty to skip): 
          Confirm appliance root password: 
          How should the engine VM network be configured (DHCP, Static)[DHCP]? 
          Add lines for the appliance itself and for this host to /etc/hosts on the engine VM?
          Note: ensuring that this host could resolve the engine VM hostname is still up to you
          (Yes, No)[No] yes
[WARNING] Please take care that this will simply add an entry for this host under /etc/hosts on the engine VM. If in the past you added other entries there, recovering them is up to you.
[ INFO  ] Stage: Setup validation
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Connecting to the Engine
[ INFO  ] Waiting for the engine to complete disk creation. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] Still waiting for engine VM backup disk to be created. This may take several minutes...
[ INFO  ] The engine VM backup disk is now ready
[ INFO  ] Shutting down the current engine VM
[ INFO  ] Creating a backup of the engine VM disk (could take a few minutes depending on archive size)


[ INFO  ] Successfully created
[ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
[ INFO  ] Validating pre-allocated volume size
[ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
[ INFO  ] Injecting engine backup
[ INFO  ] Backup successfully injected
[ INFO  ] Image successfully imported from OVF
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Trying to get a fresher copy of vm configuration from the OVF_STORE
[ INFO  ] Found an OVF for HE VM, trying to convert
[ INFO  ] Got vm.conf from OVF_STORE
[ INFO  ] Running engine-setup on the appliance
          |- Preparing to restore:
          |- - Unpacking file '/root/backup.bck'
          |- ------------------------------------------------------------------------------
          |- Please note:
          |- A Reports dump was found inside the backup.
          |- This version does not support Reports, and the dump and configuration files
          |- will not be restored.
          |- ------------------------------------------------------------------------------
          |- Restoring:
          |- - Files
          |- ------------------------------------------------------------------------------
          |- Please note:
          |- Operating system is different from the one used during backup.
          |- Current operating system: redhat7
          |- Operating system at backup: redhat6
          |- Apache httpd configuration will not be restored.
          |- You will be asked about it on the next engine-setup run.
          |- ------------------------------------------------------------------------------
          |- Provisioning PostgreSQL users/databases:
          |- - user 'engine', database 'engine'
          |- - user 'ovirt_engine_history', database 'ovirt_engine_history'
          |- Restoring:
          |- - Engine database 'engine'
          |-   - Cleaning up temporary tables in engine database 'engine'
          |-   - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine database
          |- ------------------------------------------------------------------------------
          |- Please note:
          |- The engine database was backed up at 2017-01-04 13:30:11.000000000 -0500 .
          |- Objects that were added, removed or changed after this date, such as virtual
          |- machines, disks, etc., are missing in the engine, and will probably require
          |- recovery or recreation.
          |- ------------------------------------------------------------------------------
          |- - DWH database 'ovirt_engine_history'
          |- You should now run engine-setup.
          |- Done.
          |- HE_APPLIANCE_ENGINE_RESTORE_SUCCESS
[ INFO  ] Engine backup successfully restored
          |- [ INFO  ] Stage: Initializing
          |- [ INFO  ] Stage: Environment setup
          |-           Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']
          |-           Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170104151505-edmf66.log
          |-           Version: otopi-1.5.2 (otopi-1.5.2-1.el7ev)
          |- [ INFO  ] Stage: Environment packages setup
          |- [ INFO  ] Stage: Programs detection
          |- [ INFO  ] Stage: Environment setup
          |- [ INFO  ] Stage: Environment customization
          |-          
          |-           Welcome to the RHEV 4.0 setup/upgrade.
          |-           Please read the RHEV 4.0 install guide
          |-           https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/4.0/html/Installation_Guide/index.html.
          |-           Please refer to the RHEV Upgrade Helper application
          |-           https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.
          |-           Would you like to proceed? (Yes, No) [Yes]: 
          |-          
          |-           --== PRODUCT OPTIONS ==--
          |-          
          |-           Configure Image I/O Proxy on this host? (Yes, No) [Yes]: 
          |-          
          |-           --== PACKAGES ==--
          |-          
          |-          
          |-           --== NETWORK CONFIGURATION ==--
          |-          
          |- [ INFO  ] firewalld will be configured as firewall manager.
          |-          
          |-           --== DATABASE CONFIGURATION ==--
          |-          
          |-           The detected DWH database size is 25 MB.
          |-           Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.
          |-           If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.
          |-           Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: 
          |-          
          |-           --== OVIRT ENGINE CONFIGURATION ==--
          |-          
          |-          
          |-           --== STORAGE CONFIGURATION ==--
          |-          
          |-          
          |-           --== PKI CONFIGURATION ==--
          |-          
          |-          
          |-           --== APACHE CONFIGURATION ==--
          |-          
          |-          
          |-           --== SYSTEM CONFIGURATION ==--
          |-          
          |-          
          |-           --== MISC CONFIGURATION ==--
          |-          
          |-          
          |-           --== END OF CONFIGURATION ==--
          |-          
          |- [ INFO  ] Stage: Setup validation
          |- [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. 
          |- [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set.
          |- [ INFO  ] Stage: Clean up
          |-           Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170104151505-edmf66.log
          |- [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170104151507-setup.conf'
          |- [ INFO  ] Stage: Pre-termination
          |- [ INFO  ] Stage: Termination
          |- [ ERROR ] Execution of setup failed
          |- HE_APPLIANCE_ENGINE_SETUP_FAIL
[ ERROR ] Engine setup failed on the appliance
[ ERROR ] Failed to execute stage 'Closing up': Engine setup failed on the appliance Please check its log on the appliance. 
[ INFO  ] Stage: Clean up
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine upgrade failed: this system is not reliable, you can use --rollback-upgrade option to recover the engine VM disk from a backup
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170104203619-yd3320.log

alma04 /]# hosted-engine --vm-status


!! Cluster is in GLOBAL MAINTENANCE mode !!



--== Host 1 status ==--

Status up-to-date                  : True
Hostname                           : alma03.qa.lab.tlv.redhat.com
Host ID                            : 1
Engine status                      : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : fe28ba5d
Host timestamp                     : 17034
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=17034 (Wed Jan  4 20:26:01 2017)
        host-id=1
        score=3400
        maintenance=False
        state=GlobalMaintenance
        stopped=False


--== Host 2 status ==--

Status up-to-date                  : True
Hostname                           : alma04.qa.lab.tlv.redhat.com
Host ID                            : 2
Engine status                      : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : de9050c2
Host timestamp                     : 12977
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=12977 (Wed Jan  4 22:25:40 2017)
        host-id=2
        score=3400
        maintenance=False
        state=GlobalMaintenance
        stopped=False


!! Cluster is in GLOBAL MAINTENANCE mode !!

7-Upgrade helper being mentioned during the upgrade.

Moving this bug to verified and opening another to investigate the failure.

Comment 16 Nikolai Sednev 2017-01-04 20:40:43 UTC
Setup failure was probably caused by backup data, which was taken from engine, while environment was not in global maintenance, so when upgraded, environment had to be set in to the global maintenance and then during the upgrade at latest steps, data from backup of the engine was not aligned with the environment status.
In backup the whole environment was not in global maintenance, while during the upgrade environment was in global maintenance, so this mismatch caused for the failure.
In order to avoid such failure, backup must be collected on engine, while whole environment is already in global maintenance and only then proceed to the upgrade.

Comment 17 Nikolai Sednev 2017-01-04 20:45:05 UTC
Adding the https://bugzilla.redhat.com/show_bug.cgi?id=1403903 here, forth to Comment #16.

Comment 18 Nikolai Sednev 2017-01-04 22:01:21 UTC
Just in case of issue happened in Comment #15, there is always a way back to return to previous engine's disk, using "hosted-engine --rollback-upgrade" functionality from host:

alma04 /]# hosted-engine --rollback-upgrade
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will rollback the engine VM from a previous upgrade attempt.
          This procedure will restore an engine VM image from a backup taken during an upgrade attempt.
          The result of any action occurred after the backup creation instant could be definitively lost.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170104225550-hj1gyt.log
          Version: otopi-1.5.2 (otopi-1.5.2-1.el7ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Checking maintenance mode
[ INFO  ] The engine VM is down.
[ INFO  ] Stage: Environment customization
[ INFO  ] Answer file successfully loaded
          The following backup disk have been found on your system:
                [1] - hosted-engine-backup-20170104203956
          Please select one of them  (1) [1]: 
[ INFO  ] Bridge ovirtmgmt already created
[ INFO  ] Stage: Setup validation
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Restoring a backup of the engine VM disk (could take a few minutes depending on archive size)
[ INFO  ] Successfully restored
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Stage: Clean up
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully rolled back
[ INFO  ] Please exit global maintenance mode to restart the engine VM.
[ INFO  ] Please note that the engine VM ssh keys have changed. Please remove the engine VM entry in ssh known_hosts on your clients.

Restore was successful.

Comment 19 Yedidyah Bar David 2017-01-05 08:19:02 UTC
This bug is about backup, not upgrade. The flow is:

Run engine-backup --mode=backup on 3.6.

It should mention the upgrade helper. Did it? Can't see this above.

This can, but doesn't have to, be part of upgrade:
1. Run the upgrade tool on 3.6
2. It tells you to run engine-backup as above
etc.

Comment 20 Nikolai Sednev 2017-01-05 09:06:04 UTC
     |- [ INFO  ] Stage: Environment customization
          |-          
          |-           Welcome to the RHEV 4.0 setup/upgrade.
          |-           Please read the RHEV 4.0 install guide
          |-           https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/4.0/html/Installation_Guide/index.html.
          |-           Please refer to the RHEV Upgrade Helper application
          |-           https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.
          |-           Would you like to proceed? (Yes, No) [Yes]:

Comment 21 Yedidyah Bar David 2017-01-05 13:45:36 UTC
(In reply to Nikolai Sednev from comment #20)
>      |- [ INFO  ] Stage: Environment customization
>           |-          
>           |-           Welcome to the RHEV 4.0 setup/upgrade.
>           |-           Please read the RHEV 4.0 install guide
>           |-          
> https://access.redhat.com/site/documentation/en-US/
> Red_Hat_Enterprise_Virtualization/4.0/html/Installation_Guide/index.html.
>           |-           Please refer to the RHEV Upgrade Helper application
>           |-           https://access.redhat.com/labs/rhevupgradehelper/
> which will guide you in the upgrading process.
>           |-           Would you like to proceed? (Yes, No) [Yes]:

This is 4.0 engine-setup.

(In reply to Nikolai Sednev from comment #15)
>           Please take a backup of the current engine running this command on
> the engine VM:
>            engine-backup --mode=backup --archive-compressor=gzip
> --file=engine_backup.tar.gz --log=engine_backup.log
>           Then copy the backup archive to this host and input here its path
> when ready.
>           Please specify path to engine backup archive you would like to
> restore on the new appliance: /backup/backup.bck

You were asked here, by the hosted-engine migration tool, to run engine-backup on the 3.6 engine. You apparently did, and provided the result, but didn't include the output-to-terminal it generated, so we can't see if you got a similar message. Did you?

Comment 22 Yaniv Lavi 2017-01-05 14:42:14 UTC
*** Bug 1397409 has been marked as a duplicate of this bug. ***

Comment 23 Nikolai Sednev 2017-01-05 15:12:59 UTC
[root@nsednev-he-1 ~]# engine-backup --mode=backup --archive-compressor=gzip --file=engine_backup.tar.gz --log=engine_backup.log
Backing up:
Notifying engine
- Files
- Engine database 'engine'
- DWH database 'ovirt_engine_history'
- Reports database 'ovirt_engine_reports'
Packing into file 'engine_backup.tar.gz'
Notifying engine
Done.
------------------------------------------------------------------------
Please note:

If you are running engine-backup as part of a migration to 4.0,
please refer to the Red Hat Virtualization Upgrade Helper:

https://access.redhat.com/labsinfo/rhevupgradehelper
------------------------------------------------------------------------
[root@nsednev-he-1 ~]# ll
total 17592
-rw-------. 1 root root    5541 Jan  3 17:46 anaconda-ks.cfg
-rw-r--r--. 1 root root 8964644 Jan  4 23:56 backup
-rw-r--r--. 1 root root    3665 Jan  4 23:56 engine_backup.log
-rw-r--r--. 1 root root 8964720 Jan  4 23:56 engine_backup.tar.gz
-rw-r--r--. 1 root root   17899 Jan  3 17:46 install.log
-rw-r--r--. 1 root root    6046 Jan  3 17:45 install.log.syslog
-rw-------. 1 root root    3402 Jan  3 19:46 kickstart-default-provision.log
-rw-r--r--. 1 root root    3567 Jan  4 23:56 Log_backup

Comment 25 errata-xmlrpc 2017-01-17 18:07:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0109.html


Note You need to log in before you can comment on or make changes to this bug.