Description of problem: Engine setup failed on the appliance. Followed steps from https://bugzilla.redhat.com/show_bug.cgi?id=1399053 in order to verify it, but failed with: [ ERROR ] Engine setup failed on the appliance [ ERROR ] Failed to execute stage 'Closing up': Engine setup failed on the appliance Please check its log on the appliance. [ ERROR ] Hosted Engine upgrade failed: this system is not reliable, you can use --rollback-upgrade option to recover the engine VM disk from a backup Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20161212162707-5ujgti.log Version-Release number of selected component (if applicable): Host: rhevm-appliance-20161130.0-1.el7ev.noarch vdsm-4.18.18-4.git198e48d.el7ev.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-imageio-daemon-0.4.0-0.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-vmconsole-host-1.0.4-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64 ovirt-hosted-engine-ha-2.0.6-1.el7ev.noarch ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch libvirt-client-2.0.0-10.el7_3.2.x86_64 ovirt-vmconsole-1.0.4-1.el7ev.noarch mom-0.5.8-1.el7ev.noarch ovirt-hosted-engine-setup-2.0.4.1-2.el7ev.noarch ovirt-host-deploy-1.5.3-1.el7ev.noarch Linux version 3.10.0-514.2.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Nov 16 13:15:13 EST 2016 Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Wed Nov 16 13:15:13 EST 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) How reproducible: 100% Steps to Reproduce: 1.Get 3.4.0.6 HE deployed on host with two NFS data storage domains and over NFS. 2.Make backup for your engine, copy it to your host to none root directory, while your host set to maintenance. 3.Install rhevm-appliance-20161130.0-1.el7ev.noarch on your host and run hosted-engine --upgrade-appliance Actual results: Upgrade failed. Expected results: Upgrade should pass. Additional info: See the attached sosreport from host and ovirt-hosted-engine-setup-20161212162707-5ujgti.log
Created attachment 1230829 [details] ovirt-hosted-engine-setup-20161212162707-5ujgti.log
Adding sosreport from host here https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88NDByaFhiZ3NwMW8/view?usp=sharing
1. What's the difference between this bz and bug 1399053? 2. Which upgrade documentation are you following?
(In reply to Doron Fediuck from comment #3) > 1. What's the difference between this bz and bug 1399053? > 2. Which upgrade documentation are you following? 1.I was trying to verify the 1399053, but as I've failed, so opened this bug. 2.I'm following the same steps for engine's db backup, which worked before, e.g. making engine db's backup, copying those to host on which I'm trying to make an upgrade, then during upgrade of the engine, providing the path to backup files on host.
(In reply to Nikolai Sednev from comment #4) > (In reply to Doron Fediuck from comment #3) > > 1. What's the difference between this bz and bug 1399053? > > 2. Which upgrade documentation are you following? > > 1.I was trying to verify the 1399053, but as I've failed, so opened this bug. This is indeed a different issue. The root cause here is: 2016-12-12 16:27:07 DEBUG otopi.plugins.otopi.packagers.dnfpackager dnfpackager._boot:163 Cannot initialize minidnf Traceback (most recent call last): File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 150, in _boot constants.PackEnv.DNF_DISABLED_PLUGINS File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 60, in _getMiniDNF from otopi import minidnf File "/usr/lib/python2.7/site-packages/otopi/minidnf.py", line 16, in <module> import dnf ImportError: No module named dnf > 2.I'm following the same steps for engine's db backup, which worked before, > e.g. making engine db's backup, copying those to host on which I'm trying to > make an upgrade, then during upgrade of the engine, providing the path to > backup files on host. We're now ensuring upgrade flows use the supported documentation, since there are multiple environments (el6/el7, HE/non-HE, 3.5/3.6). So unless this is a standard backup and restore (to the same version) please use only the relevant upgrade flow.
(In reply to Doron Fediuck from comment #5) > (In reply to Nikolai Sednev from comment #4) > > (In reply to Doron Fediuck from comment #3) > > > 1. What's the difference between this bz and bug 1399053? > > > 2. Which upgrade documentation are you following? > > > > 1.I was trying to verify the 1399053, but as I've failed, so opened this bug. > > This is indeed a different issue. The root cause here is: > 2016-12-12 16:27:07 DEBUG otopi.plugins.otopi.packagers.dnfpackager > dnfpackager._boot:163 Cannot initialize minidnf > Traceback (most recent call last): > File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 150, > in _boot > constants.PackEnv.DNF_DISABLED_PLUGINS > File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 60, > in _getMiniDNF > from otopi import minidnf > File "/usr/lib/python2.7/site-packages/otopi/minidnf.py", line 16, in > <module> > import dnf > ImportError: No module named dnf > > > 2.I'm following the same steps for engine's db backup, which worked before, > > e.g. making engine db's backup, copying those to host on which I'm trying to > > make an upgrade, then during upgrade of the engine, providing the path to > > backup files on host. > We're now ensuring upgrade flows use the supported documentation, since > there are multiple environments (el6/el7, HE/non-HE, 3.5/3.6). So unless > this is a standard backup and restore (to the same version) please use only > the relevant upgrade flow. 2.In this case I've tried backing up and restoring the same db on the same appliance, just to see if /root issue was fixed, before getting a bit far and make the whole upgrade flow.
(In reply to Doron Fediuck from comment #5) > This is indeed a different issue. The root cause here is: > 2016-12-12 16:27:07 DEBUG otopi.plugins.otopi.packagers.dnfpackager > dnfpackager._boot:163 Cannot initialize minidnf > Traceback (most recent call last): > File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 150, > in _boot > constants.PackEnv.DNF_DISABLED_PLUGINS > File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 60, > in _getMiniDNF > from otopi import minidnf > File "/usr/lib/python2.7/site-packages/otopi/minidnf.py", line 16, in > <module> > import dnf > ImportError: No module named dnf This is not critical. In RHEL and CentOS there's no dnf and yum is used as fallback.
As per instructions provided on the failure: RuntimeError: Engine setup failed on the appliance Please check its log on the appliance. Can you please attach engine setup logs from within the appliance?
Created attachment 1231169 [details] ovirt-engine-setup-20161212190249-4t6x44.log
The real issue is here: 2016-12-12 19:02:49 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:164 dbenv: {'OVESETUP_DWH_DB/database': 'ovirt_engine_history', 'OVESETUP_DWH_DB/host': 'localhost', 'OVESETUP_DWH_DB/port': 5432, 'OVESETUP_DWH_DB/securedHostValidation': False, 'OVESETUP_DWH_DB/secured': False, 'OVESETUP_DWH_DB/password': '1', 'OVESETUP_DWH_DB/user': 'ovirt_engine_history'} 2016-12-12 19:02:49 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:177 Database: 'None', Statement: ' select count(*) as count from pg_catalog.pg_tables where schemaname = 'public'; ', args: {} 2016-12-12 19:02:49 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:182 Creating own connection 2016-12-12 19:02:49 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:189 database connection failed Traceback (most recent call last): File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 187, in getCredentials ] = self.isNewDatabase() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 370, in isNewDatabase transaction=False, File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 191, in execute database=database, File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 125, in connect sslmode=sslmode, File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect conn = _connect(dsn, connection_factory=connection_factory, async=async) OperationalError: could not connect to server: Connection refused Is the server running on host "localhost" (::1) and accepting TCP/IP connections on port 5432? could not connect to server: Connection refused Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5432?
Nikolay, you uploaded the wrong engine-setup log file. The real issue was here on hosted-engine-setup logs: 2016-12-12 16:57:16 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND |- [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. hosted-engine-setup checked for global maintenance mode and it was fine: 2016-12-12 16:27:11 INFO otopi.plugins.gr_he_common.vm.misc misc._late_setup:65 Checking maintenance mode 2016-12-12 16:27:11 DEBUG otopi.plugins.gr_he_common.vm.misc misc._late_setup:68 hosted-engine-status: {'engine_vm_up': True, 'all_host_stats': {1: {'live-data': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=99379 (Mon Dec 12 16:26:40 2016)\nhost-id=1\nscore=3400\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n', 'hostname': 'alma04.qa.lab.tlv.redhat.com', 'host-id': 1, 'engine-status': '{"health": "good", "vm": "up", "detail": "up"}', 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'c315b57c', 'host-ts': 99379}}, 'engine_vm_host': 'alma04.qa.lab.tlv.redhat.com', 'global_maintenance': True} The point is the engine-setup is checking for global-maintenance mode in the restored DB and not on the host. So, if you really took the DB when hosted-engine-setup asked you to create a DB you are fine. Nikolay, were you trying to restore a previous DB took outside global maintenance mode? The open gap is that engine-backup is not able to restore a DB took on a hosted-engine env when not in maintenance mode.
I've manually created db backup on engine, using " engine-backup --mode=backup --file=nsednev1 --log=Log_1" I've took db backup from engine and copied it to host's none root folder, while host was not in global maintenance, then I've set host to global maintenance. When I was trying to make an upgrade, db had inside of it information about host, which was not in global maintenance.
If the user follows the instruction and he creates the backup when asked by hosted-engine-setup everything will be fine.
(In reply to Simone Tiraboschi from comment #13) > If the user follows the instruction and he creates the backup when asked by > hosted-engine-setup everything will be fine. That's true for migration. For backup/restore, we do not require maint during backup. Perhaps we should, so far we tried not to.
IMHO the docs should be updated at https://access.redhat.com/labs/rhevupgradehelper/, the first step there is "Step 1: Stop the ovirt engine service.", while "Step 4: Disable the high-availability agents on all the self-hosted engine hosts. To do this run the following command on any host in the cluster." should come first.
Also after engine's db was copied to host, the engine's service should get started back, otherwise it will fail on "hosted-engine --upgrade-appliance", as service must be up on engine during the upgrade.
What is the expected upgrade flow? 36.x->4.0.x->4.1.x or 3.6.x->4.1.x?
See (In reply to Nikolai Sednev from comment #18) > What is the expected upgrade flow? > 36.x->4.0.x->4.1.x or 3.6.x->4.1.x? See https://bugzilla.redhat.com/1420283
1)upgrade from 4.1 host to 4.0 engine won't work as environment is still 3.6 due to https://bugzilla.redhat.com/show_bug.cgi?id=1403846.
1.Deployed clean HE over two 3.6RHEL7.3 hosts, over NFS and added 2 data storage domains over NFS, got auto-imported HE-VM and HE storage domain. Components on engine: rhevm-setup-plugin-websocket-proxy-3.6.10.2-0.2.el6.noarch rhevm-doc-3.6.10-1.el6ev.noarch rhevm-webadmin-portal-3.6.10.2-0.2.el6.noarch rhevm-spice-client-x64-cab-3.6-7.el6.noarch rhevm-setup-plugins-3.6.5-1.el6ev.noarch rhevm-setup-base-3.6.10.2-0.2.el6.noarch rhevm-setup-plugin-ovirt-engine-common-3.6.10.2-0.2.el6.noarch rhevm-websocket-proxy-3.6.10.2-0.2.el6.noarch rhevm-dbscripts-3.6.10.2-0.2.el6.noarch rhevm-branding-rhev-3.6.0-10.el6ev.noarch rhevm-sdk-python-3.6.9.1-1.el6ev.noarch rhevm-dependencies-3.6.1-1.el6ev.noarch rhevm-tools-backup-3.6.10.2-0.2.el6.noarch rhevm-restapi-3.6.10.2-0.2.el6.noarch rhevm-spice-client-x86-cab-3.6-7.el6.noarch rhevm-guest-agent-common-1.0.11-6.el6ev.noarch rhevm-setup-3.6.10.2-0.2.el6.noarch rhevm-image-uploader-3.6.1-2.el6ev.noarch rhevm-setup-plugin-ovirt-engine-3.6.10.2-0.2.el6.noarch rhevm-tools-3.6.10.2-0.2.el6.noarch rhevm-log-collector-3.6.1-1.el6ev.noarch rhevm-backend-3.6.10.2-0.2.el6.noarch rhevm-spice-client-x86-msi-3.6-7.el6.noarch rhevm-lib-3.6.10.2-0.2.el6.noarch rhevm-setup-plugin-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch rhevm-vmconsole-proxy-helper-3.6.10.2-0.2.el6.noarch rhevm-extensions-api-impl-3.6.10.2-0.2.el6.noarch rhevm-3.6.10.2-0.2.el6.noarch rhev-guest-tools-iso-3.6-6.el6ev.noarch rhevm-cli-3.6.9.0-1.el6ev.noarch rhevm-userportal-3.6.10.2-0.2.el6.noarch rhevm-spice-client-x64-msi-3.6-7.el6.noarch rhevm-iso-uploader-3.6.0-1.el6ev.noarch Linux version 2.6.32-642.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Wed Apr 13 00:51:26 EDT 2016 Linux 2.6.32-642.el6.x86_64 #1 SMP Wed Apr 13 00:51:26 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.8 (Santiago) Components on hosts: ovirt-hosted-engine-ha-1.3.5.9-2.el7ev.noarch libvirt-client-2.0.0-10.el7_3.4.x86_64 mom-0.5.6-1.el7ev.noarch vdsm-4.17.37-1.el7ev.noarch ovirt-hosted-engine-setup-1.3.7.3-1.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-setup-lib-1.0.1-1.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64 ovirt-host-deploy-1.4.1-1.el7ev.noarch rhevm-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016 Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 2)Created guest-VM with bootable disk. 3)Set puma18 to maintenance from WEBUI and updated it to 4.1 repos, then updated it to latest 4.1 components e.g. "yum update ovirt-hosted-engine-setup -y", then "yum update -y" and then rebooted the host. 6)Host got updated to these components: mom-0.5.9-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-vmconsole-host-1.0.4-1.el7ev.noarch vdsm-4.19.6-1.el7ev.x86_64 qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64 ovirt-host-deploy-1.6.0-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-setup-lib-1.1.0-1.el7ev.noarch libvirt-client-2.0.0-10.el7_3.5.x86_64 ovirt-hosted-engine-ha-2.1.0.3-1.el7ev.noarch ovirt-hosted-engine-setup-2.1.0.3-1.el7ev.noarch Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017 Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 4)Activated puma18 via WEBUI and waited until it's score became positive (3400). 5)Selected puma18 as SPM. 6)Migrated HE-VM from puma19 (3.6 host) to puma18 (4.1 host). 7)Installed on puma18 rhevm-appliance-4.0.20170227.0-1.el7ev.noarch.rpm. 8)Created backup on engine "engine-backup --mode=backup --file=nsednev_from_puma19_rhevm_3_6 --log=Log_nsednev_from_puma19_rhevm_3_6", while environment is not in global maintenance, to verify bug 1403903. 9)Copied backup files to puma18 (4.1 host on which HE-VM was running) to "/backup" directory. 10)Set puma18 in global maintenance, so step 11 could be accomplished. 11)Ran hosted-engine --upgrade-appliance on puma18. 12)Successfully upgraded the engine from 3.6 to 4.0: puma18 ~]# hosted-engine --upgrade-appliance [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup During customization use CTRL-D to abort. ================================================================================== Welcome to the oVirt Self Hosted Engine setup/Upgrade tool. Please refer to the oVirt install guide: https://www.ovirt.org/documentation/how-to/hosted-engine/#fresh-install Please refer to the oVirt upgrade guide: https://www.ovirt.org/documentation/how-to/hosted-engine/#upgrade-hosted-engine ================================================================================== Continuing will upgrade the engine VM running on this hosts deploying and configuring a new appliance. If your engine VM is already based on el7 you can also simply upgrade the engine there. This procedure will create a new disk on the hosted-engine storage domain and it will backup there the content of your current engine VM disk. The new el7 based appliance will be deployed over the existing disk destroying its content; at any time you will be able to rollback using the content of the backup disk. You will be asked to take a backup of the running engine and copy it to this host. The engine backup will be automatically injected and recovered on the new appliance. Are you sure you want to continue? (Yes, No)[Yes]: Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170305144438-4cclp3.log Version: otopi-1.6.0 (otopi-1.6.0-1.el7ev) [ INFO ] Detecting available oVirt engine appliances [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Checking maintenance mode [ INFO ] The engine VM is running on this host [ INFO ] Stage: Environment customization [ INFO ] Answer file successfully loaded Enter engine admin password: Confirm engine admin password: [ INFO ] Acquiring internal CA cert from the engine [ INFO ] The following CA certificate is going to be used, please immediately interrupt if not correct: [ INFO ] Issuer: C=US, O=scl.lab.tlv.redhat.com, CN=nsednev-he-4.scl.lab.tlv.redhat.com.89576, Subject: C=US, O=scl.lab.tlv.redhat.com, CN=nsednev-he-4.scl.lab.tlv.redhat.com.89576, Fingerprint (SHA-1): D7E41C9B457311C6635CB27EA6F1B1DF4B7ED0D6 [ INFO ] Bridge ovirtmgmt already created [WARNING] Unable to uniquely detect the interface where Bridge ovirtmgmt has been created on, [u'vnet0', u'enp4s0f0'] appear to be valid alternatives [ INFO ] Checking SPM status on this host [ INFO ] Connecting to the Engine [ INFO ] This upgrade tool is running on the SPM host The following appliance have been found on your system: [1] - The RHEV-M Appliance image (OVA) - 4.0.20170227.0-1.el7ev [2] - Directly select an OVA file Please select an appliance (1, 2) [1]: [ INFO ] Verifying its sha1sum [ INFO ] Checking OVF archive content (could take a few minutes depending on archive size) [ INFO ] Checking OVF XML content (could take a few minutes depending on archive size) Please specify the size of the VM disk in GB: [50]: [ INFO ] Connecting to the Engine [ INFO ] The hosted-engine storage domain has enough free space to contain a new backup disk. [ INFO ] Checking version requirements [ INFO ] Connecting to the Engine [ INFO ] All the datacenters and clusters are at a compatible level [WARNING] Unsupported upgrade path This procedure has been designed and tested only for upgrading the engine VM from ['3.6'] to ['4.0']. Any other usage is highly experimental and potentially dangerous: Current engine: 3.6 Selected appliance: 4.0.20170227.0-1.el7ev Do you want to abort the upgrade process? (Yes, No) [Yes]: no [WARNING] Proceeding on an unsupported and highly experimental path [ INFO ] Detecting host timezone. Enter root password that will be used for the engine appliance (leave it empty to skip): Confirm appliance root password: Enter ssh public key for the root user that will be used for the engine appliance (leave it empty to skip): [WARNING] Skipping appliance root ssh public key Do you want to enable ssh access for the root user (yes, no, without-password) [yes]: Please take a backup of the current engine running this command on the engine VM: engine-backup --mode=backup --archive-compressor=gzip --file=engine_backup.tar.gz --log=engine_backup.log Then copy the backup archive to this host and input here its path when ready. Please specify path to engine backup archive you would like to restore on the new appliance: /backup/nsednev_from_puma19_rhevm_3_6 [ INFO ] Validating backup file '/backup/nsednev_from_puma19_rhevm_3_6' [ INFO ] Validating authentication plugins [ INFO ] Connecting to the Engine [ INFO ] '/backup/nsednev_from_puma19_rhevm_3_6' is a sane backup file [WARNING] Please take care that this will simply add an entry for this host under /etc/hosts on the engine VM. If in the past you added other entries there, recovering them is up to you. [ INFO ] Stage: Setup validation [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Connecting to the Engine [ INFO ] Waiting for the engine to complete disk creation. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] Still waiting for engine VM backup disk to be created. This may take several minutes... [ INFO ] The engine VM backup disk is now ready [ INFO ] Shutting down the current engine VM [ INFO ] Creating a backup of the engine VM disk (could take a few minutes depending on archive size) [ INFO ] Successfully created [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) [ INFO ] Validating pre-allocated volume size [ INFO ] Uploading volume to data domain (could take a few minutes depending on archive size) [ INFO ] Injecting engine backup [ INFO ] Backup successfully injected [ INFO ] Image successfully imported from OVF [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Reloading vm.conf from the shared storage domain [ INFO ] Trying to get a fresher copy of vm configuration from the OVF_STORE [ INFO ] Found an OVF for HE VM, trying to convert [ INFO ] Got vm.conf from OVF_STORE [ INFO ] Running engine-setup on the appliance |- Preparing to restore: |- - Unpacking file '/root/nsednev_from_puma19_rhevm_3_6' |- Restoring: |- - Files |- ------------------------------------------------------------------------------ |- Please note: |- Operating system is different from the one used during backup. |- Current operating system: redhat7 |- Operating system at backup: redhat6 |- Apache httpd configuration will not be restored. |- You will be asked about it on the next engine-setup run. |- ------------------------------------------------------------------------------ |- Provisioning PostgreSQL users/databases: |- - user 'engine', database 'engine' |- Restoring: |- - Engine database 'engine' |- - Cleaning up temporary tables in engine database 'engine' |- - Inserting DbJustRestored VdcOption in engine database |- ------------------------------------------------------------------------------ |- Please note: |- The engine database was backed up at 2017-03-05 14:25:50.000000000 +0200 . |- Objects that were added, removed or changed after this date, such as virtual |- machines, disks, etc., are missing in the engine, and will probably require |- recovery or recreation. |- ------------------------------------------------------------------------------ |- You should now run engine-setup. |- Done. |- HE_APPLIANCE_ENGINE_RESTORE_SUCCESS [ INFO ] Engine backup successfully restored |- [ INFO ] Stage: Initializing |- [ INFO ] Stage: Environment setup |- Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf'] |- Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170305150314-tznjmq.log |- Version: otopi-1.5.2 (otopi-1.5.2-1.el7ev) |- [ INFO ] The engine DB has been restored from a backup |- [ INFO ] Stage: Environment packages setup |- [ INFO ] Stage: Programs detection |- [ INFO ] Stage: Environment setup |- [ INFO ] Stage: Environment customization |- |- Welcome to the RHEV 4.0 setup/upgrade. |- Please read the RHEV 4.0 install guide |- https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/4.0/html/Installation_Guide/index.html. |- Please refer to the RHEV Upgrade Helper application |- https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process. |- Would you like to proceed? (Yes, No) [Yes]: |- |- --== PRODUCT OPTIONS ==-- |- |- Configure Image I/O Proxy on this host? (Yes, No) [Yes]: |- Please note: Data Warehouse is required for the engine. If you choose to not configure it on this host, you have to configure it on a remote host, and then configure the engine on this host so that it can access the database of the remote Data Warehouse host. |- Configure Data Warehouse on this host (Yes, No) [Yes]: |- |- --== PACKAGES ==-- |- |- |- --== NETWORK CONFIGURATION ==-- |- |- [ INFO ] firewalld will be configured as firewall manager. |- |- --== DATABASE CONFIGURATION ==-- |- |- Where is the DWH database located? (Local, Remote) [Local]: |- Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications. |- Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]: |- |- --== OVIRT ENGINE CONFIGURATION ==-- |- |- |- --== STORAGE CONFIGURATION ==-- |- |- |- --== PKI CONFIGURATION ==-- |- |- |- --== APACHE CONFIGURATION ==-- |- |- |- --== SYSTEM CONFIGURATION ==-- |- |- |- --== MISC CONFIGURATION ==-- |- |- Please choose Data Warehouse sampling scale: |- (1) Basic |- (2) Full |- (1, 2)[1]: |- |- --== END OF CONFIGURATION ==-- |- |- [ INFO ] Stage: Setup validation |- [ INFO ] Cleaning stale zombie tasks and commands |- |- --== CONFIGURATION PREVIEW ==-- |- |- Application mode : both |- Default SAN wipe after delete : False |- Firewall manager : firewalld |- Update Firewall : True |- Host FQDN : nsednev-he-4.scl.lab.tlv.redhat.com |- Engine database secured connection : False |- Engine database user name : engine |- Engine database name : engine |- Engine database host : localhost |- Engine database port : 5432 |- Engine database host name validation : False |- DWH database secured connection : False |- DWH database host : localhost |- DWH database user name : ovirt_engine_history |- DWH database name : ovirt_engine_history |- DWH database port : 5432 |- DWH database host name validation : False |- Engine installation : True |- PKI organization : scl.lab.tlv.redhat.com |- Configure local Engine database : True |- Set application as default page : True |- Configure Apache SSL : True |- DWH installation : True |- Configure local DWH database : True |- Engine Host FQDN : nsednev-he-4.scl.lab.tlv.redhat.com |- Configure Image I/O Proxy : True |- Configure VMConsole Proxy : True |- Configure WebSocket Proxy : True |- [ INFO ] Cleaning async tasks and compensations |- [ INFO ] Unlocking existing entities |- [ INFO ] Checking the Engine database consistency |- [ INFO ] Stage: Transaction setup |- [ INFO ] Stopping engine service |- [ INFO ] Stopping ovirt-fence-kdump-listener service |- [ INFO ] Stopping dwh service |- [ INFO ] Stopping Image I/O Proxy service |- [ INFO ] Stopping websocket-proxy service |- [ INFO ] Stage: Misc configuration |- [ INFO ] Stage: Package installation |- [ INFO ] Stage: Misc configuration |- [ INFO ] Upgrading CA |- [ INFO ] Creating PostgreSQL 'ovirt_engine_history' database |- [ INFO ] Configuring PostgreSQL |- [ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20170305150324.CXePP1.dump'. |- [ INFO ] Creating/refreshing Engine database schema |- [ INFO ] Creating/refreshing DWH database schema |- [ INFO ] Configuring Image I/O Proxy |- [ INFO ] Configuring WebSocket Proxy |- [ INFO ] Creating/refreshing Engine 'internal' domain database schema |- [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' |- [ INFO ] Stage: Transaction commit |- [ INFO ] Stage: Closing up |- [ INFO ] Starting engine service |- [ INFO ] Starting dwh service |- [ INFO ] Restarting ovirt-vmconsole proxy service |- |- --== SUMMARY ==-- |- |- [ INFO ] Restarting httpd |- Web access is enabled at: |- http://nsednev-he-4.scl.lab.tlv.redhat.com:80/ovirt-engine |- https://nsednev-he-4.scl.lab.tlv.redhat.com:443/ovirt-engine |- Internal CA D7:E4:1C:9B:45:73:11:C6:63:5C:B2:7E:A6:F1:B1:DF:4B:7E:D0:D6 |- SSH fingerprint: 3f:ae:99:ac:54:9c:f2:4f:23:ec:bc:be:aa:5a:bf:77 |- |- --== END OF SUMMARY ==-- |- |- [ INFO ] Stage: Clean up |- Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170305150314-tznjmq.log |- [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170305150417-setup.conf' |- [ INFO ] Stage: Pre-termination |- [ INFO ] Stage: Termination |- [ INFO ] Execution of setup completed successfully |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS [ INFO ] Engine-setup successfully completed [ INFO ] Engine is still unreachable [ INFO ] Engine is still not reachable, waiting... [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ INFO ] Connecting to the Engine [ INFO ] Connecting to the Engine [ INFO ] Registering the hosted-engine backup disk in the DB [ INFO ] Waiting for the engine to complete disk registration. This may take several minutes... [ INFO ] The engine VM backup disk is now ready [ INFO ] Shutting down the engine VM [ INFO ] Stage: Clean up [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Hosted Engine successfully upgraded [ INFO ] Please exit global maintenance mode to restart the engine VM. [ INFO ] Please note that the engine VM ssh keys have changed. Please remove the engine VM entry in ssh known_hosts on your clients. You have new mail in /var/spool/mail/root 13)Removed environment from global maintenance e.g. "hosted-engine --set-maintenance --mode=none". Components on upgraded 4.0 engine: rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch rhev-guest-tools-iso-4.0-7.el7ev.noarch rhevm-setup-plugins-4.0.0.3-1.el7ev.noarch rhevm-doc-4.0.7-1.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhevm-guest-agent-common-1.0.12-4.el7ev.noarch rhevm-branding-rhev-4.0.0-7.el7ev.noarch rhevm-4.0.7.3-0.1.el7ev.noarch Linux version 3.10.0-514.6.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Feb 17 19:21:31 EST 2017 Linux 3.10.0-514.6.2.el7.x86_64 #1 SMP Fri Feb 17 19:21:31 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 14)Set puma19 to maintenance from WEBUI and updated it to 4.1 repos, then updated it to latest 4.1 components e.g. "yum update ovirt-hosted-engine-setup -y", then "yum update -y" and then rebooted the host. 15)Host got updated to these components: mom-0.5.9-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-vmconsole-host-1.0.4-1.el7ev.noarch vdsm-4.19.6-1.el7ev.x86_64 qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64 ovirt-host-deploy-1.6.0-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch ovirt-imageio-common-1.0.0-0.el7ev.noarch ovirt-imageio-daemon-1.0.0-0.el7ev.noarch ovirt-setup-lib-1.1.0-1.el7ev.noarch libvirt-client-2.0.0-10.el7_3.5.x86_64 ovirt-hosted-engine-ha-2.1.0.3-1.el7ev.noarch ovirt-hosted-engine-setup-2.1.0.3-1.el7ev.noarch Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017 Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) 16)Activated puma19 via WEBUI and waited until it's score became positive (3400). 17)Bumped up host cluster compatibility mode from 3.6 to 4.0. 18)Bumped up data center compatibility mode from 3.6 to 4.0. 19)Upgraded engine only from 4.0 to latest 4.1, using global maintenance on hosts. Moving to verified.