Bug 1394740

Summary: Hosted Engine upgrade is failing during engine-setup if the db backup file is not in /root/
Product: Red Hat Enterprise Virtualization Manager Reporter: nijin ashok <nashok>
Component: ovirt-hosted-engine-setupAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED ERRATA QA Contact: Nikolai Sednev <nsednev>
Severity: high Docs Contact:
Priority: medium    
Version: 4.0.4CC: lsurette, mavital, melewis, mkalinin, mwest, sbonazzo, ykaul, ylavi
Target Milestone: ovirt-4.1.0-alphaKeywords: Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: hosted-engine | infra, hosted-engine
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, a self-hosted engine upgrade failed during engine-setup if the database backup file was not loaded from /root/ on the host. This has now been corrected.
Story Points: ---
Clone Of:
: 1399053 (view as bug list) Environment:
Last Closed: 2017-04-25 00:57:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1396193, 1403854    
Bug Blocks: 1399053    

Description nijin ashok 2016-11-14 11:03:25 UTC
Description of problem:

While upgrading the HE setup from 3.6 to 4.0, the engine-setup within the VM will fail if the db backup file is copied in any place other than /root/.

The script always put the db backup under /root/ in the appliance image.

        g.upload(
            self._backup_src,
            os.path.join('/root/', os.path.basename(self._backup_src))
        )

However while executing the engine-backup command, we are providing backup location as OVEHOSTED_UPGRADE/backupFileName which can change accordingly where the user has put the backup file in the hypervisor.

                engine_restore = (
                    ' - engine-backup --mode=restore --file={backup_file}'
                    ' --log=engine_restore.log --restore-permissions'
                    ' --provision-db {p_dwh_db} {p_reports_db}'
                    ' 1>{port}'
                    ' 2>&1\n'
                    ' - if [ $? -eq 0 ];'
                    ' then echo "{success_string}" >{port};'
                    ' else echo "{fail_string}" >{port};'
                    ' fi\n'
                ).format(
                    backup_file=self.environment[
                        ohostedcons.Upgrade.BACKUP_FILE
                    ],

So the upgrade will fails with below errors.

===
2016-11-14 21:46:18 DEBUG otopi.context context.dumpEnvironment:770 ENV OVEHOSTED_UPGRADE/backupFileName=str:'/tmp/engine_backup.tar.gz'

2016-11-14 19:53:41 DEBUG otopi.plugins.gr_he_common.vm.cloud_init cloud_init._misc:869 engine_restore:  - engine-backup --mode=restore --file=/tmp/engine_backup.tar.gz --log=engine_restore.log --restore-permissions --provision-db   1>/dev/virtio-ports/org.ovirt.hosted-engine-setup.0 2>&1
 - if [ $? -eq 0 ]; then echo "HE_APPLIANCE_ENGINE_RESTORE_SUCCESS" >/dev/virtio-ports/org.ovirt.hosted-engine-setup.0; else echo "HE_APPLIANCE_ENGINE_RESTORE_FAIL" >/dev/virtio-ports/org.ovirt.hosted-engine-setup.0; fi


 2016-11-14 21:46:18 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 |- FATAL: /tmp/engine_backup.tar.gz does not exist
2016-11-14 21:46:18 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 |- HE_APPLIANCE_ENGINE_RESTORE_FAIL
2016-11-14 21:46:18 ERROR otopi.plugins.gr_he_common.engine.health health._closeup:157 Engine backup restore failed on the appliance
===

Version-Release number of selected component (if applicable):

rhevm-4.0.4.4-0.1.el7ev.noarch
ovirt-hosted-engine-setup-2.0.2.2-2.el7ev.noarch

How reproducible:

100%

Steps to Reproduce:

1. Put the DB backup in any location other than /root/

2.  Run hosted-engine --upgrade-appliance and this fail with error

Actual results:

hosted-engine --upgrade-appliance is not working

Expected results:

hosted-engine --upgrade-appliance should be working irrespective of the backup file location.

Additional info:

Comment 2 Marina Kalinin 2016-11-19 03:48:20 UTC
Isn't this a duplicate of this bz#1392741 ?

Comment 3 Yaniv Kaul 2016-11-21 07:19:33 UTC
*** Bug 1392741 has been marked as a duplicate of this bug. ***

Comment 4 Sandro Bonazzola 2016-11-21 13:47:01 UTC
Yaniv already marked bz#1392741 as duplicate.

Comment 8 Nikolai Sednev 2017-02-08 19:52:25 UTC
Moving to verified forth to:
Please take a backup of the current engine running this command on the engine VM:
           engine-backup --mode=backup --archive-compressor=gzip --file=engine_backup.tar.gz --log=engine_backup.log
          Then copy the backup archive to this host and input here its path when ready.
          Please specify path to engine backup archive you would like to restore on the new appliance: /backup/nsednev_from_alma04_rhevm_3_6
[ INFO  ] Validating backup file '/backup/nsednev_from_alma04_rhevm_3_6'
[ INFO  ] Validating authentication plugins
[ INFO  ] Connecting to the Engine
[ INFO  ] '/backup/nsednev_from_alma04_rhevm_3_6' is a sane backup file

Components on host:
vdsm-4.19.4-1.el7ev.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
mom-0.5.8-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.10.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Jan 30 11:07:00 EST 2017
Linux 3.10.0-514.10.1.el7.x86_64 #1 SMP Mon Jan 30 11:07:00 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)