Bug 1956413 - HE restore fails.
Summary: HE restore fails.
Keywords:
Status: NEW
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: General
Version: 2.5.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.4.7
: ---
Assignee: Yedidyah Bar David
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-03 16:09 UTC by Nikolai Sednev
Modified: 2021-05-11 07:33 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
oVirt Team: Integration
lsvaty: blocker-


Attachments (Terms of Use)
sosreport from host alma03 (15.47 MB, application/x-xz)
2021-05-03 16:09 UTC, Nikolai Sednev
no flags Details
answers-20210503185122.conf (2.56 KB, text/plain)
2021-05-03 16:38 UTC, Nikolai Sednev
no flags Details
ovirt-hosted-engine-setup-20210503182906-b1kwig.log (490.01 KB, text/plain)
2021-05-03 16:39 UTC, Nikolai Sednev
no flags Details

Description Nikolai Sednev 2021-05-03 16:09:24 UTC
Created attachment 1779048 [details]
sosreport from host alma03

Description of problem:
HE restore fails with this error:
[ ERROR ] fatal: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]: FAILED! => {"changed": true, "cmd": ["engine-setup", "--accept-defaults", "--config-append=/root/ovirt-engine-answers"], "delta": "0:01:58.524768", "end": "2021-05-03 18:49:11.903297", "msg": "non-zero return code", "rc": 1, "start": "2021-05-03 18:47:13.378529", "stderr": "", "stderr_lines": [], "stdout": "[ INFO  ] Stage: Initializing\n[ INFO  ] Stage: Environment setup\n          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /root/ovirt-engine-answers\n          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log\n          Version: otopi-1.9.4 (otopi-1.9.4-1.el8ev)\n[ INFO  ] The engine DB has been restored from a backup\n[ INFO  ] Stage: Environment packages setup\n[ INFO  ] Stage: Programs detection\n[ INFO  ] Stage: Environment setup (late)\n[ INFO  ] Stage: Environment customization\n         \n          Welcome to the RHV 4.4 setup/upgrade.\n          Please read the RHV 4.4 upgrade guide\n          https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/upgrade_guide/index\n          Please refer to the RHV Upgrade Helper application\n          https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.\n          Would you like to proceed? (Yes, No) [Yes]: \n         \n          --== PRODUCT OPTIONS ==--\n         \n[ INFO  ] ovirt-provider-ovn already installed, skipping.\n         \n          --== PACKAGES ==--\n         \n[ INFO  ] Checking for product updates...\n[ INFO  ] No product updates found\n         \n          --== NETWORK CONFIGURATION ==--\n         \n[ INFO  ] firewalld will be configured as firewall manager.\n         \n          --== DATABASE CONFIGURATION ==--\n         \n         \n          Found the following problems in PostgreSQL configuration for the Engine database:\n           It is required to be at most '0.01'\n           It is required to be at most '0.075'\n           It is required to be at least '6'\n           It is required to be at least '8192'\n         \n          Please note the following required changes in postgresql.conf on 'localhost':\n            'autovacuum_vacuum_scale_factor' is currently '0.2'. It is required to be at most '0.01'.\n            'autovacuum_analyze_scale_factor' is currently '0.1'. It is required to be at most '0.075'.\n            'autovacuum_max_workers' is currently '3'. It is required to be at least '6'.\n            'work_mem' is currently '4096'. It is required to be at least '8192'.\n          postgresql.conf is usually in /var/lib/pgsql/data,  or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes.\n          The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: \n          The detected DWH database size is 29.89146327972412 MB.\n          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.\n          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.\n          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: \n          Perform full vacuum on the oVirt engine history\n          database ovirt_engine_history@localhost?\n          This operation may take a while depending on this setup health and the\n          configuration of the db vacuum process.\n          See https://www.postgresql.org/docs/12/sql-vacuum.html\n          (Yes, No) [No]: \n         \n          --== OVIRT ENGINE CONFIGURATION ==--\n         \n          Perform full vacuum on the engine database engine@localhost?\n          This operation may take a while depending on this setup health and the\n          configuration of the db vacuum process.\n          See https://www.postgresql.org/docs/12/sql-vacuum.html\n          (Yes, No) [No]: \n         \n          --== STORAGE CONFIGURATION ==--\n         \n         \n          --== PKI CONFIGURATION ==--\n         \n         \n          --== APACHE CONFIGURATION ==--\n         \n         \n          --== SYSTEM CONFIGURATION ==--\n         \n         \n          --== MISC CONFIGURATION ==--\n         \n         \n          --== END OF CONFIGURATION ==--\n         \n[ INFO  ] Stage: Setup validation\n[ INFO  ] Cleaning stale zombie tasks and commands\n         \n          --== CONFIGURATION PREVIEW ==--\n         \n          Application mode                        : both\n          Default SAN wipe after delete           : False\n          Host FQDN                               : nsednev-he-1.qa.lab.tlv.redhat.com\n          Firewall manager                        : firewalld\n          Update Firewall                         : True\n          Set up Cinderlib integration            : False\n          Configure local Engine database         : True\n          Set application as default page         : True\n          Configure Apache SSL                    : True\n          Engine database host                    : localhost\n          Engine database port                    : 5432\n          Engine database secured connection      : False\n          Engine database host name validation    : False\n          Engine database name                    : engine\n          Engine database user name               : engine\n          Engine installation                     : True\n          PKI organization                        : qa.lab.tlv.redhat.com\n          Renew PKI                               : False\n          Set up ovirt-provider-ovn               : True\n          Grafana integration                     : False\n          Configure WebSocket Proxy               : True\n          DWH installation                        : True\n          DWH database host                       : localhost\n          DWH database port                       : 5432\n          DWH database secured connection         : False\n          DWH database host name validation       : False\n          DWH database name                       : ovirt_engine_history\n          DWH database user name                  : ovirt_engine_history\n          Backup DWH database                     : True\n          Configure VMConsole Proxy               : True\n[ INFO  ] Cleaning async tasks and compensations\n[ INFO  ] Unlocking existing entities\n[ INFO  ] Checking the Engine database consistency\n[ INFO  ] Stage: Transaction setup\n[ INFO  ] Stopping engine service\n[ INFO  ] Stopping ovirt-fence-kdump-listener service\n[ INFO  ] Stopping dwh service\n[ INFO  ] Stopping vmconsole-proxy service\n[ INFO  ] Stopping websocket-proxy service\n[ INFO  ] Stage: Misc configuration (early)\n[ INFO  ] Updating PostgreSQL configuration\n[ INFO  ] Stage: Package installation\n[ INFO  ] Stage: Misc configuration\n[ INFO  ] Upgrading CA\n[ INFO  ] Updating OVN SSL configuration\n[ INFO  ] Updating OVN timeout configuration\n[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump'.\n[ INFO  ] Creating/refreshing DWH database schema\n[ INFO  ] Configuring WebSocket Proxy\n[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump'.\n[ INFO  ] Creating/refreshing Engine database schema\n[ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/create_views.sql\n[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed\n[ INFO  ] DNF Performing DNF transaction rollback\n[WARNING] Rollback of DWH database postponed to Stage \"Clean up\"\n[ INFO  ] Rolling back database schema\n[ INFO  ] Clearing Engine database engine\n[ INFO  ] Restoring Engine database engine\n[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump' to database localhost:engine.\n[ INFO  ] Stage: Clean up\n          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log\n[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210503184848-setup.conf'\n[WARNING] Rollback of DWH database started\n          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.\n[ INFO  ] Clearing DWH database ovirt_engine_history\n[ INFO  ] Restoring DWH database ovirt_engine_history\n[ INFO  ] Restoring file '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump' to database localhost:ovirt_engine_history.\n[ INFO  ] Stage: Pre-termination\n[ INFO  ] Stage: Termination\n[ ERROR ] Execution of setup failed", "stdout_lines": ["[ INFO  ] Stage: Initializing", "[ INFO  ] Stage: Environment setup", "          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /root/ovirt-engine-answers", "          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log", "          Version: otopi-1.9.4 (otopi-1.9.4-1.el8ev)", "[ INFO  ] The engine DB has been restored from a backup", "[ INFO  ] Stage: Environment packages setup", "[ INFO  ] Stage: Programs detection", "[ INFO  ] Stage: Environment setup (late)", "[ INFO  ] Stage: Environment customization", "         ", "          Welcome to the RHV 4.4 setup/upgrade.", "          Please read the RHV 4.4 upgrade guide", "          https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/upgrade_guide/index", "          Please refer to the RHV Upgrade Helper application", "          https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.", "          Would you like to proceed? (Yes, No) [Yes]: ", "         ", "          --== PRODUCT OPTIONS ==--", "         ", "[ INFO  ] ovirt-provider-ovn already installed, skipping.", "         ", "          --== PACKAGES ==--", "         ", "[ INFO  ] Checking for product updates...", "[ INFO  ] No product updates found", "         ", "          --== NETWORK CONFIGURATION ==--", "         ", "[ INFO  ] firewalld will be configured as firewall manager.", "         ", "          --== DATABASE CONFIGURATION ==--", "         ", "         ", "          Found the following problems in PostgreSQL configuration for the Engine database:", "           It is required to be at most '0.01'", "           It is required to be at most '0.075'", "           It is required to be at least '6'", "           It is required to be at least '8192'", "         ", "          Please note the following required changes in postgresql.conf on 'localhost':", "            'autovacuum_vacuum_scale_factor' is currently '0.2'. It is required to be at most '0.01'.", "            'autovacuum_analyze_scale_factor' is currently '0.1'. It is required to be at most '0.075'.", "            'autovacuum_max_workers' is currently '3'. It is required to be at least '6'.", "            'work_mem' is currently '4096'. It is required to be at least '8192'.", "          postgresql.conf is usually in /var/lib/pgsql/data,  or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes.", "          The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: ", "          The detected DWH database size is 29.89146327972412 MB.", "          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.", "          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.", "          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: ", "          Perform full vacuum on the oVirt engine history", "          database ovirt_engine_history@localhost?", "          This operation may take a while depending on this setup health and the", "          configuration of the db vacuum process.", "          See https://www.postgresql.org/docs/12/sql-vacuum.html", "          (Yes, No) [No]: ", "         ", "          --== OVIRT ENGINE CONFIGURATION ==--", "         ", "          Perform full vacuum on the engine database engine@localhost?", "          This operation may take a while depending on this setup health and the", "          configuration of the db vacuum process.", "          See https://www.postgresql.org/docs/12/sql-vacuum.html", "          (Yes, No) [No]: ", "         ", "          --== STORAGE CONFIGURATION ==--", "         ", "         ", "          --== PKI CONFIGURATION ==--", "         ", "         ", "          --== APACHE CONFIGURATION ==--", "         ", "         ", "          --== SYSTEM CONFIGURATION ==--", "         ", "         ", "          --== MISC CONFIGURATION ==--", "         ", "         ", "          --== END OF CONFIGURATION ==--", "         ", "[ INFO  ] Stage: Setup validation", "[ INFO  ] Cleaning stale zombie tasks and commands", "         ", "          --== CONFIGURATION PREVIEW ==--", "         ", "          Application mode                        : both", "          Default SAN wipe after delete           : False", "          Host FQDN                               : nsednev-he-1.qa.lab.tlv.redhat.com", "          Firewall manager                        : firewalld", "          Update Firewall                         : True", "          Set up Cinderlib integration            : False", "          Configure local Engine database         : True", "          Set application as default page         : True", "          Configure Apache SSL                    : True", "          Engine database host                    : localhost", "          Engine database port                    : 5432", "          Engine database secured connection      : False", "          Engine database host name validation    : False", "          Engine database name                    : engine", "          Engine database user name               : engine", "          Engine installation                     : True", "          PKI organization                        : qa.lab.tlv.redhat.com", "          Renew PKI                               : False", "          Set up ovirt-provider-ovn               : True", "          Grafana integration                     : False", "          Configure WebSocket Proxy               : True", "          DWH installation                        : True", "          DWH database host                       : localhost", "          DWH database port                       : 5432", "          DWH database secured connection         : False", "          DWH database host name validation       : False", "          DWH database name                       : ovirt_engine_history", "          DWH database user name                  : ovirt_engine_history", "          Backup DWH database                     : True", "          Configure VMConsole Proxy               : True", "[ INFO  ] Cleaning async tasks and compensations", "[ INFO  ] Unlocking existing entities", "[ INFO  ] Checking the Engine database consistency", "[ INFO  ] Stage: Transaction setup", "[ INFO  ] Stopping engine service", "[ INFO  ] Stopping ovirt-fence-kdump-listener service", "[ INFO  ] Stopping dwh service", "[ INFO  ] Stopping vmconsole-proxy service", "[ INFO  ] Stopping websocket-proxy service", "[ INFO  ] Stage: Misc configuration (early)", "[ INFO  ] Updating PostgreSQL configuration", "[ INFO  ] Stage: Package installation", "[ INFO  ] Stage: Misc configuration", "[ INFO  ] Upgrading CA", "[ INFO  ] Updating OVN SSL configuration", "[ INFO  ] Updating OVN timeout configuration", "[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump'.", "[ INFO  ] Creating/refreshing DWH database schema", "[ INFO  ] Configuring WebSocket Proxy", "[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump'.", "[ INFO  ] Creating/refreshing Engine database schema", "[ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/create_views.sql", "[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed", "[ INFO  ] DNF Performing DNF transaction rollback", "[WARNING] Rollback of DWH database postponed to Stage \"Clean up\"", "[ INFO  ] Rolling back database schema", "[ INFO  ] Clearing Engine database engine", "[ INFO  ] Restoring Engine database engine", "[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump' to database localhost:engine.", "[ INFO  ] Stage: Clean up", "          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log", "[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210503184848-setup.conf'", "[WARNING] Rollback of DWH database started", "          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.", "[ INFO  ] Clearing DWH database ovirt_engine_history", "[ INFO  ] Restoring DWH database ovirt_engine_history", "[ INFO  ] Restoring file '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump' to database localhost:ovirt_engine_history.", "[ INFO  ] Stage: Pre-termination", "[ INFO  ] Stage: Termination", "[ ERROR ] Execution of setup failed"]}
[ INFO  ] TASK [redhat.rhv.engine_setup : Clean temporary files]
[ INFO  ] changed: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ INFO  ] changed: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Create destination directory]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch the value of HOST_KEY_CHECKING]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get the username running the deploy]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Register the engine FQDN as a host]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ INFO  ] changed: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch logs from the engine VM]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Create destination directory]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Clean local storage pools]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Destroy local storage-pool localvm36a8agza]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Undefine local storage-pool localvm36a8agza]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Destroy local storage-pool 6d8b651b-6666-4aa1-8dec-b321e2e9adac]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Undefine local storage-pool 6d8b651b-6666-4aa1-8dec-b321e2e9adac]
[ INFO  ] changed: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20210503185122.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20210503182906-b1kwig.log


Version-Release number of selected component (if applicable):
rhvm-appliance-4.4-20210402.1.el8ev.x86_64
ovirt-hosted-engine-ha-2.4.6-1.el8ev.noarch
ovirt-hosted-engine-setup-2.5.0-2.el8ev.noarch
Red Hat Enterprise Linux release 8.4 (Ootpa)
Linux 4.18.0-304.el8.x86_64 #1 SMP Tue Apr 6 05:19:59 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

How reproducible:
100%

Steps to Reproduce:
1.Deploy HE over NFS with latest bits and upgrade the engine's bits to latest.
2.Bump up the host cluster and the data center to 4.6 from 4.5.
3.Make the backup and try to restore on the same cleanly reprovisioned host.

Actual results:
Restore fails.

Expected results:
Restore should succeed.

Additional info:
Logs from host alma03 attached.

Comment 1 Nikolai Sednev 2021-05-03 16:38:10 UTC
Created attachment 1779051 [details]
answers-20210503185122.conf

Comment 2 Nikolai Sednev 2021-05-03 16:39:23 UTC
Created attachment 1779052 [details]
ovirt-hosted-engine-setup-20210503182906-b1kwig.log

Comment 3 Martin Perina 2021-05-04 04:44:13 UTC
There is following error in engine setup log:

psql:/usr/share/ovirt-engine/dbscripts/create_views.sql:1000: ERROR:  column vm_templates.effective_bios_type does not exist
LINE 76:     vm_templates.effective_bios_type AS effective_bios_type,
             ^
FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/create_views.sql

2021-05-03 18:48:07,214+0300 ERROR otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema schema._misc:530 schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/create_views.sql
2021-05-03 18:48:07,215+0300 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 532, in _misc
    raise RuntimeError(_('Engine schema refresh failed'))
RuntimeError: Engine schema refresh failed


Looking at the db scripts below patches are handling this table column:


https://gerrit.ovirt.org/c/ovirt-engine/+/111656
  - this is included in the 1st 4.4.6 build

https://gerrit.ovirt.org/c/ovirt-engine/+/113462
  - this is included in ovirt-engine-4.4.6.4

Are you sure that you are not that ovirt-engine version from the backup is the same as the version you are restoring to?

Comment 4 Yedidyah Bar David 2021-05-04 05:54:53 UTC
(In reply to Martin Perina from comment #3)
> Are you sure that you are not that ovirt-engine version from the backup is
> the same as the version you are restoring to?

See also bug 1932392. If we want to handle also the hosted-engine restore case - meaning, fail there early, and not just as a result of fixing bug 1932392 - perhaps open another bug (or we can use current, if that's the case, and then please change the subject).

Comment 5 Arik 2021-05-06 07:55:07 UTC
I think Martin's observation is right and then this particular case is not that interesting as the changes to the bios type field in the database were done before the GA of ovirt 4.4.6


Note You need to log in before you can comment on or make changes to this bug.