Bug 1956413

Created attachment 1779048 [details]
sosreport from host alma03

Description of problem:
HE restore fails with this error:
[ ERROR ] fatal: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]: FAILED! => {"changed": true, "cmd": ["engine-setup", "--accept-defaults", "--config-append=/root/ovirt-engine-answers"], "delta": "0:01:58.524768", "end": "2021-05-03 18:49:11.903297", "msg": "non-zero return code", "rc": 1, "start": "2021-05-03 18:47:13.378529", "stderr": "", "stderr_lines": [], "stdout": "[ INFO  ] Stage: Initializing\n[ INFO  ] Stage: Environment setup\n          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /root/ovirt-engine-answers\n          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log\n          Version: otopi-1.9.4 (otopi-1.9.4-1.el8ev)\n[ INFO  ] The engine DB has been restored from a backup\n[ INFO  ] Stage: Environment packages setup\n[ INFO  ] Stage: Programs detection\n[ INFO  ] Stage: Environment setup (late)\n[ INFO  ] Stage: Environment customization\n         \n          Welcome to the RHV 4.4 setup/upgrade.\n          Please read the RHV 4.4 upgrade guide\n          https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/upgrade_guide/index\n          Please refer to the RHV Upgrade Helper application\n          https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.\n          Would you like to proceed? (Yes, No) [Yes]: \n         \n          --== PRODUCT OPTIONS ==--\n         \n[ INFO  ] ovirt-provider-ovn already installed, skipping.\n         \n          --== PACKAGES ==--\n         \n[ INFO  ] Checking for product updates...\n[ INFO  ] No product updates found\n         \n          --== NETWORK CONFIGURATION ==--\n         \n[ INFO  ] firewalld will be configured as firewall manager.\n         \n          --== DATABASE CONFIGURATION ==--\n         \n         \n          Found the following problems in PostgreSQL configuration for the Engine database:\n           It is required to be at most '0.01'\n           It is required to be at most '0.075'\n           It is required to be at least '6'\n           It is required to be at least '8192'\n         \n          Please note the following required changes in postgresql.conf on 'localhost':\n            'autovacuum_vacuum_scale_factor' is currently '0.2'. It is required to be at most '0.01'.\n            'autovacuum_analyze_scale_factor' is currently '0.1'. It is required to be at most '0.075'.\n            'autovacuum_max_workers' is currently '3'. It is required to be at least '6'.\n            'work_mem' is currently '4096'. It is required to be at least '8192'.\n          postgresql.conf is usually in /var/lib/pgsql/data,  or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes.\n          The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: \n          The detected DWH database size is 29.89146327972412 MB.\n          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.\n          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.\n          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: \n          Perform full vacuum on the oVirt engine history\n          database ovirt_engine_history@localhost?\n          This operation may take a while depending on this setup health and the\n          configuration of the db vacuum process.\n          See https://www.postgresql.org/docs/12/sql-vacuum.html\n          (Yes, No) [No]: \n         \n          --== OVIRT ENGINE CONFIGURATION ==--\n         \n          Perform full vacuum on the engine database engine@localhost?\n          This operation may take a while depending on this setup health and the\n          configuration of the db vacuum process.\n          See https://www.postgresql.org/docs/12/sql-vacuum.html\n          (Yes, No) [No]: \n         \n          --== STORAGE CONFIGURATION ==--\n         \n         \n          --== PKI CONFIGURATION ==--\n         \n         \n          --== APACHE CONFIGURATION ==--\n         \n         \n          --== SYSTEM CONFIGURATION ==--\n         \n         \n          --== MISC CONFIGURATION ==--\n         \n         \n          --== END OF CONFIGURATION ==--\n         \n[ INFO  ] Stage: Setup validation\n[ INFO  ] Cleaning stale zombie tasks and commands\n         \n          --== CONFIGURATION PREVIEW ==--\n         \n          Application mode                        : both\n          Default SAN wipe after delete           : False\n          Host FQDN                               : nsednev-he-1.qa.lab.tlv.redhat.com\n          Firewall manager                        : firewalld\n          Update Firewall                         : True\n          Set up Cinderlib integration            : False\n          Configure local Engine database         : True\n          Set application as default page         : True\n          Configure Apache SSL                    : True\n          Engine database host                    : localhost\n          Engine database port                    : 5432\n          Engine database secured connection      : False\n          Engine database host name validation    : False\n          Engine database name                    : engine\n          Engine database user name               : engine\n          Engine installation                     : True\n          PKI organization                        : qa.lab.tlv.redhat.com\n          Renew PKI                               : False\n          Set up ovirt-provider-ovn               : True\n          Grafana integration                     : False\n          Configure WebSocket Proxy               : True\n          DWH installation                        : True\n          DWH database host                       : localhost\n          DWH database port                       : 5432\n          DWH database secured connection         : False\n          DWH database host name validation       : False\n          DWH database name                       : ovirt_engine_history\n          DWH database user name                  : ovirt_engine_history\n          Backup DWH database                     : True\n          Configure VMConsole Proxy               : True\n[ INFO  ] Cleaning async tasks and compensations\n[ INFO  ] Unlocking existing entities\n[ INFO  ] Checking the Engine database consistency\n[ INFO  ] Stage: Transaction setup\n[ INFO  ] Stopping engine service\n[ INFO  ] Stopping ovirt-fence-kdump-listener service\n[ INFO  ] Stopping dwh service\n[ INFO  ] Stopping vmconsole-proxy service\n[ INFO  ] Stopping websocket-proxy service\n[ INFO  ] Stage: Misc configuration (early)\n[ INFO  ] Updating PostgreSQL configuration\n[ INFO  ] Stage: Package installation\n[ INFO  ] Stage: Misc configuration\n[ INFO  ] Upgrading CA\n[ INFO  ] Updating OVN SSL configuration\n[ INFO  ] Updating OVN timeout configuration\n[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump'.\n[ INFO  ] Creating/refreshing DWH database schema\n[ INFO  ] Configuring WebSocket Proxy\n[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump'.\n[ INFO  ] Creating/refreshing Engine database schema\n[ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/create_views.sql\n[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed\n[ INFO  ] DNF Performing DNF transaction rollback\n[WARNING] Rollback of DWH database postponed to Stage \"Clean up\"\n[ INFO  ] Rolling back database schema\n[ INFO  ] Clearing Engine database engine\n[ INFO  ] Restoring Engine database engine\n[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump' to database localhost:engine.\n[ INFO  ] Stage: Clean up\n          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log\n[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210503184848-setup.conf'\n[WARNING] Rollback of DWH database started\n          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.\n[ INFO  ] Clearing DWH database ovirt_engine_history\n[ INFO  ] Restoring DWH database ovirt_engine_history\n[ INFO  ] Restoring file '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump' to database localhost:ovirt_engine_history.\n[ INFO  ] Stage: Pre-termination\n[ INFO  ] Stage: Termination\n[ ERROR ] Execution of setup failed", "stdout_lines": ["[ INFO  ] Stage: Initializing", "[ INFO  ] Stage: Environment setup", "          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /root/ovirt-engine-answers", "          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log", "          Version: otopi-1.9.4 (otopi-1.9.4-1.el8ev)", "[ INFO  ] The engine DB has been restored from a backup", "[ INFO  ] Stage: Environment packages setup", "[ INFO  ] Stage: Programs detection", "[ INFO  ] Stage: Environment setup (late)", "[ INFO  ] Stage: Environment customization", "         ", "          Welcome to the RHV 4.4 setup/upgrade.", "          Please read the RHV 4.4 upgrade guide", "          https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/upgrade_guide/index", "          Please refer to the RHV Upgrade Helper application", "          https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.", "          Would you like to proceed? (Yes, No) [Yes]: ", "         ", "          --== PRODUCT OPTIONS ==--", "         ", "[ INFO  ] ovirt-provider-ovn already installed, skipping.", "         ", "          --== PACKAGES ==--", "         ", "[ INFO  ] Checking for product updates...", "[ INFO  ] No product updates found", "         ", "          --== NETWORK CONFIGURATION ==--", "         ", "[ INFO  ] firewalld will be configured as firewall manager.", "         ", "          --== DATABASE CONFIGURATION ==--", "         ", "         ", "          Found the following problems in PostgreSQL configuration for the Engine database:", "           It is required to be at most '0.01'", "           It is required to be at most '0.075'", "           It is required to be at least '6'", "           It is required to be at least '8192'", "         ", "          Please note the following required changes in postgresql.conf on 'localhost':", "            'autovacuum_vacuum_scale_factor' is currently '0.2'. It is required to be at most '0.01'.", "            'autovacuum_analyze_scale_factor' is currently '0.1'. It is required to be at most '0.075'.", "            'autovacuum_max_workers' is currently '3'. It is required to be at least '6'.", "            'work_mem' is currently '4096'. It is required to be at least '8192'.", "          postgresql.conf is usually in /var/lib/pgsql/data,  or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes.", "          The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: ", "          The detected DWH database size is 29.89146327972412 MB.", "          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.", "          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.", "          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: ", "          Perform full vacuum on the oVirt engine history", "          database ovirt_engine_history@localhost?", "          This operation may take a while depending on this setup health and the", "          configuration of the db vacuum process.", "          See https://www.postgresql.org/docs/12/sql-vacuum.html", "          (Yes, No) [No]: ", "         ", "          --== OVIRT ENGINE CONFIGURATION ==--", "         ", "          Perform full vacuum on the engine database engine@localhost?", "          This operation may take a while depending on this setup health and the", "          configuration of the db vacuum process.", "          See https://www.postgresql.org/docs/12/sql-vacuum.html", "          (Yes, No) [No]: ", "         ", "          --== STORAGE CONFIGURATION ==--", "         ", "         ", "          --== PKI CONFIGURATION ==--", "         ", "         ", "          --== APACHE CONFIGURATION ==--", "         ", "         ", "          --== SYSTEM CONFIGURATION ==--", "         ", "         ", "          --== MISC CONFIGURATION ==--", "         ", "         ", "          --== END OF CONFIGURATION ==--", "         ", "[ INFO  ] Stage: Setup validation", "[ INFO  ] Cleaning stale zombie tasks and commands", "         ", "          --== CONFIGURATION PREVIEW ==--", "         ", "          Application mode                        : both", "          Default SAN wipe after delete           : False", "          Host FQDN                               : nsednev-he-1.qa.lab.tlv.redhat.com", "          Firewall manager                        : firewalld", "          Update Firewall                         : True", "          Set up Cinderlib integration            : False", "          Configure local Engine database         : True", "          Set application as default page         : True", "          Configure Apache SSL                    : True", "          Engine database host                    : localhost", "          Engine database port                    : 5432", "          Engine database secured connection      : False", "          Engine database host name validation    : False", "          Engine database name                    : engine", "          Engine database user name               : engine", "          Engine installation                     : True", "          PKI organization                        : qa.lab.tlv.redhat.com", "          Renew PKI                               : False", "          Set up ovirt-provider-ovn               : True", "          Grafana integration                     : False", "          Configure WebSocket Proxy               : True", "          DWH installation                        : True", "          DWH database host                       : localhost", "          DWH database port                       : 5432", "          DWH database secured connection         : False", "          DWH database host name validation       : False", "          DWH database name                       : ovirt_engine_history", "          DWH database user name                  : ovirt_engine_history", "          Backup DWH database                     : True", "          Configure VMConsole Proxy               : True", "[ INFO  ] Cleaning async tasks and compensations", "[ INFO  ] Unlocking existing entities", "[ INFO  ] Checking the Engine database consistency", "[ INFO  ] Stage: Transaction setup", "[ INFO  ] Stopping engine service", "[ INFO  ] Stopping ovirt-fence-kdump-listener service", "[ INFO  ] Stopping dwh service", "[ INFO  ] Stopping vmconsole-proxy service", "[ INFO  ] Stopping websocket-proxy service", "[ INFO  ] Stage: Misc configuration (early)", "[ INFO  ] Updating PostgreSQL configuration", "[ INFO  ] Stage: Package installation", "[ INFO  ] Stage: Misc configuration", "[ INFO  ] Upgrading CA", "[ INFO  ] Updating OVN SSL configuration", "[ INFO  ] Updating OVN timeout configuration", "[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump'.", "[ INFO  ] Creating/refreshing DWH database schema", "[ INFO  ] Configuring WebSocket Proxy", "[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump'.", "[ INFO  ] Creating/refreshing Engine database schema", "[ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/create_views.sql", "[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed", "[ INFO  ] DNF Performing DNF transaction rollback", "[WARNING] Rollback of DWH database postponed to Stage \"Clean up\"", "[ INFO  ] Rolling back database schema", "[ INFO  ] Clearing Engine database engine", "[ INFO  ] Restoring Engine database engine", "[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20210503184749.g1a6x721.dump' to database localhost:engine.", "[ INFO  ] Stage: Clean up", "          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210503184714-xhxzrg.log", "[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210503184848-setup.conf'", "[WARNING] Rollback of DWH database started", "          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.", "[ INFO  ] Clearing DWH database ovirt_engine_history", "[ INFO  ] Restoring DWH database ovirt_engine_history", "[ INFO  ] Restoring file '/var/lib/ovirt-engine-dwh/backups/dwh-20210503184736.tl1ovc35.dump' to database localhost:ovirt_engine_history.", "[ INFO  ] Stage: Pre-termination", "[ INFO  ] Stage: Termination", "[ ERROR ] Execution of setup failed"]}
[ INFO  ] TASK [redhat.rhv.engine_setup : Clean temporary files]
[ INFO  ] changed: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ INFO  ] changed: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Create destination directory]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch the value of HOST_KEY_CHECKING]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get the username running the deploy]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Register the engine FQDN as a host]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ INFO  ] changed: [localhost -> nsednev-he-1.qa.lab.tlv.redhat.com]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch logs from the engine VM]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Create destination directory]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Clean local storage pools]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Destroy local storage-pool localvm36a8agza]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Undefine local storage-pool localvm36a8agza]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Destroy local storage-pool 6d8b651b-6666-4aa1-8dec-b321e2e9adac]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Undefine local storage-pool 6d8b651b-6666-4aa1-8dec-b321e2e9adac]
[ INFO  ] changed: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20210503185122.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20210503182906-b1kwig.log


Version-Release number of selected component (if applicable):
rhvm-appliance-4.4-20210402.1.el8ev.x86_64
ovirt-hosted-engine-ha-2.4.6-1.el8ev.noarch
ovirt-hosted-engine-setup-2.5.0-2.el8ev.noarch
Red Hat Enterprise Linux release 8.4 (Ootpa)
Linux 4.18.0-304.el8.x86_64 #1 SMP Tue Apr 6 05:19:59 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

How reproducible:
100%

Steps to Reproduce:
1.Deploy HE over NFS with latest bits and upgrade the engine's bits to latest.
2.Bump up the host cluster and the data center to 4.6 from 4.5.
3.Make the backup and try to restore on the same cleanly reprovisioned host.

Actual results:
Restore fails.

Expected results:
Restore should succeed.

Additional info:
Logs from host alma03 attached.