Created attachment 1691763 [details] Jenkins stdout + engine-setup log Description of problem: This Issue is a consiquence of the following BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1839665 The engine-setup fails to roll back due to "cannot use a string pattern on a bytes-like object". This occurred as a part of Jenkins flow for upgrading the 4.4 env to latest build rhv-4.4.1-1: Creating/refreshing Engine database schema\n[ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_04_0650_add_fk_to_vm_checkpoint_disk_map.sql\n[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed\n[ INFO ] DNF Performing DNF transaction rollback\n[ INFO ] DNF Repository ansible-2-rhel8-nightly-rhv-4.4 is listed more than once in the configuration\n[ INFO ] DNF Repository jb-eap-7.3-for-rhel-8-server-rpms is listed more than once in the configuration\n[ INFO ] DNF Repository rhel-8-openstack-cinderlib-rpms is listed more than once in the configuration\n[ ERROR ] DNF module 'dnf.history' has no attribute 'open_history'\n[WARNING] Rollback of DWH database postponed to Stage \"Clean up\"\n[ INFO ] Rolling back database schema\n[ INFO ] Clearing Engine database engine\n[ INFO ] Restoring Engine database engine\n[ INFO ] Restoring file '/var/lib/ovirt-engine/backups/engine-20200524094253.ncgb6klw.dump' to database localhost:engine.\n[ ERROR ] Engine database rollback failed: cannot use a string pattern on a bytes-like object\n[ INFO ] Stage: Clean up\n Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200524094034-xuwab0.log\n[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200524094347-setup.conf'\n[WARNING] Rollback of DWH database started\n This might be a long process, but it should be safe to start the engine service before it finishes, if needed.\n[ INFO ] Clearing DWH database ovirt_engine_history\n[ INFO ] Restoring DWH database ovirt_engine_history\n[ INFO ] Restoring file '/var/lib/ovirt-engine-dwh/backups/dwh-20200524094236.dep1o6wu.dump' to database localhost:ovirt_engine_history.\n[ ERROR ] DWH database rollback failed: cannot use a string pattern on a bytes-like object\n[ INFO ] Stage: Pre-termination\n[ INFO ] Stage: Termination\n[ ERROR ] Execution of setup failed", "stdout_lines": ["[ INFO ] Stage: Initializing", "[ INFO ] Stage: Environment setup", " Version-Release number of selected component (if applicable): rhv-4.4.1-1 How reproducible: Reproduced only once when running the Jenkins upgrade flow. Attaching engine setup std out print [1] and full log of the responsible Jenkins job [2], as well as the link to it [3] [1] engine_setup_stdout.txt [2] jenkins_job.log [3] https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rhv-ge-deploy-4.4/1066/consoleFull
Are there some reproduction steps? Otherwise this will be sanity only.
The reproduction steps are generally to make engine-setup fail after it already updated the database (perhaps preferably both engine and dwh DBs) - after 'Creating/refreshing * database schema' - and see that it rolls back correctly. Manually, you can do this simply by pressing Ctrl-C at the right time. Automatically, you can do something like: - install, setup and backup a 4.3 engine+dwh (no need to be done specifically for this flow, btw - you can take a backup done elsewhere) - Install 4.4 engine+dwh, and: otopi-debug-plugins - Restore the backup from first step - Run: OTOPI_FORCE_FAIL_STAGE=STAGE_MISC OTOPI_FORCE_FAIL_PRIORITY=PRIORITY_LAST engine-setup --offline --accept-defaults Alternatively, for verifying just current bug (and not real 4.3->4.4 upgrade), you can upgrade to same version: - install, setup and backup engine+dwh - engine-cleanup. To automate: engine-cleanup --otopi-environment='OVESETUP_CORE/remove=bool:True OVESETUP_CORE/engineStop=bool:True' (and also, until we fix bug 1811593: /bin/rm /etc/pki/ovirt-engine/keys/engine.p12) - restore and setup the same as previous flow In both cases, make sure you only get these errors: [ ERROR ] Failed to execute stage 'Misc configuration': Force Fail: stage 11 priority 90000 (or something similar, if you pressed Ctrl-C) [ ERROR ] Execution of setup failed Sadly, I now tried this, with grafana set up, and it did have another error, related to grafna's db user. Filed bug 1846212 for this, so if you try latter flow (meaning, all in 4.4), please do not configure grafana (for current bug).
(In reply to Yedidyah Bar David from comment #3) > - Run: OTOPI_FORCE_FAIL_STAGE=STAGE_MISC > OTOPI_FORCE_FAIL_PRIORITY=PRIORITY_LAST engine-setup --offline > --accept-defaults We default to 'Yes', in 'Configure grafana?', so: [snip] > Sadly, I now tried this, with grafana set up, and it did have another error, > related to grafna's db user. Filed bug 1846212 for this, so if you try > latter flow (meaning, all in 4.4), please do not configure grafana (for > current bug). To make it not configure grafana, for now, you can e.g.: OTOPI_FORCE_FAIL_PRIORITY=PRIORITY_LAST engine-setup --offline --accept-defaults --otopi-environment=OVESETUP_GRAFANA_CORE/enable=bool:False
Verified on ovirt-engine-setup-4.4.1.2-0.10.el8ev.noarch Rollback of DB is successful (no string pattern for bytes-like object error appears) when engine-setup fails after creating DB schemas. I ran the upgrade from 4.3 flow as that seemed to be the main point of this bug.
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.