Description of problem: during 3.3 to 3.4 upgrade (av4) setup tries to create the plpgsql lanauge although it already exists psql:/var/lib/ovirt-engine/backups/engine-20140320150209.6Eb70h.sql:16: ERROR: language "plpgsql" already exists attaching upgrade log
Created attachment 876827 [details] setup log
Currently there is no real problem - we ignore the errors of psql and just log them. So it bothers just those that are looking at the log. The real problem is that we might fail to notice real errors. A proper fix can be one of: 1. massage the dump a bit before restoring it, as was done by dbscripts/restore.sh until recently: sed -i -e '/^CREATE DATABASE /d' -e '/ALTER DATABASE /d' -e '/^DROP /d' -e '/^CREATE SCHEMA/d' -e '/^ALTER TABLE ONLY public\./d' -e '/^ALTER FUNCTION public\.uuid_/d' -e '/^CREATE PROCEDURAL LANGUAGE plpgsql/d' -e '/^ALTER PROCEDURAL LANGUAGE plpgsql/d' -e 's/^CREATE FUNCTION uuid_/CREATE OR REPLACE FUNCTION uuid_/g' -e 's?/tmp?'`pwd`'?' -e 's?\$\$PATH\$\$?'`pwd`'?' restore.sql and than set ON_ERROR_STOP and fail if psql fails 2. Don't change current code but check the errors and only ignore expected ones, and raise an exception about unexpected ones. engine-backup currently does that. This will not help someone just reading the log - the errors will still be there, unless we also stop logging them (or log them filtered).
Created attachment 1073609 [details] engine upgrade Verified with upgrade from 3.5.4 to 3.6. Log file attached.
Gonza - seems like you tried a normal successful upgrade. This bug is about rollback - where engine-setup failed and tries to rollback. To reproduce/verify you need to: 1. somehow make it 'cleanly' fail. This can be done e.g. by adding an invalid file /usr/share/ovirt-engine/dbscripts/upgrade/03_05_9999_test.sql (or 03_06_9999_test.sql for upgrades to/in 3.6) 2. make pg_restore (or psql) then emit errors during restoring the backed up dump - e.g. by filling up the disk of pg or something like that. In an unfixed version rollback will continue silently, in a fixed version it will output something to the user. See also bug 1244720 for a concrete example. I think you can still keep this bug "verified" - otherwise, bug 1244720 would not have happened... Setting needinfo on you in case you feel like some more tests are due.
Created attachment 1074383 [details] upgrade fail
From log file: # sed -n 28840,28846p /tmp/ovirt-engine-setup-20150917120502-64dwd1.log 2015-09-17 12:09:21 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:780 db restore rc 1 stderr ['pg_restore: [archiver (db)] Error while PROCESSING TOC:', 'pg_restore: [archiver (db)] Error from TOC entry 2401; 2612 16388 PROCEDURAL LANGUAGE plpgsql engine', 'pg_restore: [archiver (db)] could not execute query: ERROR: language "plpgsql" already exists', ' Command was: CREATE PROCEDURAL LANGUAGE plpgsql;', '', '', '', 'WARNING: errors ignored on restore: 1'] 2015-09-17 12:09:21 ERROR otopi.ovirt_engine_setup.engine_common.database database.restore:795 Errors while restoring engine database, please check the log file for details 2015-09-17 12:09:21 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:800 Errors unfiltered during restore: pg_restore: [archiver (db)] Error from TOC entry 2401; 2612 16388 PROCEDURAL LANGUAGE plpgsql engine Command was: CREATE PROCEDURAL LANGUAGE plpgsql;
Verified with upgrade from: rhevm-3.5.7-0.1.el6ev.noarch to: rhevm-3.6.2-0.1.el6.noarch [ ERROR ] Engine database rollback failed: Command '/usr/bin/pg_restore -w -h localhost -p 5432 -U engine -d engine --jobs=2 --no-owner --no-privileges /var/lib/ovirt-engine/backups/engine-20160112164414.2GqhCB.dump' failed to execute: [Errno 8] Exec format error