Description of problem:
during 3.3 to 3.4 upgrade (av4) setup tries to create the plpgsql lanauge although it already exists
psql:/var/lib/ovirt-engine/backups/engine-20140320150209.6Eb70h.sql:16: ERROR: language "plpgsql" already exists
attaching upgrade log
Created attachment 876827 [details]
Currently there is no real problem - we ignore the errors of psql and just log them. So it bothers just those that are looking at the log.
The real problem is that we might fail to notice real errors.
A proper fix can be one of:
1. massage the dump a bit before restoring it, as was done by dbscripts/restore.sh until recently:
sed -i -e '/^CREATE DATABASE /d' -e '/ALTER DATABASE /d' -e '/^DROP /d' -e '/^CREATE SCHEMA/d' -e '/^ALTER TABLE ONLY public\./d' -e '/^ALTER FUNCTION public\.uuid_/d' -e '/^CREATE PROCEDURAL LANGUAGE plpgsql/d' -e '/^ALTER PROCEDURAL LANGUAGE plpgsql/d' -e 's/^CREATE FUNCTION uuid_/CREATE OR REPLACE FUNCTION uuid_/g' -e 's?/tmp?'`pwd`'?' -e 's?\$\$PATH\$\$?'`pwd`'?' restore.sql
and than set ON_ERROR_STOP and fail if psql fails
2. Don't change current code but check the errors and only ignore expected ones, and raise an exception about unexpected ones. engine-backup currently does that.
This will not help someone just reading the log - the errors will still be there, unless we also stop logging them (or log them filtered).
Created attachment 1073609 [details]
Verified with upgrade from 3.5.4 to 3.6.
Log file attached.
Gonza - seems like you tried a normal successful upgrade.
This bug is about rollback - where engine-setup failed and tries to rollback.
To reproduce/verify you need to:
1. somehow make it 'cleanly' fail. This can be done e.g. by adding an invalid
(or 03_06_9999_test.sql for upgrades to/in 3.6)
2. make pg_restore (or psql) then emit errors during restoring the backed
up dump - e.g. by filling up the disk of pg or something like that.
In an unfixed version rollback will continue silently, in a fixed version
it will output something to the user.
See also bug 1244720 for a concrete example.
I think you can still keep this bug "verified" - otherwise, bug 1244720 would
not have happened...
Setting needinfo on you in case you feel like some more tests are due.
Created attachment 1074383 [details]
From log file:
# sed -n 28840,28846p /tmp/ovirt-engine-setup-20150917120502-64dwd1.log
2015-09-17 12:09:21 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:780 db restore rc 1 stderr ['pg_restore: [archiver (db)] Error while PROCESSING TOC:', 'pg_restore: [archiver (db)] Error from TOC entry 2401; 2612 16388 PROCEDURAL LANGUAGE plpgsql engine', 'pg_restore: [archiver (db)] could not execute query: ERROR: language "plpgsql" already exists', ' Command was: CREATE PROCEDURAL LANGUAGE plpgsql;', '', '', '', 'WARNING: errors ignored on restore: 1']
2015-09-17 12:09:21 ERROR otopi.ovirt_engine_setup.engine_common.database database.restore:795 Errors while restoring engine database, please check the log file for details
2015-09-17 12:09:21 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:800 Errors unfiltered during restore:
pg_restore: [archiver (db)] Error from TOC entry 2401; 2612 16388 PROCEDURAL LANGUAGE plpgsql engine
Command was: CREATE PROCEDURAL LANGUAGE plpgsql;
Verified with upgrade from:
[ ERROR ] Engine database rollback failed: Command '/usr/bin/pg_restore -w -h localhost -p 5432 -U engine -d engine --jobs=2 --no-owner --no-privileges /var/lib/ovirt-engine/backups/engine-20160112164414.2GqhCB.dump' failed to execute: [Errno 8] Exec format error