Bug 1078871 - SQL errors during rollback are not checked
Summary: SQL errors during rollback are not checked
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Yedidyah Bar David
QA Contact: Gonza
Depends On: 1176457 1244720
TreeView+ depends on / blocked
Reported: 2014-03-20 13:17 UTC by Ohad Basan
Modified: 2016-03-11 07:34 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-03-11 07:34:02 UTC
oVirt Team: Integration
Target Upstream Version:

Attachments (Terms of Use)
setup log (1.80 MB, text/x-log)
2014-03-20 13:17 UTC, Ohad Basan
no flags Details
engine upgrade (2.17 MB, text/plain)
2015-09-15 12:29 UTC, Gonza
no flags Details
upgrade fail (1.36 MB, text/plain)
2015-09-17 10:15 UTC, Gonza
no flags Details

System ID Priority Status Summary Last Updated
oVirt gerrit 37252 master MERGED packaging: setup: Report errors during db restore Never

Description Ohad Basan 2014-03-20 13:17:15 UTC
Description of problem:

during 3.3 to 3.4 upgrade (av4) setup tries to create the plpgsql lanauge although it already exists
psql:/var/lib/ovirt-engine/backups/engine-20140320150209.6Eb70h.sql:16: ERROR:  language "plpgsql" already exists

attaching upgrade log

Comment 1 Ohad Basan 2014-03-20 13:17:57 UTC
Created attachment 876827 [details]
setup log

Comment 2 Yedidyah Bar David 2014-03-26 13:58:46 UTC
Currently there is no real problem - we ignore the errors of psql and just log them. So it bothers just those that are looking at the log.

The real problem is that we might fail to notice real errors.

A proper fix can be one of:

1. massage the dump a bit before restoring it, as was done by dbscripts/restore.sh until recently:

        sed -i -e '/^CREATE DATABASE /d' -e '/ALTER DATABASE /d' -e '/^DROP /d' -e '/^CREATE SCHEMA/d' -e '/^ALTER TABLE ONLY public\./d' -e '/^ALTER FUNCTION public\.uuid_/d' -e '/^CREATE PROCEDURAL LANGUAGE plpgsql/d' -e '/^ALTER PROCEDURAL LANGUAGE plpgsql/d' -e 's/^CREATE FUNCTION uuid_/CREATE OR REPLACE FUNCTION uuid_/g' -e 's?/tmp?'`pwd`'?'  -e 's?\$\$PATH\$\$?'`pwd`'?' restore.sql

and than set ON_ERROR_STOP and fail if psql fails

2. Don't change current code but check the errors and only ignore expected ones, and raise an exception about unexpected ones. engine-backup currently does that.
This will not help someone just reading the log - the errors will still be there, unless we also stop logging them (or log them filtered).

Comment 5 Gonza 2015-09-15 12:29:54 UTC
Created attachment 1073609 [details]
engine upgrade

Verified with upgrade from 3.5.4 to 3.6.

Log file attached.

Comment 6 Yedidyah Bar David 2015-09-16 09:09:03 UTC
Gonza - seems like you tried a normal successful upgrade.

This bug is about rollback - where engine-setup failed and tries to rollback.

To reproduce/verify you need to:

1. somehow make it 'cleanly' fail. This can be done e.g. by adding an invalid
file /usr/share/ovirt-engine/dbscripts/upgrade/03_05_9999_test.sql
(or 03_06_9999_test.sql for upgrades to/in 3.6)

2. make pg_restore (or psql) then emit errors during restoring the backed
up dump - e.g. by filling up the disk of pg or something like that.

In an unfixed version rollback will continue silently, in a fixed version
it will output something to the user.

See also bug 1244720 for a concrete example.

I think you can still keep this bug "verified" - otherwise, bug 1244720 would
not have happened...

Setting needinfo on you in case you feel like some more tests are due.

Comment 7 Gonza 2015-09-17 10:15:14 UTC
Created attachment 1074383 [details]
upgrade fail

Comment 8 Gonza 2015-09-17 10:21:31 UTC
From log file:

#  sed -n 28840,28846p /tmp/ovirt-engine-setup-20150917120502-64dwd1.log

2015-09-17 12:09:21 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:780 db restore rc 1 stderr ['pg_restore: [archiver (db)] Error while PROCESSING TOC:', 'pg_restore: [archiver (db)] Error from TOC entry 2401; 2612 16388 PROCEDURAL LANGUAGE plpgsql engine', 'pg_restore: [archiver (db)] could not execute query: ERROR:  language "plpgsql" already exists', '    Command was: CREATE PROCEDURAL LANGUAGE plpgsql;', '', '', '', 'WARNING: errors ignored on restore: 1']
2015-09-17 12:09:21 ERROR otopi.ovirt_engine_setup.engine_common.database database.restore:795 Errors while restoring engine database, please check the log file for details
2015-09-17 12:09:21 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:800 Errors unfiltered during restore:

pg_restore: [archiver (db)] Error from TOC entry 2401; 2612 16388 PROCEDURAL LANGUAGE plpgsql engine
    Command was: CREATE PROCEDURAL LANGUAGE plpgsql;

Comment 10 Gonza 2016-01-13 08:08:58 UTC
Verified with upgrade from:

[ ERROR ] Engine database rollback failed: Command '/usr/bin/pg_restore -w -h localhost -p 5432 -U engine -d engine --jobs=2 --no-owner --no-privileges /var/lib/ovirt-engine/backups/engine-20160112164414.2GqhCB.dump' failed to execute: [Errno 8] Exec format error

Note You need to log in before you can comment on or make changes to this bug.