Bug 1518802

Created attachment 1360388 [details]
setup log

Why not perform in-place upgrade?

We tried the in-place and it failed with the same error
ERROR: Data directory /var/opt/rh/rh-postgresql95/lib/pgsql/data is not empty!

We used default options too and the engine-setup failed with error, that /var/lib/pgsql/data does not exist. So we tried to not delete it and it failed also.

The error is here:

2017-11-29 16:04:22,624+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.dbmsupgrade plugin.execute:926 execute-output: ('/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup', '--upgrade', '--upgrade-from=postgresql') stderr:
 * upgrading from 'postgresql.service' to 'rh-postgresql95-postgresql.service'
 * Upgrading database.
ERROR: Data directory /var/opt/rh/rh-postgresql95/lib/pgsql/data is not empty!
ERROR: Upgrade failed.
 * See /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log for details.

but on a clean env /var/opt/rh/rh-postgresql95/lib/pgsql/data should be empty

The problem seems not to be in the environment and clean rh. We run into similar issue again with upgrade engine 4.1.9.1 -> 4.2.1.3.

It ran the upgrade to postgres 9.5 twice, like in the setup log I provided previously.

first upgrade:
2018-01-26 07:47:33,147+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.dbmsupgrade plugin.execute:926 execute-output: ('/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup', '-upgrade', '-upgrade-from=postgresql') stderr:

    upgrading from 'postgresql.service' to 'rh-postgresql95-postgresql.service'
    Upgrading database.
    Upgraded OK.
    WARNING: The configuration files were replaced by default configuration.
    WARNING: The previous configuration and data are stored in folder
    WARNING: /var/lib/pgsql/data.
    See /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log for details.
    2018-01-26 07:47:33,148+0200 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.db.dbmsupgrade postgres.prepare:793 PostgreSQL has been successfully upgraded, starting the new instance (rh-postgresql95-postgresql).

delete old db:
2018-01-26 07:47:34,338+0200 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.db.dbmsupgrade postgres.commit:848 Cleaning the previous PostgreSQL data directory

second upgrade:
2018-01-26 07:47:34,511+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.dbmsupgrade plugin.execute:926 execute-output: ('/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup', '-upgrade', '-upgrade-from=postgresql') stderr:

    upgrading from 'postgresql.service' to 'rh-postgresql95-postgresql.service'
    ERROR: config file /var/lib/pgsql/data/postgresql.conf is not readable or does not exist
    FATAL: Old cluster in '/var/lib/pgsql/data' does not seem to be initialized

Created attachment 1386551 [details]
upgrade log 4.1.9 -> 4.2.1

The issue seams here:

2018-01-26 07:47:09,493+0200 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.dbmsupgrade dbmsupgrade._customization:87 Engine DB and DWH one are on two distinct PostgreSQL instances and both of them have to be upgraded.

The setup assumes that Engine DB and DWH DB are on two distinct DBMS instances while instead they are on the same DBMS.
This check https://github.com/oVirt/ovirt-dwh/blob/master/packaging/setup/plugins/ovirt-engine-setup/ovirt-engine-dwh/db/dbmsupgrade.py#L69 for some reason fails.

I think it could be just another side effects of https://bugzilla.redhat.com/show_bug.cgi?id=1535935

Didi?

OK,
I got it!

In your answer-file you have:
OVESETUP_DB/port=str:5432
OVESETUP_DWH_DB/port=int:5432

and so the check on https://github.com/oVirt/ovirt-dwh/blob/master/packaging/setup/plugins/ovirt-engine-setup/ovirt-engine-dwh/db/dbmsupgrade.py#L69 simply failed due to the different data type (str vs int).

Lucie, how did you got that answerfile?

On a plain answer file from 4.1.8 engine-setup I got 
 OVESETUP_DB/port=int:5432
 OVESETUP_DWH_DB/port=int:5432
so it should work as expected there.

Not sure on how you got
 OVESETUP_DB/port=str:5432

Nice catch, but I wonder if we should somehow make the test more robust, if possible. Such as mark the pg instance as "upgraded" inside itself (perhaps pg already has a way to do this) and then check again before actually upgrading.

I'm not sure where I got the first answer file, but now in tests it was generated with ansible from template
https://github.com/oVirt/ovirt-ansible-engine-setup/blob/master/templates/answerfile_4.2_upgrade.txt.j2
but it is deleted in the end of upgrade and I have only the one generated with egine-setup after. And as you say, in the answer file is mixed db port as int and str.

4.2 answer file generated with port=str:xxx not int. When old answer file is used with mixed type by the port, upgrade did not fail (both engine+dwh and separate dwh).
Also upgrade automation tests have no problem with answer file.

verified in ovirt-engine-4.2.2.4-0.1.el7.noarch

This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.