Bug 1573091

Summary: Do no force DB patch version and settings in setup for remote databases.
Product: [oVirt] ovirt-engine Reporter: Yaniv Lavi <ylavi>
Component: Setup.EngineAssignee: Yedidyah Bar David <didi>
Status: CLOSED CURRENTRELEASE QA Contact: Lucie Leistnerova <lleistne>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.3.2CC: bugs, lsvaty, ylavi
Target Milestone: ovirt-4.2.4Flags: rule-engine: ovirt-4.2+
rule-engine: exception+
lsvaty: testing_ack+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.2.4 Doc Type: If docs needed, set a value
Doc Text:
engine-setup now allows using a remote PostgreSQL database with a different Z version - e.g. 9.5.9 client (the engine machine) can use a 9.5.8 remote database server. engine-setup also allows forcing it to ignore all PostgreSQL sanity/configuration checks. Doc team: See also comment 6 for latter. I'd rather not include the details in the doc text. For oVirt I added text to [1]. For RHV we might want a KB article. [1] https://ovirt.org/develop/developer-guide/engine/engine-setup/
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-26 08:39:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yaniv Lavi 2018-04-30 07:51:59 UTC
Description of problem:
Currently we force users to specific DB version and setting for postgresql instance in setup of a remote database.
If the database is remote, we can't force setting as it is managed by the user.
We can recommend or warn, but must have a way to override this.

I also do not think we should force a specific patch version for the database, since there should be no differences in these versions other than bug fixes.

Version-Release number of selected component (if applicable):
4.2.0

How reproducible:
100%

Steps to Reproduce:
1. Create a postgresql database version 9.5.12.
2. Add custom settings to auto vacuum and connection pool.
3. Run engine setup and choose remote database
4. Enter the database details.


Actual results:
Fails on version and settings.

Expected results:
Warn on setting and do not fail on version checking.

Additional info:

Comment 1 Yedidyah Bar David 2018-05-01 05:19:04 UTC
(In reply to Yaniv Lavi from comment #0)

Please be more specific:

> Expected results:
> Warn on setting

Only warn, on all of them?

None should be mandatory?

Just warn? Also prompt and ask for confirmation?

> and do not fail on version checking.

At all?

Even different major? E.g. 9.5 client with 8.4 (e.g. el6) server?

Prompt, saying rollback might not be possible?

Do not backup if impossible? Quoting pg_dump(1):

       Because pg_dump is used to transfer data to newer versions of
       PostgreSQL, the output of pg_dump can be expected to load into
       PostgreSQL server versions newer than pg_dump's version.  pg_dump can
       also dump from PostgreSQL servers older than its own version.
       (Currently, servers back to version 7.0 are supported.) However,
       pg_dump cannot dump from PostgreSQL servers newer than its own major
       version; it will refuse to even try, rather than risk making an invalid
       dump. Also, it is not guaranteed that pg_dump's output can be loaded
       into a server of an older major version — not even if the dump was
       taken from a server of that version. Loading a dump file into an older
       server may require manual editing of the dump file to remove syntax not
       understood by the older server. Use of the --quote-all-identifiers
       option is recommended in cross-version cases, as it can prevent
       problems arising from varying reserved-word lists in different
       PostgreSQL versions.

Or perhaps always try to backup but do not fail setup if backup failed? And probably always try to restore if backup succeeded, regardless of versions?

Comment 2 Yedidyah Bar David 2018-05-01 05:28:15 UTC
See also thread "Upgrade to 4.1":

http://lists.ovirt.org/pipermail/users/2017-April/thread.html#81383

Comment 3 Yedidyah Bar David 2018-05-03 07:03:59 UTC
90866 only removes the check for same Z, keeping same X.Y requirement, no other change. More answers, also to comment 1, are still welcome.

See also a newer thread, starting here:

http://lists.ovirt.org/pipermail/users/2018-May/088717.html

Comment 4 Yaniv Lavi 2018-05-03 13:20:48 UTC
(In reply to Yedidyah Bar David from comment #1)
> 
> Only warn, on all of them?

Yes, you can note this in the machine.
That the user didn't optimize the DB.

> 
> None should be mandatory?

No. It makes no sense to force this on a managed postgresql.

> 
> > and do not fail on version checking.
> 
> At all?
> 
 
If we support 9.5, any 9.5.x should work.
Not an exact match or at least allow to override.

Comment 5 Sandro Bonazzola 2018-05-07 07:58:52 UTC
(In reply to Yaniv Lavi from comment #4)
> (In reply to Yedidyah Bar David from comment #1)
> > 
> > Only warn, on all of them?
> 
> Yes, you can note this in the machine.
> That the user didn't optimize the DB.
> 
> > 
> > None should be mandatory?
> 
> No. It makes no sense to force this on a managed postgresql.

Are you sure about this? Not enforcing some of the configuration, if not all, may make the deployment, if not functional, pretty difficult to support.


> 
> > 
> > > and do not fail on version checking.
> > 
> > At all?
> > 
>  
> If we support 9.5, any 9.5.x should work.
> Not an exact match or at least allow to override.

This is handled by https://gerrit.ovirt.org/#/c/90866/ just checking for 9.5 instead of 9.5.z

Comment 6 Yedidyah Bar David 2018-05-07 09:18:48 UTC
90958 makes engine-setup ignore (and warn about) all invalid postgresql configuration, if answer file includes:

OVESETUP_CONFIG/forceInvalidPGConf=bool:True

I hope this should be enough for everyone.

People that intend to use this option should know very well what they are doing. This is definitely not supported and IMO will not be documented anywhere except for this bugzilla comment.

Whoever that finds this comment and intends to use it, or even worse - to recommend others to use it, please don't. Instead, for your current needs, fix pg conf, and if you think engine-setup is too strict, please open another bug/RFE to make it less so, similarly to patch 90866.

Comment 7 Yedidyah Bar David 2018-05-07 09:51:26 UTC
BTW, this applies to both local and remote databases. We do not keep in postinstall the answer to the questions asking about remote/local and automatic/manual, so do not know this on upgrades. Code that has to decide about this checks if db host is 'localhost', which is a kind of hack. I do not see a reason to check this - if we allow to enforce, do this for local db as well.

Comment 8 Yedidyah Bar David 2018-05-16 08:18:33 UTC
Summary of current patches (not merged yet):

90866 makes the test require only same X.Y versions of postgresql client and server. So e.g. 9.5.8 will work with 9.5.9 but not with 9.6.something.

90958 adds an answer-file boolean key that if set ignores all tests (but still warns).

Yaniv, is this enough?

Comment 9 Yaniv Lavi 2018-05-17 11:01:29 UTC
(In reply to Yedidyah Bar David from comment #8)
> Summary of current patches (not merged yet):
> 
> 90866 makes the test require only same X.Y versions of postgresql client and
> server. So e.g. 9.5.8 will work with 9.5.9 but not with 9.6.something.
> 
> 90958 adds an answer-file boolean key that if set ignores all tests (but
> still warns).
> 
> Yaniv, is this enough?

Yes.

Comment 10 Lucie Leistnerova 2018-05-31 13:45:10 UTC
Engine with postgres 9.5.9 installed with remote db postgres 9.5.13 without any error. OVESETUP_CONFIG/forceInvalidPGConf=bool:True shows only warnings 

[WARNING] Ignoring invalid PostgreSql configuration for DWH: host = '10.37.140.8', key = 'autovacuum_vacuum_scale_factor', current value = '0.2'. It is required to be at most '0.01'.
....

Without OVESETUP_CONFIG/forceInvalidPGConf engine-setup requires config changes on remote machine.

Engine with postgres 9.5.9 can't be installed with remote db postgres 9.2.23 even with OVESETUP_CONFIG/forceInvalidPGConf set to true.

[ ERROR ] Failed to execute stage 'Environment customization': Please upgrade engine PostgreSQL server to 9.5.9 and retry.


verified in ovirt-engine-setup-4.2.4-0.1.el7.noarch

Comment 11 Yedidyah Bar David 2018-06-03 06:38:39 UTC
(In reply to Lucie Leistnerova from comment #10)
> Engine with postgres 9.5.9 can't be installed with remote db postgres 9.2.23
> even with OVESETUP_CONFIG/forceInvalidPGConf set to true.
> 
> [ ERROR ] Failed to execute stage 'Environment customization': Please
> upgrade engine PostgreSQL server to 9.5.9 and retry.
> 

Lucie - thanks for checking this, I didn't think about this flow. It's in a different part in the code - not in checking configuration but in checking if we need to upgrade (from 9.2 to 9.5).

Yaniv - is that ok?

Current code will refuse using a remote server whose X.Y version is lower than the client's. It should be easy to allow this if the user set forceInvalidPGConf, if you want.

Comment 12 Yaniv Lavi 2018-06-06 15:23:04 UTC
(In reply to Yedidyah Bar David from comment #11)
> (In reply to Lucie Leistnerova from comment #10)
> > Engine with postgres 9.5.9 can't be installed with remote db postgres 9.2.23
> > even with OVESETUP_CONFIG/forceInvalidPGConf set to true.
> > 
> > [ ERROR ] Failed to execute stage 'Environment customization': Please
> > upgrade engine PostgreSQL server to 9.5.9 and retry.
> > 
> 
> Lucie - thanks for checking this, I didn't think about this flow. It's in a
> different part in the code - not in checking configuration but in checking
> if we need to upgrade (from 9.2 to 9.5).
> 
> Current code will refuse using a remote server whose X.Y version is lower
> than the client's. It should be easy to allow this if the user set
> forceInvalidPGConf, if you want.

Yes it is fine.

Comment 13 Sandro Bonazzola 2018-06-26 08:39:42 UTC
This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.