Bug 1666065

+++ This bug was initially created as a clone of Bug #1518697 +++

Description of problem:
When generating answer file after succssful run of engine-setup (upgrade 4.1 to 4.2) some values are not stored. Therefore automated upgrade with this answer file requries manual attendence. 

Relevenat otopi questions:

This issue is blocking automated testing of upgrade process.
[WARNING] This release requires PostgreSQL server 9.5.9 but the engine database is currently hosted on PostgreSQL server 9.2.23
          This tool can automatically upgrade PostgreSQL. Automatically upgrade? (Yes, No) [Yes]: 
          The size of the PostgreSQL instance used by engine database is 64 MB.
          The destination DB will be created under '/var/opt/rh/rh-postgresql95/lib/pgsql/data' where you currently have 25.5 GB available.
          This tool can perform the DBMS upgrade:
           - copying the data files to the new instance
           - in-place hard-linking them
          Upgrading in-place is faster and doesn't require 64 MB free on the target directory, but it cannot be automatically rolled-back on failures. Please ensure you are able to restore your DBMS instance using engine-backup or other means (DB backup, FS backup, LVM snapshot).
          The in-place upgrade has the data files of the source instance on the same filesystem of the target instance (hard-links).
          Do you want to perform an in-place upgrade? (Yes, No) [No]: 
          Do you want to automatically clean up the old data directory on success to reclaim its space (64 MB)? (Yes, No) [Yes]: 
[ INFO  ] Any further action on the DB will be performed only after PostgreSQL has been successfully upgraded to 9.5.
[ INFO  ] Engine DB and DWH one are on two distinct PostgreSQL instances and both of them have to be upgraded.
[WARNING] This release requires PostgreSQL server 9.5.9 but the DWH database is currently hosted on PostgreSQL server 9.2.23
          This tool can automatically upgrade PostgreSQL. Automatically upgrade? (Yes, No) [Yes]: 
          The size of the PostgreSQL instance used by DWH database is 64 MB.
          The destination DB will be created under '/var/opt/rh/rh-postgresql95/lib/pgsql/data' where you currently have 25.5 GB available.
          This tool can perform the DBMS upgrade:
           - copying the data files to the new instance
           - in-place hard-linking them
          Upgrading in-place is faster and doesn't require 64 MB free on the target directory, but it cannot be automatically rolled-back on failures. Please ensure you are able to restore your DBMS instance using engine-backup or other means (DB backup, FS backup, LVM snapshot).
          The in-place upgrade has the data files of the source instance on the same filesystem of the target instance (hard-links).
          Do you want to perform an in-place upgrade? (Yes, No) [No]: 
          Do you want to automatically clean up the old data directory on success to reclaim its space (64 MB)? (Yes, No) [Yes]: 
[ INFO  ] Any further action on the DB will be performed only after PostgreSQL has been successfully upgraded to 9.5.


Version-Release number of selected component (if applicable):
otopi-1.7.3-1.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. run engine-setup to upgrade rhv-4.1 to rhv-4.2
2. get generated answer file
3. run engine-setup --config--append=answerfile.txt

Actual results:
Requires manual input on stated questions.

Expected results:
Should not need manual interructions.

Additional info:
Please provide WA for this and specify which env values to inject to our answer file so I can lower the priority, for now it is AutomationBlocker.

--- Additional comment from Yedidyah Bar David on 2017-11-29 13:39:14 UTC ---

Is it really an automation blocker? Can't you use --accept-defaults? I realize this blocks automated testing of the non-default-answers flows, some of which are quite relevant, mainly testing rollback (do you do that?).

--- Additional comment from Lukas Svaty on 2017-11-29 13:54:15 UTC ---

Can't-do, we like to track changes among RHV releases, where accept-defaults would just ignore them.

--- Additional comment from Yedidyah Bar David on 2017-12-05 09:18:37 UTC ---

/me is wondering if we should fix instead bug 1396925...

--- Additional comment from Lukas Svaty on 2017-12-07 16:38:46 UTC ---

Didi do we have env vars we can set in answer file or outside of it to WA this issue in automation? So we will unblock QA for upgrade automation process until this o BZ#1396925 is resolved.

--- Additional comment from Yedidyah Bar David on 2017-12-10 06:52:18 UTC ---

Sadly, no.

I suggest to use accept-defaults for now. Perhaps you can make this optional, so you can easily turn off/on as needed, or run part of the tests with it (those that test further things, for which you do not care much about engine-setup itself) and part without it (where you actually want to test engine-setup).

--- Additional comment from Yedidyah Bar David on 2017-12-19 10:56:45 UTC ---

Lukas - now pushed and verified a patch for bug 1396925. Didn't merge yet. CI builds it though, so you can use the generated RPMs for testing, if you want.

Would you consider it a good enough solution for current bug as well?

Do you think it's a good time to get it into 4.2, or it's too late?

Merging it means you now have two kinds of answer files - one generated by engine-setup and one (optionally) by otopi. otopi's currently has no cli option to cause it to be generated - see the gerrit page.

To _use_ them, you keep using the existing --config-append option.

But, to use otopi's, you'd have to re-generate your answer file (by running manually with DIALOG/answerFile=str:/some/file and perhaps editing).

So to fully migrate QE to otopi's, you'd need to convert all your answer files. Nothing prevents doing this gradually, though. But once we merge it, we are much more likely to not ever fix anymore bugs like current, instead replying "Please use otopi's answerfile". What do you think?

--- Additional comment from Yedidyah Bar David on 2017-12-19 11:10:19 UTC ---

To clarify (also for Sandro):

I didn't do anything specific for current bug, but if we have agreement, I'd like to push a patch to engine-setup to replace the existing generated answer files (both the ones in /var/lib/ovirt-engine/setup/answers and the one of '--generate-answer') with otopi-generated ones.

In theory, we can do this also in the middle of z-stream, because officially we always say that answer files are valid only for the exact same version/env they were created with. But in reality, I expect such a change to cause some noise, so it's probably better to introduce it in .0 - 4.2.0 or 4.3.0.

--- Additional comment from Sandro Bonazzola on 2017-12-19 11:34:41 UTC ---

Agreed, targeting to 4.3.0

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-12-19 11:34:48 UTC ---

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

--- Additional comment from Lukas Svaty on 2018-01-08 07:48:32 UTC ---

Agreed with using OTOPI's answer file. If possible it would be preferred from QE side in 4.2, as this is currently there is no automated way to add non-default answers. However, I understood that this may cause significant regressions.

--- Additional comment from Yedidyah Bar David on 2018-01-08 08:03:04 UTC ---

There are two mostly-independent parts here:

1. Merging bug 1396925 and using its generated answer files explicitly. In theory this is harmless. In practice this has the risk, that if e.g. QE move on to verify everything with otopi-style files, they might not run into bugs caused by using old-style files, so will not notice them. So if we go that way, we should make sure we have good coverage of both.

2. Make the files generated by engine-setup (/var/lib and --generate-answer) also have the content of bug 1396925, otopi-style answer-files. As I wrote in comment 7, I expect this to cause some noise, although I didn't do a full analysis of the kinds of problems that we should expect. But do note that existing, old-style answer files will continue to work exactly the same way, so it's not that kind of breakage. Most noise, assuming we do not have bugs :-), will probably be from people noticing the change and wondering what they need to do, if at all. More noise will come from cases where we decide to not fix some bug because it does not happen with otopi-generated files, like current bug :-), because they then feel like we force them to do a rather drastic change (rewrite/regenerate all of their answer files) in the middle of a z-stream.

--- Additional comment from Yedidyah Bar David on 2018-01-08 09:48:19 UTC ---

Lukas - merged https://gerrit.ovirt.org/85551 . So once it's passing the change-queue you can use it from master snapshot. Please use it and tell me if you have issues, probably as comments on bug 1396925 or as new bugs. Now added there a comment about how to use etc.

Keeping current bug on 4.3 for now.

--- Additional comment from Yedidyah Bar David on 2018-01-10 08:11:53 UTC ---

Removing AutomationBlocker, it should now be possible to automate this using bug 1396925. Keeping current bug as-is for now, for 4.3.

--- Additional comment from Yedidyah Bar David on 2018-02-05 09:38:22 UTC ---

Lukas, I understand you asked to move this to 4.2.2 for QE to be able to use. I do not think you need this. You can use the functionality of bug 1396925 right now, without touching engine-setup - everything should work. Please ping me with any issues/questions. Please move bug back to 4.3, I do not want to make bug 1396925 behavior the default one in the middle of 4.2, at least not without significant testing.

--- Additional comment from Yedidyah Bar David on 2018-02-05 09:45:45 UTC ---

After a private discussion with Lukas, moving back to 4.3, and cloning to a test-only bug for testing in 4.2.

--- Additional comment from Lukas Svaty on 2018-02-05 10:02:33 UTC ---

We do not require it however, it's nice to have. After the conversation on IRC, this change had a huge impact on answer file and automated way of engine-setup, thus I recommend leaving it on 4.3.

--- Additional comment from Yedidyah Bar David on 2018-02-28 12:13:37 UTC ---

Tareq, why do you think it's still an AutomationBlocker? See also comment 13.

--- Additional comment from Yedidyah Bar David on 2018-02-28 12:26:57 UTC ---

Also note that I merged today [1] to otopi (master branch only), which changes the behavior a bit. Some relevant flows to test with it:

1.
engine-setup --otopi-environment="DIALOG/answerFile=str:/root/ans-otopi-1"

Kill it at 'PREVIEW'

engine-setup --config-append=/root/ans-otopi-1

2.
engine-setup --otopi-environment="DIALOG/answerFile=str:/root/ans-otopi-2" --accept-defaults

Kill it at 'PREVIEW'

engine-setup --config-append=/root/ans-otopi-2

Compare the behavior of (1.) and (2.). With [1], (2.) will have answers for more questions (all? not sure).

[1] https://gerrit.ovirt.org/88278

--- Additional comment from Tareq Alayan on 2018-03-07 15:35:11 UTC ---

Becuase it just broke our upgrade tests that was working before.

--- Additional comment from Yedidyah Bar David on 2018-03-07 15:44:25 UTC ---

OK, understood. So you might keep Regression, but I do not agree with AutomationBlocker. If the current state of 4.2 allows QE automation, with current bug unfixed, the bug does not block automation :-) Seriously, please invest time in bug 1396925, and then adapt your 4.2 automation to use it. That's the only way to get serious testing for it.

--- Additional comment from Tareq Alayan on 2018-03-11 08:24:41 UTC ---

It is not blocking *all* automation it only blocks testing upgrade automatically. 
Svaty will you modify our code to work with the new style answere file?

--- Additional comment from Yedidyah Bar David on 2018-03-11 09:40:22 UTC ---

Note that no *code* (logic) changes are needed, only data - generating new answer files (once) to replace your existing ones. And you do not need to change anything in where you *use* them - you pass them to '--config-append' just like the legacy ones - so can do this gradually.

--- Additional comment from Lukas Svaty on 2018-03-19 11:55:18 UTC ---

Tareq, you just need to regenerate answer file for upgrade inside the oVirt repository.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-08-22 14:29:47 UTC ---

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

--- Additional comment from Lucie Leistnerova on 2018-09-10 08:47:46 UTC ---

postgres upgrade (4.1 -> 4.3) answers stored and running other upgrade with the answer file doesn't need any more interaction

verified in ovirt-engine-setup-4.3.0-0.0.master.20180903111244.git94dce75.el7.noarch.rpm

--- Additional comment from Sandro Bonazzola on 2018-11-02 14:38:36 UTC ---

This bugzilla is included in oVirt 4.2.7 release, published on November 2nd 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.7 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

--- Additional comment from Sandro Bonazzola on 2018-11-02 14:50:38 UTC ---

Closed by mistake, moving back to qa -> verified