Bug 1991171

Documentation is stating wrong requirement only about major versions:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/administration_guide/index#Restoring_a_Backup_with_the_engine-backup_Command
With major changes done within 4.4.z we can't really keep supporting restoring on any 4.4.z

Documentation should also ask to upgrade the appliance deployment before restoring from backup, thought the error massage can be improved as well.

Some comments/thoughts:

1. This behavior is deliberate, started with the fix to bug 1932392.

2. One possible solution is that we'll release an appliance with every engine release. Right now that's not very easy to commit to, as it involves quite some manual work.

3. Another solution/workaround is to patch the code, or add an enginevm_before_engine_setup hook, to update relevant/all packages. This should be rather easy, but won't work if the updated packages are not available - e.g. on a disconnected system.

4. We can also rethink bug 1932392, although I do not think there is any fault in the reasoning there, in theory.

Another workaround: Use he_pause_before_engine_setup, so that you can login to the engine machine, register/subscribe/add channels, update packages, then continue. See also bug 1960188, bug 1959273.

Another solution, more complex but might be worth considering:

Add another option to hosted-engine --deploy, e.g. "--restore-from-image", which would allow restoring the engine VM from an image in some format (e.g. qcow/ova) instead of the appliance, and provide tools/docs for taking a backup of the engine machine in that format.

If we go that way, I'd personally never consider it as the only option - I think we should always keep the option to "start from scratch" using --restore-from-file, also to clean accumulated dirt on the VM - and such a '--restore-from-image' option would likely take more time/space to use, for both backups and restores - because the image will also include exactly this "dirt", specifically also logs, databases, etc. - but be much simpler to use.

In a way, we already do this - that's "he_appliance_ova" - but:
1. We do not document how, or provide tools for, converting an existing machine to such an image
2. We do not test this, so will likely run into issues initially.

Right now, the engine refuses to export the engine VM - this tries to create a snapshot on it, which fails with:

2021-08-09 14:01:37,354+02 WARN  [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-283767) [0bba83fc-fcce-4fcd-a531-1d46ba94eafc] Validation of action 'CreateSnapshotForVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__CREATE,VAR__TYPE__SNAPSHOT,ACTION_TYPE_FAILED_CANNOT_RUN_ACTION_ON_NON_MANAGED_VM
2021-08-09 14:01:37,390+02 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engine-Thread-283767) [0bba83fc-fcce-4fcd-a531-1d46ba94eafc] Failed to create VM snapshot

I think first and foremost we should document the need to upgrade.
We can think of even implementing enginevm_before_engine_setup hook by default attempting to update - it would still rely on having channels properly configured but with proper documentation (and more suggestive error message) it should be fine. In fact, I would start with that
- suggest to upgrade in engine-backup on version mismatch
- fix docs to require upgrading prior to restore using one of the methods (hook or pause), and fix the "major version" in comment #2.

(In reply to Michal Skrivanek from comment #7)
> I think first and foremost we should document the need to upgrade.

Not sure what you mean here. The only thing users can do in this flow
is "upgrade" the appliance before restoring. That does not help if it's
too old.

> We can think of even implementing enginevm_before_engine_setup hook by
> default attempting to update - it would still rely on having channels
> properly configured but with proper documentation (and more suggestive error
> message) it should be fine.

If we decide to patch the code, no need to implement that as a hook -
can be done directly. A hook is useful as a workaround.

Not sure how a code patch (as opposed to a custom hook) can help with
registration, channels, etc.

> In fact, I would start with that
> - suggest to upgrade in engine-backup on version mismatch

Text suggestions are welcome.

> - fix docs to require upgrading prior to restore using one of the methods
> (hook or pause), and fix the "major version" in comment #2.

Yes, that definitely requires a doc update - we now require strictly >=, not
just major.minor.

I thought about some other option: If restore failed, pause the deployment and let the user fix and continue, like we did in bug 1893385.

QE: To reproduce/verify this bug:

1. Deploy hosted-engine
2. Update the engine to the latest available one.
3. Take a backup with engine-backup.
4. Deploy with --restore-from-file, with an appliance older than the one in (2.), but >= 4.4.7.3 (see bug 1932392). Often, that's your only choice, because we do not release an appliance per each engine release.

With a broken version, it will fail.

With a fixed version, it will prompt the user with something like:

[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Run engine-backup]
[ INFO  ] engine-backup --mode=restore failed:
         FATAL: Backup was created by version '4.4.8.4_master' and can not be restored using the installed version 4.4.7.6
         You can now connect from this host to the bootstrap engine VM using ssh as root and the temporary IP address - 192.168.222.83 - and fix this issue. Please continue only after the backup is restored.
         To retry the command that failed, you can run, on the bootstrap engine VM:
         engine-backup --mode=restore --file=/root/engine_backup --provision-all-databases --restore-permissions
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Create temporary lock file]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Pause execution until /tmp/ansible.oyhxe1k2_he_setup_lock is removed, delete it once ready to proceed]

Then, you should be able to:

1. Ssh from the deploy host to the local engine vm using the noted IP address (192.168.222.83, in the example above)
2. Update the engine to at least the backup version (4.4.7.6, in above example),  or likely latest. For RHV, this most likely means to register/subscribe/configure channels as needed, then 'dnf update ovirt-engine'.
3. Run the restore command as noted (in the example)
4. Once it finishes successfully, you can logout from the engine VM, and remove the lock file on the deploy host (/tmp/ansible.oyhxe1k2_he_setup_lock, in the example above).
5. Deploy should then continue successfully.

Steps were followed and procedure passed, but it has to be properly documented by doc team ( they have separate bug for that )

Eli, two notes about the doc text:

1. "if the latest appliance is older than the latest Manager" - not necessarily the _latest_ engine (manager), but the one used when taking the backup.

2. "and fix the restore issue." - it might be worth adding that this particular error should be handled by registering the engine machine, configuring channels, updating the engine packages, and then trying again to run the restore command. See comment 10 for details. Instead of the doc-text, you might want to add this to the documentation, in bug 1991914.

Thanks!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHV Engine and Host Common Packages security update [ovirt-4.4.8]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3461

Due to QE capacity, we are not going to cover this issue in our automation