Bug 1832905 - engine-backup --mode=verify is broken
Summary: engine-backup --mode=verify is broken
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Backup-Restore.Engine
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.4.1
: 4.4.1.4
Assignee: Yedidyah Bar David
QA Contact: Guilherme Santos
URL:
Whiteboard:
Depends On:
Blocks: 1848877 1868929
TreeView+ depends on / blocked
 
Reported: 2020-05-07 13:05 UTC by Yedidyah Bar David
Modified: 2020-08-18 13:28 UTC (History)
4 users (show)

Fixed In Version: ovirt-engine-4.4.1.4
Doc Type: Bug Fix
Doc Text:
In previous versions, `engine-backup --mode=verify` passed even if `pg_restore` emitted errors. The current release fixes this issue. The `engine-backup --mode=verify` command correctly fails if `pg_restore` emits errors.
Clone Of:
: 1848877 1868929 (view as bug list)
Environment:
Last Closed: 2020-08-05 06:24:50 UTC
oVirt Team: Integration
Embargoed:
pm-rhel: ovirt-4.4+
sbonazzo: blocker?
mtessun: planning_ack+
sbonazzo: devel_ack+
lleistne: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 108877 0 master MERGED packaging: engine-backup: Fail verify on pg_restore errors 2020-09-06 05:52:04 UTC
oVirt gerrit 108878 0 master MERGED packaging: engine-backup: Pass '-f -' to pg_restore 2020-09-06 05:52:03 UTC
oVirt gerrit 108941 0 ovirt-engine-4.3 MERGED packaging: engine-backup: Fail verify on pg_restore errors 2020-09-06 05:52:03 UTC
oVirt gerrit 108942 0 ovirt-engine-4.3 MERGED packaging: engine-backup: Pass '-f -' to pg_restore 2020-09-06 05:52:04 UTC
oVirt gerrit 109138 0 master MERGED packaging: engine-backup: Pass '-f -' to pg_restore extra users 2020-09-06 05:52:03 UTC
oVirt gerrit 109183 0 ovirt-engine-4.3 MERGED packaging: engine-backup: Pass '-f -' to pg_restore extra users 2020-09-06 05:52:03 UTC

Description Yedidyah Bar David 2020-05-07 13:05:04 UTC
Description of problem:

engine-backup --mode=verify:

1. Does not pass '-f -' to pg_restore, which makes it fail, because in recent versions it does not default to stdout anymore.

2. Despite (1.), engine-backup does not fail or log this.

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Always

Steps to Reproduce:
1. engine-backup --file=f1
2. engine-backup --mode=verify --file=f1
3.

Actual results:
Passes, although pg_restore failed

Expected results:
Should pass, and if pg_restore still fails should abort and log the errors

Additional info:

I already pushed two separate patches for this to gerrit, then decided to also open this bug, and consider backporting to 4.3, mainly for (1.) - so that if pg_restore fails we notice.

Verifying this with each patch separately was easy, but for QE, who gets a single fixed build, it's a bit harder, as it should not normally fail. If you want negative testing, you'll have to do something like:
1. engine-backup --file=f1
2. mkdir f1.d
3. cd f1.d
4. tar xpf ../f1
5. Somehow cause damage to db/engine_backup.db - e.g.:
dd if=/dev/zero of=db/engine_backup.db bs=100k seek=1 count=0
6. tar cvzf ../f2 .
7. cd ..
8. engine-backup --mode=verify --file=f2

Comment 1 Yedidyah Bar David 2020-05-19 08:51:43 UTC
Sorry, found another place calling pg_restore.

Comment 2 RHEL Program Management 2020-05-19 08:51:51 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 3 Sandro Bonazzola 2020-05-21 07:50:43 UTC
$ git tag --contains 0b28b1dead18425574957ce40892947a16f32f4b
ovirt-engine-4.3.10.2
ovirt-engine-4.3.10.3
$ git tag --contains 2dba99c0977a9cc26e10d1ddf3357d799bc70a2b
ovirt-engine-4.3.10.2
ovirt-engine-4.3.10.3

Comment 4 Sandro Bonazzola 2020-05-21 07:58:46 UTC
https://gerrit.ovirt.org/#/c/109138/  was not backported to 4.3, moving out to 4.3.11

Comment 5 Yedidyah Bar David 2020-05-21 08:13:24 UTC
Wrote doc text which applies only for 4.3. In 4.4, also the passing of '-f -' is mandatory. For completeness, and in case we might want to use later PostgreSQL versions also in 4.3, I backported these patches as well.

Comment 7 Guilherme Santos 2020-07-08 13:24:56 UTC
Verified on:
ovirt-engine-4.4.1.7-0.3.el8ev.noarch

Steps:
1. the ones described on description for QE
2. engine-backup --mode=verify --file=f1

Results
It fails for f2, aborting the process and logging the errors: 
# cat /var/log/ovirt-engine-backup/ovirt-engine-verify-20200708152253.log
2020-07-08 15:22:53 36388: Start of engine-backup mode verify scope all file f2
2020-07-08 15:22:53 36388: OUTPUT: Start of engine-backup with mode 'verify'
2020-07-08 15:22:53 36388: OUTPUT: scope: all
2020-07-08 15:22:53 36388: OUTPUT: archive file: f2
2020-07-08 15:22:53 36388: OUTPUT: log file: /var/log/ovirt-engine-backup/ovirt-engine-verify-20200708152253.log
2020-07-08 15:22:53 36388: OUTPUT: Verifying:
2020-07-08 15:22:53 36388: OUTPUT: - Unpacking file 'f2'
2020-07-08 15:22:53 36388: Opening tarball f2 to /tmp/engine-backup.rqRi16I3n2
2020-07-08 15:22:53 36388: Verifying hash
2020-07-08 15:22:53 36388: FATAL: Checksum verification failed

But it succeeds for f1

Comment 8 Sandro Bonazzola 2020-08-05 06:24:50 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 10 Yedidyah Bar David 2020-08-18 06:26:06 UTC
doc text looks good to me. Does it still matter? Bug fixed in 4.4.1, closed.


Note You need to log in before you can comment on or make changes to this bug.