Bug 1832905

Summary:	engine-backup --mode=verify is broken
Product:	[oVirt] ovirt-engine	Reporter:	Yedidyah Bar David <didi>
Component:	Backup-Restore.Engine	Assignee:	Yedidyah Bar David <didi>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Guilherme Santos <gdeolive>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.3.0	CC:	bugs, lleistne, mtessun, rdlugyhe
Target Milestone:	ovirt-4.4.1	Keywords:	ZStream
Target Release:	4.4.1.4	Flags:	pm-rhel: ovirt-4.4+ sbonazzo: blocker? mtessun: planning_ack+ sbonazzo: devel_ack+ lleistne: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovirt-engine-4.4.1.4	Doc Type:	Bug Fix
Doc Text:	In previous versions, `engine-backup --mode=verify` passed even if `pg_restore` emitted errors. The current release fixes this issue. The `engine-backup --mode=verify` command correctly fails if `pg_restore` emits errors.	Story Points:	---
Clone Of:
Clones:	1848877 1868929 (view as bug list)		Environment:
Last Closed:	2020-08-05 06:24:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1848877, 1868929

Description Yedidyah Bar David 2020-05-07 13:05:04 UTC

Description of problem:

engine-backup --mode=verify:

1. Does not pass '-f -' to pg_restore, which makes it fail, because in recent versions it does not default to stdout anymore.

2. Despite (1.), engine-backup does not fail or log this.

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Always

Steps to Reproduce:
1. engine-backup --file=f1
2. engine-backup --mode=verify --file=f1
3.

Actual results:
Passes, although pg_restore failed

Expected results:
Should pass, and if pg_restore still fails should abort and log the errors

Additional info:

I already pushed two separate patches for this to gerrit, then decided to also open this bug, and consider backporting to 4.3, mainly for (1.) - so that if pg_restore fails we notice.

Verifying this with each patch separately was easy, but for QE, who gets a single fixed build, it's a bit harder, as it should not normally fail. If you want negative testing, you'll have to do something like:
1. engine-backup --file=f1
2. mkdir f1.d
3. cd f1.d
4. tar xpf ../f1
5. Somehow cause damage to db/engine_backup.db - e.g.:
dd if=/dev/zero of=db/engine_backup.db bs=100k seek=1 count=0
6. tar cvzf ../f2 .
7. cd ..
8. engine-backup --mode=verify --file=f2

Comment 1 Yedidyah Bar David 2020-05-19 08:51:43 UTC

Sorry, found another place calling pg_restore.

Comment 2 RHEL Program Management 2020-05-19 08:51:51 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 3 Sandro Bonazzola 2020-05-21 07:50:43 UTC

$ git tag --contains 0b28b1dead18425574957ce40892947a16f32f4b
ovirt-engine-4.3.10.2
ovirt-engine-4.3.10.3
$ git tag --contains 2dba99c0977a9cc26e10d1ddf3357d799bc70a2b
ovirt-engine-4.3.10.2
ovirt-engine-4.3.10.3

Comment 4 Sandro Bonazzola 2020-05-21 07:58:46 UTC

https://gerrit.ovirt.org/#/c/109138/  was not backported to 4.3, moving out to 4.3.11

Comment 5 Yedidyah Bar David 2020-05-21 08:13:24 UTC

Wrote doc text which applies only for 4.3. In 4.4, also the passing of '-f -' is mandatory. For completeness, and in case we might want to use later PostgreSQL versions also in 4.3, I backported these patches as well.

Comment 7 Guilherme Santos 2020-07-08 13:24:56 UTC

Verified on:
ovirt-engine-4.4.1.7-0.3.el8ev.noarch

Steps:
1. the ones described on description for QE
2. engine-backup --mode=verify --file=f1

Results
It fails for f2, aborting the process and logging the errors: 
# cat /var/log/ovirt-engine-backup/ovirt-engine-verify-20200708152253.log
2020-07-08 15:22:53 36388: Start of engine-backup mode verify scope all file f2
2020-07-08 15:22:53 36388: OUTPUT: Start of engine-backup with mode 'verify'
2020-07-08 15:22:53 36388: OUTPUT: scope: all
2020-07-08 15:22:53 36388: OUTPUT: archive file: f2
2020-07-08 15:22:53 36388: OUTPUT: log file: /var/log/ovirt-engine-backup/ovirt-engine-verify-20200708152253.log
2020-07-08 15:22:53 36388: OUTPUT: Verifying:
2020-07-08 15:22:53 36388: OUTPUT: - Unpacking file 'f2'
2020-07-08 15:22:53 36388: Opening tarball f2 to /tmp/engine-backup.rqRi16I3n2
2020-07-08 15:22:53 36388: Verifying hash
2020-07-08 15:22:53 36388: FATAL: Checksum verification failed

But it succeeds for f1

Comment 8 Sandro Bonazzola 2020-08-05 06:24:50 UTC

This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 10 Yedidyah Bar David 2020-08-18 06:26:06 UTC

doc text looks good to me. Does it still matter? Bug fixed in 4.4.1, closed.