Bug 1879373

Summary:	grafana backup might fail
Product:	[oVirt] ovirt-engine	Reporter:	Yedidyah Bar David <didi>
Component:	Backup-Restore.Engine	Assignee:	Yedidyah Bar David <didi>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Pavel Novotny <pnovotny>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.4.1	CC:	bugs, sbonazzo, sradco
Target Milestone:	ovirt-4.4.3	Keywords:	ZStream
Target Release:	---	Flags:	pm-rhel: ovirt-4.4+ pm-rhel: planning_ack+ sbonazzo: devel_ack+ lleistne: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	engine-backup used to backup grafana's database as a plain file, which failed if it was updated during the backup. Now, it uses sqlite's .backup command for that, which succeeds even during updates, and keeps the database consistent. As part of this change, grafana was split out from '--scope=files' to its own scope, '--scope=grafanadb'. Setups that use engine-backup with specific scopes, should be updated accordingly as needed.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-02 11:34:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yedidyah Bar David 2020-09-16 06:21:23 UTC

Description of problem:

See [1][2]:

2020-09-15 05:23:48 94681: Backing up files to /tmp/engine-backup.B47Gi5uemW/tar/files
tar: var/lib/grafana: file changed as we read it

We should use other means to backup grafana - not using plain tar.

Either using sqlite tools, or grafana-specific tools (if it has any, didn't check).

[1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_basic-suite-master_nightly/390/

[2] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_basic-suite-master_nightly/390/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap_pytest.py/lago-basic-suite-master-engine/_var_log/ost-engine-backup/log.txt

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Seldom

Steps to Reproduce:
1. Setup engine+dwh+grafana
2. engine-backup
3.

Actual results:
If grafana happened to update its database during backup, it will fail as above

Expected results:
engine-backup should always succeed, backing up grafana at the state it was in when engine-backup started

Additional info:

Comment 1 Yedidyah Bar David 2020-09-16 06:28:44 UTC

Workaround: Try again. grafana does not usually update its db often, a next attempt will likely succeed.

Comment 2 Yedidyah Bar David 2020-09-16 12:07:15 UTC

Couldn't find something specific for grafana. For SQLite, something like [1] should do.

This does not seem to be packaged for centos/fedora. Found an old fc23 failed copr build [2].

Perhaps it's enough to use the old lock-copy-unlock approach explained in the beginning of [3].
I think that the 3 shortcomings detailed there are probably irrelevant for us.

[1] https://pypi.org/project/sqlitebck/

[2] https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/build/270131/

[3] https://sqlite.org/backup.html

Comment 3 Yedidyah Bar David 2020-09-16 12:19:10 UTC

OK, seems like simplest is:

sqlite3 grafana.db '.backup backupfile'

Comment 6 Sandro Bonazzola 2020-11-11 06:45:32 UTC

This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.