Bug 1879373 - grafana backup might fail
Summary: grafana backup might fail
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Backup-Restore.Engine
Version: 4.4.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.4.3
: ---
Assignee: Yedidyah Bar David
QA Contact: Pavel Novotny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-16 06:21 UTC by Yedidyah Bar David
Modified: 2021-04-14 06:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
engine-backup used to backup grafana's database as a plain file, which failed if it was updated during the backup. Now, it uses sqlite's .backup command for that, which succeeds even during updates, and keeps the database consistent. As part of this change, grafana was split out from '--scope=files' to its own scope, '--scope=grafanadb'. Setups that use engine-backup with specific scopes, should be updated accordingly as needed.
Clone Of:
Environment:
Last Closed: 2020-11-02 11:34:19 UTC
oVirt Team: Integration
Embargoed:
pm-rhel: ovirt-4.4+
pm-rhel: planning_ack+
sbonazzo: devel_ack+
lleistne: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 111336 0 master MERGED packaging: backup: Backup grafana using sqlite3 2020-10-13 06:47:03 UTC
oVirt gerrit 111401 0 master MERGED packaging: spec: Require sqlite 2020-10-13 06:47:13 UTC
oVirt gerrit 114268 0 master MERGED packaging: backup: Fix grafana backup 2021-04-18 05:49:30 UTC

Description Yedidyah Bar David 2020-09-16 06:21:23 UTC
Description of problem:

See [1][2]:

2020-09-15 05:23:48 94681: Backing up files to /tmp/engine-backup.B47Gi5uemW/tar/files
tar: var/lib/grafana: file changed as we read it

We should use other means to backup grafana - not using plain tar.

Either using sqlite tools, or grafana-specific tools (if it has any, didn't check).

[1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_basic-suite-master_nightly/390/

[2] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_basic-suite-master_nightly/390/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap_pytest.py/lago-basic-suite-master-engine/_var_log/ost-engine-backup/log.txt

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Seldom

Steps to Reproduce:
1. Setup engine+dwh+grafana
2. engine-backup
3.

Actual results:
If grafana happened to update its database during backup, it will fail as above

Expected results:
engine-backup should always succeed, backing up grafana at the state it was in when engine-backup started

Additional info:

Comment 1 Yedidyah Bar David 2020-09-16 06:28:44 UTC
Workaround: Try again. grafana does not usually update its db often, a next attempt will likely succeed.

Comment 2 Yedidyah Bar David 2020-09-16 12:07:15 UTC
Couldn't find something specific for grafana. For SQLite, something like [1] should do.

This does not seem to be packaged for centos/fedora. Found an old fc23 failed copr build [2].

Perhaps it's enough to use the old lock-copy-unlock approach explained in the beginning of [3].
I think that the 3 shortcomings detailed there are probably irrelevant for us.

[1] https://pypi.org/project/sqlitebck/

[2] https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/build/270131/

[3] https://sqlite.org/backup.html

Comment 3 Yedidyah Bar David 2020-09-16 12:19:10 UTC
OK, seems like simplest is:

sqlite3 grafana.db '.backup backupfile'

Comment 6 Sandro Bonazzola 2020-11-11 06:45:32 UTC
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.