Bug 1217820 (Engine_Change_Block_Tracking) - [RFE] API for changed blocks/sectors for a disk for incremental backup usage
Summary: [RFE] API for changed blocks/sectors for a disk for incremental backup usage
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: Engine_Change_Block_Tracking
Product: ovirt-engine
Classification: oVirt
Component: RFEs
Version: ---
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.4.1
: 4.4.1
Assignee: Daniel Erez
QA Contact: Ilan Zuckerman
URL:
Whiteboard:
: 1608452 (view as bug list)
Depends On: 1207657 1207659 1518988 1518989 1589508 1658343 1734975 1734976
Blocks: 1139877 1547560 1573335 1609341 1647195 1691340 1694070
TreeView+ depends on / blocked
 
Reported: 2015-05-01 19:10 UTC by Ademar Reis
Modified: 2021-08-23 10:30 UTC (History)
25 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of: 1207659
Environment:
Last Closed: 2020-07-08 08:27:10 UTC
oVirt Team: Storage
Embargoed:
pm-rhel: ovirt-4.4+
aefrat: testing_plan_complete+
ylavi: planning_ack+
pm-rhel: devel_ack+
pm-rhel: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-43087 0 None None None 2021-08-23 10:30:02 UTC
oVirt gerrit 99952 0 'None' MERGED core: CBT restore flow support 2020-09-30 08:26:41 UTC
oVirt gerrit 99953 0 'None' MERGED core: disk transfer - pass backup_url to ticket 2020-09-30 08:26:40 UTC
oVirt gerrit 99954 0 'None' MERGED core: fix ticket transfer size 2020-09-30 08:26:40 UTC
oVirt gerrit 99955 0 'None' MERGED core: disk transfer - specified backupId implies NBD 2020-09-30 08:26:40 UTC
oVirt gerrit 99956 0 'None' MERGED core: disk transfer - validate format for VM backup 2020-09-30 08:26:40 UTC
oVirt gerrit 99957 0 'None' MERGED core: start backup - handle failure on redefine checkpoints 2020-09-30 08:26:40 UTC

Description Ademar Reis 2015-05-01 19:10:55 UTC
More recent references:

http://comments.gmane.org/gmane.comp.emulators.qemu/303834
https://github.com/qemu/qemu/blob/master/docs/bitmaps.md

(notice the libvirt API/design is still a work in progress)


+++ This bug was initially created as a clone of Bug #1207659 +++
+++ This bug was initially created as a clone of Bug #1207657 +++

QEMU is getting incremental live backup, where a dirty block bitmap is available to third-party applications so they can read only blocks that changed since the previous backup.

Upstream references:
http://lists.gnu.org/archive/html/qemu-devel/2015-03/msg04501.html
http://lists.gnu.org/archive/html/qemu-devel/2015-03/msg05796.html

Comment 1 Michal Skrivanek 2015-05-04 08:19:24 UTC
if I'm not mistaken this is http://wiki.qemu.org/Features/Livebackup
flagging storage

Comment 2 Ademar Reis 2015-05-05 13:06:11 UTC
(In reply to Michal Skrivanek from comment #1)
> if I'm not mistaken this is http://wiki.qemu.org/Features/Livebackup

Unfortunately not. The wiki above is very outdated, from a previous try at live-backup. Updating the wiki is in our to-do list.

> flagging storage

storage indeed, thanks.

Comment 3 Ademar Reis 2015-05-11 18:25:32 UTC
(In reply to Ademar Reis from comment #2)
> (In reply to Michal Skrivanek from comment #1)
> > if I'm not mistaken this is http://wiki.qemu.org/Features/Livebackup
> 
> Unfortunately not. The wiki above is very outdated, from a previous try at
> live-backup. Updating the wiki is in our to-do list.
> 

http://wiki.qemu.org/Features/IncrementalBackup

(thanks to John Snow)

Comment 5 Yaniv Lavi 2018-08-05 15:38:19 UTC
*** Bug 1608452 has been marked as a duplicate of this bug. ***

Comment 6 Sandro Bonazzola 2019-01-28 09:44:11 UTC
This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

Comment 8 Avihai 2019-08-12 12:59:08 UTC
Hi Daniel,

In order to qa_ack this but I need to know if we(QE) have the capacity meaning I need to know:

1) The basic scenario/s (go/ no go flows) of how to test this feature.

2) Do we have a build (downstream 4.3/4.4 complete build not integration mode) to test the basic scenario on?

3) Are all API's (libvirt) available?  
I see this bug is dependant on bug 1734975 and bug 1734976 which are in NEW state.

Comment 9 Avihai 2020-04-21 05:57:41 UTC
(In reply to Avihai from comment #8)
> Hi Daniel,
> 
> In order to qa_ack this but I need to know if we(QE) have the capacity
> meaning I need to know:
> 
> 1) The basic scenario/s (go/ no go flows) of how to test this feature.
> 
> 2) Do we have a build (downstream 4.3/4.4 complete build not integration
> mode) to test the basic scenario on?
> 
> 3) Are all API's (libvirt) available?  
> I see this bug is dependant on bug 1734975 and bug 1734976 which are in NEW
> state.
I see all dependant bugs are verified so disregard this question.
Questions 1) and 2) are still relevant for 4.4.

As the responsibility for this feature changed to Eyal, I'm changing NEEDINFO accordingly.
Please also change the assignee for this bug.

Comment 10 Eyal Shenitzky 2020-04-27 08:18:27 UTC
(In reply to Avihai from comment #8)
> Hi Daniel,
> 
> In order to qa_ack this but I need to know if we(QE) have the capacity
> meaning I need to know:
> 
> 1) The basic scenario/s (go/ no go flows) of how to test this feature.

Please use - https://ovirt.org/develop/release-management/features/storage/incremental-backup.html

> 
> 2) Do we have a build (downstream 4.3/4.4 complete build not integration
> mode) to test the basic scenario on?

Yes, already available in the latest beta of 4.4

Comment 11 Sandro Bonazzola 2020-05-18 14:46:48 UTC
Moved to 4.4.1 not being marked as blocker for 4.4.0 and we are preparing to GA.

Comment 12 Ilan Zuckerman 2020-06-28 07:43:12 UTC
> Incremental backup flow:
> 
> 1. The customer has a VM with disks and wants to create a backup for it.
> 2. The customer can now create a full backup for the VM disks using a new
> backup API while the VM is running.
> 3. When the backup is done, the customer can download the backup disks using
> a new API provided by imageIO.
> 4. The customer continue to work with the VM and new data is written to the
> disk.
> 5. The customer need to perform a new backup that includes the new data.
> 6. The customer can now create an incremental backup for the VM disks using
> a new backup API while the VM is running, 
>    this backup will include only the blocks that changed on the disks.
> 8. When the backup is done, the customer can download the incremental backup
> disks using a new API provided by imageIO, 
>    the downloaded disks will include only the data that changed in the time
> between the first
>    full backup until the beginning of the incremental backup.


This case is covered by the following TC's:

TestCase27284 This is sanity test for Restoring from full backup and checking the content
TestCase27165 This is sanity test for Restoring from incremental backup and checking the content.

Both case (and others which were tested as well) are described in the 'incremental backup test plan" here:
https://polarion.engineering.redhat.com/polarion/#/project/RHEVM3/wiki/Storage_4_0/4_4_Storage_Incremental_Backup

Verified on rhv-release-4.4.1-3-001.noarch

Comment 13 Sandro Bonazzola 2020-07-08 08:27:10 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.