Bug 1819755

Summary: [incremental backup] Incremental backup fails after redefinition of checkpoints
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Peter Krempa <pkrempa>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: yisun
Severity: high Docs Contact:
Priority: high    
Version: 8.4CC: chhu, jdenemar, jsuchane, lmen, mtessun, nsoffer, pelauter, toneata, virt-maint, xuzhang, ymankad
Target Milestone: rcKeywords: Triaged
Target Release: 8.2Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-6.0.0-17.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1821593 (view as bug list) Environment:
Last Closed: 2020-05-05 09:59:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1821593    
Attachments:
Description Flags
reproducer script none

Description Peter Krempa 2020-04-01 13:25:03 UTC
Created attachment 1675408 [details]
reproducer script

Description of problem:
Incremental backup fails with 'error: Requested operation is not valid: could not locate checkpoint 'testchck2' for incremental backup' when the checkpoints were created by redefinition from stored XMLs.

Version-Release number of selected component (if applicable):
6.0.0-17.el8 and upstream 

How reproducible:
100%

Steps to Reproduce:
1. Create some checkpoints 
2. Dump the XMLs
3. undefine the domain
4. define the domain again
5. define the checkpoints via virsh checkpoint-create --redefine
6. attempt an incremental backup

(See attached script for exact reproduction steps)

Actual results:
error: Requested operation is not valid: could not locate checkpoint 'testchck2' for incremental backup'

Expected results:
backup succeeds

Additional info:
Note that the script expects a VM definition stored as backup-test.xml of a domain named 'backup-test' which has no disks defined. The script creates data in /tmp/

Comment 1 Peter Krempa 2020-04-01 13:26:13 UTC
Patches posted upstream:

https://www.redhat.com/archives/libvir-list/2020-April/msg00013.html

Comment 7 Martin Tessun 2020-04-08 12:51:32 UTC
*** Bug 1821593 has been marked as a duplicate of this bug. ***

Comment 12 yisun 2020-04-14 09:07:14 UTC
Thanks Peter for the reproduce script. 
Test result is PASS with libvirt-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
Steps as follow:

[root@dell-per740xd-11 bz1819755]# sh test.sh
+ echo prepare/cleanup domain
prepare/cleanup domain
+ virsh destroy backup-test
error: failed to get domain 'backup-test'

+ virsh undefine backup-test --snapshots-metadata --checkpoints-metadata
error: failed to get domain 'backup-test'

+ virsh define backup-test.xml
Domain backup-test defined from backup-test.xml

+ rm -rf /tmp/backup-test-images
+ mkdir /tmp/backup-test-images
+ echo prepare storage
prepare storage
+ qemu-img create -f qcow2 /tmp/backup-test-images/backup-disk-base.qcow2 100M
Formatting '/tmp/backup-test-images/backup-disk-base.qcow2', fmt=qcow2 size=104857600 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ qemu-io -c 'write -P 0xaa 1M 1M' -f qcow2 /tmp/backup-test-images/backup-disk-base.qcow2
wrote 1048576/1048576 bytes at offset 1048576
1 MiB, 1 ops; 00.00 sec (357.416 MiB/sec and 357.4155 ops/sec)
+ qemu-img create -f qcow2 -F qcow2 -b /tmp/backup-test-images/backup-disk-base.qcow2 /tmp/backup-test-images/backup-disk-top.qcow2
Formatting '/tmp/backup-test-images/backup-disk-top.qcow2', fmt=qcow2 size=104857600 backing_file=/tmp/backup-test-images/backup-disk-base.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ qemu-io -c 'write -P 0xbb 3M 1M' -f qcow2 /tmp/backup-test-images/backup-disk-top.qcow2
wrote 1048576/1048576 bytes at offset 3145728
1 MiB, 1 ops; 00.00 sec (348.889 MiB/sec and 348.8888 ops/sec)
+ echo add disk, start vm
add disk, start vm
+ virsh attach-disk backup-test /tmp/backup-test-images/backup-disk-top.qcow2 vdb --subdriver qcow2 --config
Disk attached successfully

+ virsh start backup-test --paused
Domain backup-test started

+ echo create checkpoints and dirty bytes
create checkpoints and dirty bytes
+ virsh checkpoint-create-as backup-test --description test --name testchck1 --diskspec vda,checkpoint=no --diskspec vdb,checkpoint=bitmap
Domain checkpoint testchck1 created
+ virsh destroy backup-test
Domain backup-test destroyed

+ qemu-io -c 'write -P 0xcc 4M 1M' -f qcow2 /tmp/backup-test-images/backup-disk-top.qcow2
wrote 1048576/1048576 bytes at offset 4194304
1 MiB, 1 ops; 00.00 sec (483.616 MiB/sec and 483.6156 ops/sec)
+ virsh start backup-test --paused
Domain backup-test started

+ virsh checkpoint-create-as backup-test --description test --name testchck2 --diskspec vda,checkpoint=no --diskspec vdb,checkpoint=bitmap
Domain checkpoint testchck2 created
+ virsh destroy backup-test
Domain backup-test destroyed

+ qemu-io -c 'write -P 0xdd 5M 1M' -f qcow2 /tmp/backup-test-images/backup-disk-top.qcow2
wrote 1048576/1048576 bytes at offset 5242880
1 MiB, 1 ops; 00.00 sec (510.459 MiB/sec and 510.4591 ops/sec)
+ echo dump checkpoint xml
dump checkpoint xml
+ virsh checkpoint-dumpxml backup-test testchck1
+ virsh checkpoint-dumpxml backup-test testchck2
+ echo redefine domain
redefine domain
+ virsh undefine backup-test --snapshots-metadata --checkpoints-metadata
Domain backup-test has been undefined

+ virsh define backup-test.xml
Domain backup-test defined from backup-test.xml

+ virsh attach-disk backup-test /tmp/backup-test-images/backup-disk-top.qcow2 vdb --subdriver qcow2 --config
Disk attached successfully

+ echo redefine checkpoints
redefine checkpoints
+ virsh start backup-test --paused
Domain backup-test started

+ virsh checkpoint-create --redefine backup-test /tmp/backup-test-images/checkpoint1.xml
Domain checkpoint testchck1 created from '/tmp/backup-test-images/checkpoint1.xml'
+ virsh checkpoint-create --redefine backup-test /tmp/backup-test-images/checkpoint2.xml
Domain checkpoint testchck2 created from '/tmp/backup-test-images/checkpoint2.xml'
+ echo start backup job
start backup job
+ cat
+ virsh backup-begin backup-test --backupxml /tmp/backup-test-images/backup.xml
Backup started

+ echo copy data
copy data
+ qemu-img convert -f raw -O qcow2 nbd://localhost:1234/vdb /tmp/backup-test-images/backup-output.qcow2
+ echo terminate job
terminate job
+ virsh domjobabort backup-test

+ virsh destroy backup-test
Domain backup-test destroyed

+ echo compare output
compare output
+ qemu-img compare -f qcow2 -F qcow2 /tmp/backup-test-images/backup-disk-top.qcow2 /tmp/backup-test-images/backup-output.qcow2
Images are identical.

Comment 14 errata-xmlrpc 2020-05-05 09:59:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017