Description of problem: When attempting to do an incremental backup after taking a snapshot of the VM with memory fails with "invalid argument: missing or broken bitmap 'testchck' for disk 'vda'". Version-Release number of selected component (if applicable): libvirt-6.0.0-25.module+el8.2.1+7154+47ffd890 Steps to Reproduce: 1. Start VM with at least 1 disk. 2. Create a checkpoint (creating a backup at this point is not required to reproduce the bug, but would be normally done) virsh checkpoint-create-as backup-test --description test --name testchck --diskspec vda,checkpoint=bitmap 3. Create a live snapshot with memory: virsh snapshot-create-as backup-test snap3 --memspec snapshot=external,file=/tmp/backup-test-images/mem_s3 --diskspec vda,snapshot=external,file=/tmp/backup-test-images/disk_s3 4. do an incremental backup of 'vda' cat > /tmp/backup-test-images/backup.xml << EOF <domainbackup mode='push'> <incremental>testchck</incremental> <disks> <disk name='vda' type='file'> <driver type='qcow2'/> <target file='/tmp/backup-test-images/backup-vda.qcow2'/> </disk> </disks> </domainbackup> EOF virsh backup-begin backup-test --backupxml /tmp/backup-test-images/backup.xml Actual results: Error is reported: "invalid argument: missing or broken bitmap 'testchck' for disk 'vda' Expected results: Backup succeeds Additional info: Libvirt doesn't propagate the bitmaps correctly as qemu deactivates bitmaps after migration and thus they vanish from output of 'query-named-block-nodes'. We need to collect the results of that command prior to starting the memory migration. Note that this can't be reproduced until qemu fixes https://bugzilla.redhat.com/show_bug.cgi?id=1858739 as the snapshot step will fail.
Fixed upstream: commit 00bb850eb065eca0c65dda212c039bc76774453f Author: Peter Krempa <pkrempa> Date: Fri Jul 31 15:23:56 2020 +0200 qemu: snapshot: Collect 'query-named-block-nodes' prior to memory migration When doing an external snapshot we migrate memory to a file as a form of taking the memory state. This creates a problem as qemu deactivates all active bitmaps after a successful migration. This means that calling 'query-named-block-nodes' will return an empty list of bitmaps for devices. We use the bitmap list to propagate the active bitmaps into the overlay files being created which is required for backups to work after a snapshot. Since we wouldn't propagate anything a subsequent backup will fail with: invalid argument: missing or broken bitmap 'testchck' for disk 'vda' To fix this, we can simply collect the bitmap list prior to the migration. https://bugzilla.redhat.com/show_bug.cgi?id=1862472 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> v6.6.0-2-g00bb850eb0
Peter, I added "Dependent Products: RHV", since it sounds that this bug should affects us. Please correct me if this does not affect RHV. Do you know which version introduced this bug, and when the fix will be available in RHEL/CentOS?
Verified with: libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64 qemu-kvm-5.1.0-7.module+el8.3.0+8099+dba2fe3e.x86_64 Result: PASS [root@dell-per740xd-10 ~]# virsh start vm1 Domain vm1 started [root@dell-per740xd-10 ~]# virsh checkpoint-create-as vm1 --description test --name ck1 --diskspec vda,checkpoint=bitmap Domain checkpoint ck1 created [root@dell-per740xd-10 ~]# virsh checkpoint-list vm1 Name Creation Time ----------------------------------- ck1 2020-09-24 07:44:09 -0400 [root@dell-per740xd-10 ~]# virsh snapshot-create-as vm1 sp1 --memspec snapshot=external,file=/tmp/mem1 --diskspec vda,snapshot=external,file=/tmp/disk1 Domain snapshot sp1 created [root@dell-per740xd-10 ~]# cat bk.xml <domainbackup mode='push'> <incremental>ck1</incremental> <disks> <disk name='vda' type='file'> <driver type='qcow2'/> <target file='/tmp/backup-vda.qcow2'/> </disk> </disks> </domainbackup> [root@dell-per740xd-10 ~]# virsh backup-begin vm1 bk.xml Backup started [root@dell-per740xd-10 ~]# virsh domjobinfo vm1 --completed Job type: Completed Operation: Backup Time elapsed: 112 ms File processed: 38.625 MiB File remaining: 0.000 B File total: 38.625 MiB
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137