Bug 1862472 - incremental backup fails after an external snapshot with memory
Summary: incremental backup fails after an external snapshot with memory
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.3
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
Depends On: 1858739
Blocks: 1799015 1866329
TreeView+ depends on / blocked
 
Reported: 2020-07-31 14:00 UTC by Peter Krempa
Modified: 2020-11-17 17:50 UTC (History)
5 users (show)

Fixed In Version: libvirt-6.6.0-5.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-17 17:50:26 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Peter Krempa 2020-07-31 14:00:57 UTC
Description of problem:
When attempting to do an incremental backup after taking a snapshot of the VM with memory fails with "invalid argument: missing or broken bitmap 'testchck' for disk 'vda'".

Version-Release number of selected component (if applicable):
libvirt-6.0.0-25.module+el8.2.1+7154+47ffd890

Steps to Reproduce:
1. Start VM with at least 1 disk.

2. Create a checkpoint (creating a backup at this point is not required to reproduce the bug, but would be normally done)

virsh checkpoint-create-as backup-test --description test --name testchck --diskspec vda,checkpoint=bitmap

3. Create a live snapshot with memory:

virsh snapshot-create-as backup-test snap3 --memspec snapshot=external,file=/tmp/backup-test-images/mem_s3 --diskspec vda,snapshot=external,file=/tmp/backup-test-images/disk_s3 

4. do an incremental backup of 'vda'
             
cat > /tmp/backup-test-images/backup.xml << EOF
<domainbackup mode='push'>
  <incremental>testchck</incremental>
  <disks>
    <disk name='vda' type='file'>
      <driver type='qcow2'/>
      <target file='/tmp/backup-test-images/backup-vda.qcow2'/>
    </disk>
  </disks>
</domainbackup>
EOF
virsh backup-begin backup-test --backupxml /tmp/backup-test-images/backup.xml

Actual results:
Error is reported: "invalid argument: missing or broken bitmap 'testchck' for disk 'vda'

Expected results:
Backup succeeds

Additional info:
Libvirt doesn't propagate the bitmaps correctly as qemu deactivates bitmaps after migration and thus they vanish from output of 'query-named-block-nodes'. We need to collect the results of that command prior to starting the memory migration.

Note that this can't be reproduced until qemu fixes https://bugzilla.redhat.com/show_bug.cgi?id=1858739 as the snapshot step will fail.

Comment 1 Peter Krempa 2020-08-03 05:20:27 UTC
Fixed upstream:

commit 00bb850eb065eca0c65dda212c039bc76774453f
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Fri Jul 31 15:23:56 2020 +0200

    qemu: snapshot: Collect 'query-named-block-nodes' prior to memory migration
    
    When doing an external snapshot we migrate memory to a file as a form of
    taking the memory state. This creates a problem as qemu deactivates all
    active bitmaps after a successful migration. This means that calling
    'query-named-block-nodes' will return an empty list of bitmaps for
    devices. We use the bitmap list to propagate the active bitmaps into the
    overlay files being created which is required for backups to work after
    a snapshot. Since we wouldn't propagate anything a subsequent backup
    will fail with:
    
    invalid argument: missing or broken bitmap 'testchck' for disk 'vda'
    
    To fix this, we can simply collect the bitmap list prior to the
    migration.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1862472
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

v6.6.0-2-g00bb850eb0

Comment 2 Nir Soffer 2020-08-19 16:50:39 UTC
Peter, I added "Dependent Products: RHV", since it sounds that this bug
should affects us. Please correct me if this does not affect RHV.

Do you know which version introduced this bug, and when the fix will
be available in RHEL/CentOS?

Comment 8 yisun 2020-09-24 11:47:23 UTC
Verified with:
libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64
qemu-kvm-5.1.0-7.module+el8.3.0+8099+dba2fe3e.x86_64

Result: PASS

[root@dell-per740xd-10 ~]# virsh start vm1
Domain vm1 started

[root@dell-per740xd-10 ~]# virsh checkpoint-create-as vm1 --description test --name ck1 --diskspec vda,checkpoint=bitmap
Domain checkpoint ck1 created

[root@dell-per740xd-10 ~]# virsh checkpoint-list vm1
 Name   Creation Time
-----------------------------------
 ck1    2020-09-24 07:44:09 -0400


[root@dell-per740xd-10 ~]# virsh snapshot-create-as vm1 sp1 --memspec snapshot=external,file=/tmp/mem1 --diskspec vda,snapshot=external,file=/tmp/disk1 
Domain snapshot sp1 created

[root@dell-per740xd-10 ~]# cat bk.xml 
<domainbackup mode='push'>
  <incremental>ck1</incremental>
  <disks>
    <disk name='vda' type='file'>
      <driver type='qcow2'/>
      <target file='/tmp/backup-vda.qcow2'/>
    </disk>
  </disks>
</domainbackup>

[root@dell-per740xd-10 ~]# virsh backup-begin vm1 bk.xml
Backup started

[root@dell-per740xd-10 ~]# virsh domjobinfo vm1 --completed
Job type:         Completed   
Operation:        Backup      
Time elapsed:     112          ms
File processed:   38.625 MiB
File remaining:   0.000 B
File total:       38.625 MiB

Comment 11 errata-xmlrpc 2020-11-17 17:50:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137


Note You need to log in before you can comment on or make changes to this bug.