Bug 1867085

Implemented upstream by:

9f61d14a9e qemu: migration: Migrate block dirty bitmaps corresponding to checkpoints
1475baed36 qemu: migration: Clean up temporary bitmaps when cancelling a migration
f7eb3e233d tests: qemumigrationcookie: Add testing for block dirty bitmap migration
add6ee37ee tests: qemustatusxml2xml: Add status XML from migration with bitmaps
516906d681 qemu: domain: Store list of temporary bitmaps for migration in status XML
49957f5f69 qemu: migration_cookie: Add helpers for transforming the cookie into migration params
f5cd60d478 qemu: migration_cookie: Add XML handling for setting up bitmap migration
b0104664c6 qemu: migration_params: Add infrastructure for 'dirty-bitmaps' migration feature
b44e3ca306 qemu: blockjob: Use qemuMonitorBitmapRemove for single bitmap removal
7a8d7d75df qemu: monitor: Introduce qemuMonitorBitmapRemove
8da9c7456d qemu: migration: Create qcow2 v3 images for VIR_MIGRATE_NON_SHARED_DISK
253bce93d3 qemu: Probe whether an image is 'qcow2 v2' from query-named-block-nodes
9e855e7589 qemu: capabilities: Introduce QEMU_CAPS_MIGRATION_PARAM_BLOCK_BITMAP_MAPPING

Verified with libvirt-7.0.0-9.module+el8.4.0+10326+5e50a3b6.x86_64
===================================================================================================
Positive test
1. prepare 2 qcow2 image and use them in vm
[root@dell-per740-01 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/vdb.qcow2 100M
Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=104857600 lazy_refcounts=off refcount_bits=16
[root@dell-per740-01 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/vdc.qcow2 100M
Formatting '/var/lib/libvirt/images/vdc.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=104857600 lazy_refcounts=off refcount_bits=16


[root@dell-per740-01 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2
 vdb      /var/lib/libvirt/images/vdb.qcow2
 vdc      /var/lib/libvirt/images/vdc.qcow2

[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started

2. create checkpoint 'ck1' for vdb and vdc
[root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test --name ck1 --diskspec vda,checkpoint=no vdb,checkpoint=bitmap vdc,checkpoint=bitmap
Domain checkpoint ck1 created

3. create 1M data in vm to vdb and vdc
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210274 s, 49.9 MB/s
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdc bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214923 s, 48.8 MB/s
[root@localhost ~]# sync


4. create checkpoint 'ck2' for vdb
[root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test --name ck2 --diskspec vda,checkpoint=no vdb,checkpoint=bitmap vdc,checkpoint=no
Domain checkpoint ck2 created

5. create 1M data in vm to vdb and vdc with skip=2
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=1 skip=2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018437 s, 56.9 MB/s
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdc bs=1M count=1 skip=2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189918 s, 55.2 MB/s
[root@localhost ~]# sync


6. create checkpoint 'ck3' for vdc
[root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test --name ck3 --diskspec vda,checkpoint=no vdb,checkpoint=no vdc,checkpoint=bitmap
Domain checkpoint ck3 created


7. create 1M data to in vm to vdb and vdc with skip=4
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=1 skip=4
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184957 s, 56.7 MB/s
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdc bs=1M count=1 skip=4
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195885 s, 53.5 MB/s
[root@localhost ~]# sync

8. check vdb and vdc's images' bitmap hash value, as follow:
[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck1"}}'
{"id":"libvirt-388","error":{"class":"GenericError","desc":"Dirty bitmap 'ck1' not found"}}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck2"}}'
{"id":"libvirt-389","error":{"class":"GenericError","desc":"Dirty bitmap 'ck2' not found"}}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck3"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-390"}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck1"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-391"}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck2"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-392"}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck3"}}'
{"id":"libvirt-393","error":{"class":"GenericError","desc":"Dirty bitmap 'ck3' not found"}}

9. migrate vm to dest host
[root@dell-per740-01 ~]# virsh migrate vm1 --live qemu+ssh://lenovo-sr630-10.lab.eng.pek2.redhat.com/system --verbose --copy-storage-all
root.eng.pek2.redhat.com's password: 
Migration: [100 %]

10. check the bitmaps' hash value in dest host
[root@lenovo-sr630-10 images]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck1"}}'
{"id":"libvirt-398","error":{"class":"GenericError","desc":"Dirty bitmap 'ck1' not found"}}

[root@lenovo-sr630-10 images]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck2"}}'
{"id":"libvirt-399","error":{"class":"GenericError","desc":"Dirty bitmap 'ck2' not found"}}

[root@lenovo-sr630-10 images]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck3"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-400"}

[root@lenovo-sr630-10 images]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck1"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-401"}

[root@lenovo-sr630-10 images]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck2"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-402"}

[root@lenovo-sr630-10 images]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck3"}}'
{"id":"libvirt-403","error":{"class":"GenericError","desc":"Dirty bitmap 'ck3' not found"}}



================================================================
Negative test:
Abort the migration procedure and check if bitmaps in original image have impact
1. create a checkpoint 'ck1' for vm's vdb and vdc
[root@dell-per740-01 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2
 vdb      /var/lib/libvirt/images/vdb.qcow2
 vdc      /var/lib/libvirt/images/vdc.qcow2


[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started

[root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test --name ck1 --diskspec vda,checkpoint=no vdb,checkpoint=bitmap vdc,checkpoint=bitmap
Domain checkpoint ck1 created

2. generate some data to vdb and vdc in vm
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174449 s, 60.1 MB/s
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdc bs=1M count=2
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0286931 s, 73.1 MB/s
[root@localhost ~]# sync

3. record the hash value of 'ck1'
[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck1"}}'
{"return":{"sha256":"6c5c25c6989f62f371e512a110c82491b42cb01d0d19bad80ed8f9ea31976ef8"},"id":"libvirt-386"}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck1"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-387"}

4. do migration, but <ctrl-c> to cancel it during the process
[root@dell-per740-01 ~]# virsh migrate vm1 --live qemu+ssh://lenovo-sr630-10.lab.eng.pek2.redhat.com/system --verbose --copy-storage-all
root.eng.pek2.redhat.com's password: 
Migration: [  3 %]^C^C^C^C^C^C^C^C^C^C^Cerror: operation aborted: migration out: canceled by client

5. check the hash value of the bitmaps, they should keep same as step 3
[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck1"}}'
{"return":{"sha256":"6c5c25c6989f62f371e512a110c82491b42cb01d0d19bad80ed8f9ea31976ef8"},"id":"libvirt-428"}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck1"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-429"}


6. do migration again, this time, stop the libvirtd service on the dest host
[root@dell-per740-01 ~]# virsh migrate vm1 --live qemu+ssh://lenovo-sr630-10.lab.eng.pek2.redhat.com/system --verbose --copy-storage-all
root.eng.pek2.redhat.com's password: 
Migration: [ 23 %]error: operation failed: migration of disk vda failed: Input/output error


[root@lenovo-sr630-10 images]# systemctl stop libvirtd
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd-admin.socket
  libvirtd.socket
  libvirtd-ro.socket

7. when migration failed, check the bitmaps hash value on source vm, they should keep same as step 3
[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-1-format","name":"ck1"}}'
{"return":{"sha256":"6c5c25c6989f62f371e512a110c82491b42cb01d0d19bad80ed8f9ea31976ef8"},"id":"libvirt-477"}

[root@dell-per740-01 ~]# virsh qemu-monitor-command vm1 --cmd '{"execute": "x-debug-block-dirty-bitmap-sha256","arguments": {"node":"libvirt-2-format","name":"ck1"}}'
{"return":{"sha256":"34f3c20909ea13cc63353cd4614c43f9069a984ce024a9128da3ebfdaada67fd"},"id":"libvirt-478"}

Hi Peter,
Here are 2 questions during the test, pls help to confirm if they worked as design

Q1: If the dest host has a exisiting image, and it contains existing bitmaps, the migration won't erase it. This may cause inconsistence between source and dest host.
1. scp a image with dirty bitmap to dest host，this is to make sure iamge on dest host has dirty bitmap 'ck1'
[root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test --name ck1 --diskspec vda,checkpoint=bitmap
Domain checkpoint ck1 created

[root@dell-per740-01 ~]# virsh destroy vm1
Domain 'vm1' destroyed

[root@dell-per740-01 ~]# scp /var/lib/libvirt/images/jeos-27-x86_64.qcow2 root.eng.pek2.redhat.com:/var/lib/libvirt/images/jeos-27-x86_64.qcow2
Warning: Permanently added 'lenovo-sr630-10.lab.eng.pek2.redhat.com' (ECDSA) to the list of known hosts.
root.eng.pek2.redhat.com's password: 
jeos-27-x86_64.qcow2                                                                                                                                                                                                                                                                        100%  931MB 109.5MB/s   00:08    
<=== now the image on dest host should have a bitmap named 'ck1'

2. delete the dirty bitmap of the image on source host
[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started

[root@dell-per740-01 ~]# virsh checkpoint-delete vm1 ck1
Domain checkpoint ck1 deleted
<=== this is to make sure the image on source host has no bitmap now.

3. do migration
[root@dell-per740-01 ~]# virsh migrate vm1 --live qemu+ssh://xxxxxx.redhat.com/system --verbose --copy-storage-all
root.eng.pek2.redhat.com's password: 
Migration: [100 %]

4. on dest host, check the image info
[root@lenovo-sr630-10 images]# virsh destroy vm1
Domain 'vm1' destroyed

[root@lenovo-sr630-10 images]# qemu-img info /var/lib/libvirt/images/jeos-27-x86_64.qcow2
image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1.75 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: ck1
            granularity: 65536
    refcount bits: 16
    corrupt: false
    extended l2: false
<=== here we can see the image still has bitmap 'ck1', this is inconsistent with source host image's current status.



Q2: When migration finished, the bitmap in the source image was removed.
1. start a vm with image has no bitmaps
[root@dell-per740-01 ~]# virsh domblklist vm1
 Target   Source
--------------------------------------------------------
 vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2

[root@dell-per740-01 ~]# qemu-img info /var/lib/libvirt/images/jeos-27-x86_64.qcow2
image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started

2. create a checkpoint for vm's vda
[root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test --name ck1 --diskspec vda,checkpoint=bitmap
Domain checkpoint ck1 created

3. do migration
[root@dell-per740-01 ~]# virsh migrate vm1 --live qemu+ssh://lenovo-sr630-10.lab.eng.pek2.redhat.com/system --verbose --copy-storage-all
root.eng.pek2.redhat.com's password: 
Migration: [100 %]

4. after migration, check the image's info on source host, the bitmap not saved in that image
[root@dell-per740-01 ~]# qemu-img info /var/lib/libvirt/images/jeos-27-x86_64.qcow2
image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

5. if we use this vm on source host again, the checkpoint metadata and bitmap in image will be inconsistent
[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started
 
[root@dell-per740-01 ~]# virsh checkpoint-list vm1
 Name   Creation Time
-----------------------------------
 ck1    2021-03-16 08:20:58 -0400

[root@dell-per740-01 ~]# virsh checkpoint-delete vm1 ck1
error: Failed to delete checkpoint ck1
error: internal error: bitmap 'ck1' not found in backing chain of 'vda'

(In reply to yisun from comment #11)
> Hi Peter,
> Here are 2 questions during the test, pls help to confirm if they worked as
> design
> 
> Q1: If the dest host has a exisiting image, and it contains existing
> bitmaps, the migration won't erase it. This may cause inconsistence between
> source and dest host.
> 1. scp a image with dirty bitmap to dest host，this is to make sure iamge on
> dest host has dirty bitmap 'ck1'
> [root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test
> --name ck1 --diskspec vda,checkpoint=bitmap
> Domain checkpoint ck1 created
> 
> [root@dell-per740-01 ~]# virsh destroy vm1
> Domain 'vm1' destroyed
> 
> [root@dell-per740-01 ~]# scp /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> root.eng.pek2.redhat.com:/var/lib/libvirt/images/jeos-27-
> x86_64.qcow2
> Warning: Permanently added 'lenovo-sr630-10.lab.eng.pek2.redhat.com' (ECDSA)
> to the list of known hosts.
> root.eng.pek2.redhat.com's password: 
> jeos-27-x86_64.qcow2                                                        
> 100%  931MB 109.5MB/s   00:08    
> <=== now the image on dest host should have a bitmap named 'ck1'
> 
> 2. delete the dirty bitmap of the image on source host
> [root@dell-per740-01 ~]# virsh start vm1
> Domain 'vm1' started
> 
> [root@dell-per740-01 ~]# virsh checkpoint-delete vm1 ck1
> Domain checkpoint ck1 deleted
> <=== this is to make sure the image on source host has no bitmap now.
> 
> 3. do migration
> [root@dell-per740-01 ~]# virsh migrate vm1 --live
> qemu+ssh://xxxxxx.redhat.com/system --verbose --copy-storage-all
> root.eng.pek2.redhat.com's password: 
> Migration: [100 %]
> 
> 4. on dest host, check the image info
> [root@lenovo-sr630-10 images]# virsh destroy vm1
> Domain 'vm1' destroyed
> 
> [root@lenovo-sr630-10 images]# qemu-img info
> /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> file format: qcow2
> virtual size: 10 GiB (10737418240 bytes)
> disk size: 1.75 GiB
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     compression type: zlib
>     lazy refcounts: false
>     bitmaps:
>         [0]:
>             flags:
>                 [0]: auto
>             name: ck1
>             granularity: 65536
>     refcount bits: 16
>     corrupt: false
>     extended l2: false
> <=== here we can see the image still has bitmap 'ck1', this is inconsistent
> with source host image's current status.

This is okay. Users must create the image correctly.

Specifically, the bitmap migration is best-effort so that it can be enabled by default and doesn't break any other aspect of migration.


> Q2: When migration finished, the bitmap in the source image was removed.
> 1. start a vm with image has no bitmaps
> [root@dell-per740-01 ~]# virsh domblklist vm1
>  Target   Source
> --------------------------------------------------------
>  vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> 
> [root@dell-per740-01 ~]# qemu-img info
> /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> file format: qcow2
> virtual size: 10 GiB (10737418240 bytes)
> disk size: 1 GiB
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     compression type: zlib
>     lazy refcounts: false
>     refcount bits: 16
>     corrupt: false
>     extended l2: false
> [root@dell-per740-01 ~]# virsh start vm1
> Domain 'vm1' started
> 
> 2. create a checkpoint for vm's vda
> [root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test
> --name ck1 --diskspec vda,checkpoint=bitmap
> Domain checkpoint ck1 created
> 
> 3. do migration
> [root@dell-per740-01 ~]# virsh migrate vm1 --live
> qemu+ssh://lenovo-sr630-10.lab.eng.pek2.redhat.com/system --verbose
> --copy-storage-all
> root.eng.pek2.redhat.com's password: 
> Migration: [100 %]
> 
> 4. after migration, check the image's info on source host, the bitmap not
> saved in that image
> [root@dell-per740-01 ~]# qemu-img info
> /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> file format: qcow2
> virtual size: 10 GiB (10737418240 bytes)
> disk size: 1 GiB
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     compression type: zlib
>     lazy refcounts: false
>     refcount bits: 16
>     corrupt: false
>     extended l2: false
> 
> 5. if we use this vm on source host again, the checkpoint metadata and
> bitmap in image will be inconsistent
> [root@dell-per740-01 ~]# virsh start vm1
> Domain 'vm1' started
>  
> [root@dell-per740-01 ~]# virsh checkpoint-list vm1
>  Name   Creation Time
> -----------------------------------
>  ck1    2021-03-16 08:20:58 -0400
> 
> [root@dell-per740-01 ~]# virsh checkpoint-delete vm1 ck1
> error: Failed to delete checkpoint ck1
> error: internal error: bitmap 'ck1' not found in backing chain of 'vda'

This is weird. Please file a new bug for this case.

(In reply to Peter Krempa from comment #12)
> (In reply to yisun from comment #11)
> > Hi Peter,
> > Here are 2 questions during the test, pls help to confirm if they worked as
> > design
> > 
> > Q1: If the dest host has a exisiting image, and it contains existing
> > bitmaps, the migration won't erase it. This may cause inconsistence between
> > source and dest host.
> > 1. scp a image with dirty bitmap to dest host，this is to make sure iamge on
> > dest host has dirty bitmap 'ck1'
> > [root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test
> > --name ck1 --diskspec vda,checkpoint=bitmap
> > Domain checkpoint ck1 created
> > 
> > [root@dell-per740-01 ~]# virsh destroy vm1
> > Domain 'vm1' destroyed
> > 
> > [root@dell-per740-01 ~]# scp /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > root.eng.pek2.redhat.com:/var/lib/libvirt/images/jeos-27-
> > x86_64.qcow2
> > Warning: Permanently added 'lenovo-sr630-10.lab.eng.pek2.redhat.com' (ECDSA)
> > to the list of known hosts.
> > root.eng.pek2.redhat.com's password: 
> > jeos-27-x86_64.qcow2                                                        
> > 100%  931MB 109.5MB/s   00:08    
> > <=== now the image on dest host should have a bitmap named 'ck1'
> > 
> > 2. delete the dirty bitmap of the image on source host
> > [root@dell-per740-01 ~]# virsh start vm1
> > Domain 'vm1' started
> > 
> > [root@dell-per740-01 ~]# virsh checkpoint-delete vm1 ck1
> > Domain checkpoint ck1 deleted
> > <=== this is to make sure the image on source host has no bitmap now.
> > 
> > 3. do migration
> > [root@dell-per740-01 ~]# virsh migrate vm1 --live
> > qemu+ssh://xxxxxx.redhat.com/system --verbose --copy-storage-all
> > root.eng.pek2.redhat.com's password: 
> > Migration: [100 %]
> > 
> > 4. on dest host, check the image info
> > [root@lenovo-sr630-10 images]# virsh destroy vm1
> > Domain 'vm1' destroyed
> > 
> > [root@lenovo-sr630-10 images]# qemu-img info
> > /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > file format: qcow2
> > virtual size: 10 GiB (10737418240 bytes)
> > disk size: 1.75 GiB
> > cluster_size: 65536
> > Format specific information:
> >     compat: 1.1
> >     compression type: zlib
> >     lazy refcounts: false
> >     bitmaps:
> >         [0]:
> >             flags:
> >                 [0]: auto
> >             name: ck1
> >             granularity: 65536
> >     refcount bits: 16
> >     corrupt: false
> >     extended l2: false
> > <=== here we can see the image still has bitmap 'ck1', this is inconsistent
> > with source host image's current status.
> 
> This is okay. Users must create the image correctly.
> 
> Specifically, the bitmap migration is best-effort so that it can be enabled
> by default and doesn't break any other aspect of migration.
> 
> 
> > Q2: When migration finished, the bitmap in the source image was removed.
> > 1. start a vm with image has no bitmaps
> > [root@dell-per740-01 ~]# virsh domblklist vm1
> >  Target   Source
> > --------------------------------------------------------
> >  vda      /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > 
> > [root@dell-per740-01 ~]# qemu-img info
> > /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > file format: qcow2
> > virtual size: 10 GiB (10737418240 bytes)
> > disk size: 1 GiB
> > cluster_size: 65536
> > Format specific information:
> >     compat: 1.1
> >     compression type: zlib
> >     lazy refcounts: false
> >     refcount bits: 16
> >     corrupt: false
> >     extended l2: false
> > [root@dell-per740-01 ~]# virsh start vm1
> > Domain 'vm1' started
> > 
> > 2. create a checkpoint for vm's vda
> > [root@dell-per740-01 ~]# virsh checkpoint-create-as vm1 --description test
> > --name ck1 --diskspec vda,checkpoint=bitmap
> > Domain checkpoint ck1 created
> > 
> > 3. do migration
> > [root@dell-per740-01 ~]# virsh migrate vm1 --live
> > qemu+ssh://lenovo-sr630-10.lab.eng.pek2.redhat.com/system --verbose
> > --copy-storage-all
> > root.eng.pek2.redhat.com's password: 
> > Migration: [100 %]
> > 
> > 4. after migration, check the image's info on source host, the bitmap not
> > saved in that image
> > [root@dell-per740-01 ~]# qemu-img info
> > /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > image: /var/lib/libvirt/images/jeos-27-x86_64.qcow2
> > file format: qcow2
> > virtual size: 10 GiB (10737418240 bytes)
> > disk size: 1 GiB
> > cluster_size: 65536
> > Format specific information:
> >     compat: 1.1
> >     compression type: zlib
> >     lazy refcounts: false
> >     refcount bits: 16
> >     corrupt: false
> >     extended l2: false
> > 
> > 5. if we use this vm on source host again, the checkpoint metadata and
> > bitmap in image will be inconsistent
> > [root@dell-per740-01 ~]# virsh start vm1
> > Domain 'vm1' started
> >  
> > [root@dell-per740-01 ~]# virsh checkpoint-list vm1
> >  Name   Creation Time
> > -----------------------------------
> >  ck1    2021-03-16 08:20:58 -0400
> > 
> > [root@dell-per740-01 ~]# virsh checkpoint-delete vm1 ck1
> > error: Failed to delete checkpoint ck1
> > error: internal error: bitmap 'ck1' not found in backing chain of 'vda'
> 
> This is weird. Please file a new bug for this case.

Thanks for the info, open a new Bug 1940036 to track this, set current bz to VERIFIED then.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098