Description: [incremental_backup] Cannot create external disk&mem snapshot for a vm's whose disk img has existing checkpoint dirtybitmap Version: libvirt-6.5.0-1.module+el8.3.0+7323+d54bb644.x86_64 Steps: 1. Prepare a clean qcow2 img: [root@libvirt-rhel-8 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 1G Formatting '/var/lib/libvirt/images/test.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib [root@libvirt-rhel-8 ~]# virsh domblklist vm1 Target Source ---------------------------------------------- vda /var/lib/libvirt/images/test.qcow2 2. Start the vm [root@libvirt-rhel-8 ~]# virsh start vm1 Domain vm1 started 3. Clear libvirtd debug log for new debug info [root@libvirt-rhel-8 ~]# echo "" > /var/log/libvirtd-debug.log 4. Create a checkponit 'ck1' for vm's vda [root@libvirt-rhel-8 ~]# virsh checkpoint-create-as vm1 ck1 Domain checkpoint ck1 created 5. Create diskonly snapshot twice: [root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap1 --diskspec vda,snapshot=external,file=/tmp/disk_s1 --disk-only Domain snapshot snap1 created [root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap2 --diskspec vda,snapshot=external,file=/tmp/disk_s2 --disk-only Domain snapshot snap2 created 6. Create external mem&disk snapshot, it will be failed [root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap3 --memspec snapshot=external,file=/tmp/mem_s3 --diskspec vda,snapshot=external,file=/tmp/disk_s3 error: internal error: unable to execute QEMU command 'cont': Could not reopen qcow2 layer: Bitmap already exists: ck1 7. Create diskonly snapshot again, it'll be failed now. [root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap4 --diskspec vda,snapshot=external,file=/tmp/disk_s4 --disk-only error: internal error: unable to execute QEMU command 'query-named-block-nodes': Block device libvirt-1-format is ejected Actual result: As step 6 and 7, cannot create external snapshot Expected result: External snapshots should always be successful Additional info: If no checkpoint created, nothing wrong If create mem&disk snapshot just after the checkpoint creation step, it will be successful for the 1st time, but will be failed after that
Created attachment 1701726 [details] libvirtd.log
libvirtd debug log attached as "libvirtd.log" For now the incremental backup function is not enabled by default, we need to edit vm's xml as follow to enable it: [root@libvirt-rhel-8 ~]# virsh dumpxml vm1 <domain type='kvm' id='18' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> ... ... <qemu:capabilities> <qemu:add capability='incremental-backup'/> </qemu:capabilities> </domain>
This is a qemu bug. Pre-requisite is that there must be bitmaps in the backing image present. If 'cont' is issued after a migration completes successfuly the error is reported. There's no need to create snapshots during the lifetime of the VM. The reproducer is as follows: Create backing chain: qemu-img create -f qcow2 /tmp/backup-test-images/backup-disk-base.qcow2 100M qemu-img bitmap -f qcow2 /tmp/backup-test-images/backup-disk-base.qcow2 --add asdf qemu-img create -f qcow2 -F qcow2 -b /tmp/backup-test-images/backup-disk-base.qcow2 /tmp/backup-test-images/backup-disk-overlay.qcow2 # Note that this one is not required to reproduce the bug. It's for demonstration of one of the symptoms qemu-img bitmap -f qcow2 /tmp/backup-test-images/backup-disk-overlay.qcow2 --add bsdf Start the VM. Note that libvirt is used just to start the VM. All migration will be executed manually. virsh attach-disk backup-test /tmp/backup-test-images/backup-disk-overlay.qcow2 vda --subdriver qcow2 --config virsh start backup-test virsh qemu-monitor-command --pretty backup-test '{"execute":"stop"}' virsh qemu-monitor-command --pretty backup-test '{"execute":"query-named-block-nodes","arguments":{"flat":true}}' > /tmp/backup-test-images/nodes-migr-pre.json nc -l -p 12345 > /dev/null & virsh qemu-monitor-command --pretty backup-test '{"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"tcp:localhost:12345"}}' After migration finishes: virsh qemu-monitor-command --pretty backup-test '{"execute":"query-named-block-nodes","arguments":{"flat":true}}' > /tmp/backup-test-images/nodes-migr-post.json virsh qemu-monitor-command --pretty backup-test '{"execute":"cont"}' The last 'cont' returns: { "id": "libvirt-377", "error": { "class": "GenericError", "desc": "Could not reopen qcow2 layer: Bitmap already exists: asdf" } } Comparing the pre- and post- migration 'query-named-block-nodes' reply the '/tmp/backup-test-images/backup-disk-overlay.qcow2' image is completely missing the 'dirty-bitmaps' section after the migration, but the base image still has it. Please note that libvirt actually needs the list of bitmaps at that point to be complete as we use it to propagate active bitmaps into the new overlay when creating a snapshot as the original reproducer shows. If that is for some reason not expected to be present at that point, please file a libvirt bug explaining why it's impossible so that we can povide a workaround. Also note that if I try to create a snapshot before 'cont' qemu breaks even more as the original bugreport states' ('Block device libvirt-1-format is ejected' after query-named-block-nodes). Libvirt potentially also uses this code path during regular migration (not for snapshots) if the resume of the destination fails for some reason after the migration completes. We then attempt to re-start the source.
Please add the qemu version which was used in this instance. (But it doesn't matter that much as I've reproduced it also with current upstream)
The qemu version of comment 0 is: qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64
Test on qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420, can reproduce this issue. Test Steps: 1.Start guest with qemu cmds ... -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ ... 2.Add a persistent bitmap. { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}} 3.Quit vm (qemu)quit 4.Create a snapshot file #qemu-img create -f qcow2 /home/kvm_autotest_root/images/snapshot.qcow2 -b /home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2 20G 5.Start guest with snapshot.qcow2 and add a persistent bitmap to it. ... -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ ... { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}} 6. Check bitmap info in src. -- All bitmaps info displayed {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block105", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block043", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 69816320, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 68091904}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 70123520, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 69816320, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 7.Start guest with -incoming and start migration from src to dst. ... -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -incoming tcp:0:5000 ... {"execute":"migrate","arguments":{"uri":"tcp:0:5000"}} 8. Wait until migration completed. {"timestamp": {"seconds": 1595585607, "microseconds": 64956}, "event": "STOP"} {"execute":"query-migrate"} {"return": {"status": "completed", "setup-time": 17, "downtime": 25, "total-time": 80682, "ram": {"total": 32229892096, "postcopy-requests": 0, "dirty-sync-count": 4, "multifd-bytes": 0, "pages-per-second": 23740, "page-size": 4096, "remaining": 0, "mbps": 267.169351, "transferred": 2693554423, "duplicate": 7246988, "dirty-pages-rate": 0, "skipped": 0, "normal-bytes": 2623201280, "normal": 640430}}} 9. Check bitmap info in src. -- snapshot bitmap detailed info not displayed {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block105", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block043", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 70217728, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 70586368, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 70217728, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 10. Check bitmap info in dst. -- All bitmaps info displayed {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block138", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block029", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 70217728, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 68485120}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 70586368, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 70217728, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 11. Quit dst vm. (qemu)quit 12. Cont src vm. (qemu)c Error: Could not reopen qcow2 layer: Bitmap already exists: bitmap0 (qemu) info status VM status: paused (postmigrate) Note: If migration failed from src to dst, example dst not started, bitmaps info in src will displayed correctly, details as bellow: Test Steps: 1. Start guest with qemu cmds cmd same as above 2. Add a persistent bitmap. cmd same as above 3. Quit vm cmd same as above 4. Create a snapshot file cmd same as above 5. Start guest with snapshot.qcow2 and add a persistent bitmap to it. cmd same as above 6. Check bitmap info in src. -- All bitmaps info displayed cmd and result same as above 7. Do migration without dst. -- migration failed with {"execute":"migrate","arguments":{"uri":"tcp:0:5000"}} {"execute":"query-migrate"} {"return": {"status": "failed", "error-desc": "Failed to connect socket: Connection refused"}} 8. Check bitmaps info in src. -- All bitmaps info displayed. {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block103", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block050", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 70799360, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 69140480}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 71106560, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 70799360, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}
So far, it looks to me (with not too much background in the bitmap migration code) like the problem is that qcow2_store_persistent_bitmaps() does not release read-only persistent bitmaps, so it does not release the backing nodes’ persistent bitmaps during migration, and then there’s a conflict when we try to reload them. I think I’ll send a patch upstream to make qcow2_store_persistent_bitmaps() release read-only bitmaps and see what others think. (It’s also possible that the problem is rather on the other end, i.e., when reloading the bitmaps after continuing on the source, we should just ignore all read-only bitmaps that we have loaded already. I hope this will become clear in an upstream discussion.)
(In reply to Peter Krempa from comment #3) > Comparing the pre- and post- migration 'query-named-block-nodes' reply the > '/tmp/backup-test-images/backup-disk-overlay.qcow2' image is completely > missing the 'dirty-bitmaps' section after the migration, but the base image > still has it. > > Please note that libvirt actually needs the list of bitmaps at that point to > be complete as we use it to propagate active bitmaps into the new overlay > when creating a snapshot as the original reproducer shows. If that is for > some reason not expected to be present at that point, please file a libvirt > bug explaining why it's impossible so that we can povide a workaround. OK, this will become difficult. As far as I can see from the code, it’s entirely intentional that all bitmaps are released as part of the migration process. I would have to discuss this upstream with Vladimir from Virtuozzo first. But was that section ever not gone? I.e., does libvirt assume that this section is filled after migration, and that used to work in the past? (Sure, it isn’t gone for bitmaps in the backing chain, but to me that’s the bug, actually.) One workaround that comes to my mind is in addition to querying @dirty-bitmaps, you could go to image/format-specific/data/bitmaps, which for qcow2 images always lists all persistent bitmaps.
(In reply to Max Reitz from comment #9) > (In reply to Peter Krempa from comment #3) > > Comparing the pre- and post- migration 'query-named-block-nodes' reply the > > '/tmp/backup-test-images/backup-disk-overlay.qcow2' image is completely > > missing the 'dirty-bitmaps' section after the migration, but the base image > > still has it. > > > > Please note that libvirt actually needs the list of bitmaps at that point to > > be complete as we use it to propagate active bitmaps into the new overlay > > when creating a snapshot as the original reproducer shows. If that is for > > some reason not expected to be present at that point, please file a libvirt > > bug explaining why it's impossible so that we can povide a workaround. > > OK, this will become difficult. As far as I can see from the code, it’s > entirely intentional that all bitmaps are released as part of the migration > process. That is fair enough, but mentioning it somewhere might be a good idea. > I would have to discuss this upstream with Vladimir from Virtuozzo first. > But was that section ever not gone? I.e., does libvirt assume that this > section is filled after migration, and that used to work in the past? > (Sure, it isn’t gone for bitmaps in the backing chain, but to me that’s the > bug, actually.) It might have always vanished. That's why I've mentioned that if it's intentional, libvirt needs to be fixed. > One workaround that comes to my mind is in addition to querying > @dirty-bitmaps, you could go to image/format-specific/data/bitmaps, which > for qcow2 images always lists all persistent bitmaps. I originally thought that it was deprecated along with the 'status' field of the 'dirty-bitmaps' section, but that's not true. So I can either use that, or fetch the data before kicking off the migration, to be safe from any migration-related changes. Thus if this is really intentional and expected, please let me know so I can fix it in libvirt.
OK, thanks. Then I think I’ll send my patch upstream and hope we get some discussion on what is the best way to resolve this.
I've filed bug 1862472 to track that 'query-named-block-nodes' provides wrong results in regards to bitmaps after the migration.
Fix is upstream for 5.1: commit fe16c7ddf82b25e54cb76936a4cdf6bf9327ece5 Author: Max Reitz <mreitz> Date: Thu Jul 30 14:02:33 2020 +0200 qcow2: Release read-only bitmaps when inactivated During migration, we release all bitmaps after storing them on disk, as long as they are (1) stored on disk, (2) not read-only, and (3) consistent. (2) seems arbitrary, though. The reason we do not release them is because we do not write them, as there is no need to; and then we just forget about all bitmaps that we have not written to the file. However, read-only persistent bitmaps are still in the file and in sync with their in-memory representation, so we may as well release them just like any R/W bitmap that we have updated. It leads to actual problems, too: After migration, letting the source continue may result in an error if there were any bitmaps on read-only nodes (such as backing images), because those have not been released by bdrv_inactive_all(), but bdrv_invalidate_cache_all() attempts to reload them (which fails, because they are still present in memory). Signed-off-by: Max Reitz <mreitz> Message-Id: <20200730120234.49288-2-mreitz> Tested-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> Signed-off-by: Eric Blake <eblake>
Danilo - yet another bug that is fixed by the rebase... Needinfo for you to update data into ON_QA
Thanks for the headsup John. Adding TestOnly and Fixed-in field and moving to ON_QA @aihua QA_ACK, please?
Test on qemu-kvm-5.1.0-3.module+el8.3.0+7708+740a1315, the problem has been resolved, set bug's status to "Verified". Test steps: 1.Start guest with qemu cmds: ... -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ ... 2.Add persistent bitmap to it. { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}} 3.Quit vm (qemu)quit 4.Create a snapshot chain qemu-img create -f qcow2 -F qcow2 /home/kvm_autotest_root/images/snapshot.qcow2 -b /home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2 20G 5.Start src with snapshot image ... -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ ... 6.Add persistent bitmap to it { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}} 7.Check bitmap info in src {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block113", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block061", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 47718400, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 44761088}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 47120384, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 47718400, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 8.Start dst guest with -incoming ... -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -incoming tcp:0:5000 \ ... 9.Check bitmap info in dst {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block158", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block067", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 493883392, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 493551616, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 493883392, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 10.Write a file in src guest: (guest)dd if=/dev/urandom of=test bs=1M count=400 11.Check bitmap info in src: {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block113", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block061", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 493883392, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 491257856}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 493551616, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 493883392, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 12.Migrate from src to dst {"execute":"migrate","arguments":{"uri":"tcp:0:5000"}} {"return": {}} {"timestamp": {"seconds": 1597920364, "microseconds": 508824}, "event": "STOP"} 13.Check bitmap info in dst {"timestamp": {"seconds": 1597920364, "microseconds": 541802}, "event": "RESUME"} {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block158", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block067", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 493883392, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 491257856}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 493686784, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 493883392, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 14.Check bitmap info in src after migration {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block115", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block092", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 44044288, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 42405888, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 44044288, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} 15.Quit vm in dst (qemu)quit 16.Cont vm in src (qemu)cont {"timestamp": {"seconds": 1597920494, "microseconds": 455749}, "event": "RESUME"} 17.Check bitmap info in src {"execute":"query-named-block-nodes"} {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block113", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block061", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 808456192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 805175296}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 807534592, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 808456192, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]} And after step17, can still run migration from src to other dst.
Won't cover it, for it's testonly.
Because it has code change, so will cover this case.
RHEL AV 8.3.0 has been shipped, therefore marking this BZ CLOSED CURRENTRELEASE.