Bug 1858739 - 'cont' on source fails with 'Could not reopen qcow2 layer: Bitmap already exists: ck1' after successful migration when backing images contain block-dirty-bitmaps
Summary: 'cont' on source fails with 'Could not reopen qcow2 layer: Bitmap already ex...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.3
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 8.3
Assignee: Max Reitz
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks: 1799011 1862472
TreeView+ depends on / blocked
 
Reported: 2020-07-20 09:14 UTC by yisun
Modified: 2020-10-14 22:11 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-5.1.0-3.module+el8.3.0+7708+740a1315
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
libvirtd.log (1.12 MB, text/plain)
2020-07-20 09:18 UTC, yisun
no flags Details

Description yisun 2020-07-20 09:14:54 UTC
Description:
[incremental_backup] Cannot create external disk&mem snapshot for a vm's whose disk img has existing checkpoint dirtybitmap

Version:
libvirt-6.5.0-1.module+el8.3.0+7323+d54bb644.x86_64

Steps:
1. Prepare a clean qcow2 img:
[root@libvirt-rhel-8 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 1G
Formatting '/var/lib/libvirt/images/test.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
[root@libvirt-rhel-8 ~]# virsh domblklist vm1
 Target   Source
----------------------------------------------
 vda      /var/lib/libvirt/images/test.qcow2

2. Start the vm
[root@libvirt-rhel-8 ~]# virsh start vm1
Domain vm1 started

3. Clear libvirtd debug log for new debug info
[root@libvirt-rhel-8 ~]# echo "" > /var/log/libvirtd-debug.log

4. Create a checkponit 'ck1' for vm's vda
[root@libvirt-rhel-8 ~]# virsh checkpoint-create-as vm1 ck1
Domain checkpoint ck1 created

5. Create diskonly snapshot twice:
[root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap1 --diskspec vda,snapshot=external,file=/tmp/disk_s1 --disk-only
Domain snapshot snap1 created
[root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap2 --diskspec vda,snapshot=external,file=/tmp/disk_s2 --disk-only
Domain snapshot snap2 created

6. Create external mem&disk snapshot, it will be failed
[root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap3 --memspec snapshot=external,file=/tmp/mem_s3 --diskspec vda,snapshot=external,file=/tmp/disk_s3
error: internal error: unable to execute QEMU command 'cont': Could not reopen qcow2 layer: Bitmap already exists: ck1

7. Create diskonly snapshot again, it'll be failed now.
[root@libvirt-rhel-8 ~]# virsh snapshot-create-as vm1 snap4 --diskspec vda,snapshot=external,file=/tmp/disk_s4 --disk-only
error: internal error: unable to execute QEMU command 'query-named-block-nodes': Block device libvirt-1-format is ejected

Actual result:
As step 6 and 7, cannot create external snapshot

Expected result:
External snapshots should always be successful

Additional info:
If no checkpoint created, nothing wrong
If create mem&disk snapshot just after the checkpoint creation step, it will be successful for the 1st time, but will be failed after that

Comment 1 yisun 2020-07-20 09:18:05 UTC
Created attachment 1701726 [details]
libvirtd.log

Comment 2 yisun 2020-07-20 09:19:58 UTC
libvirtd debug log attached as "libvirtd.log"

For now the incremental backup function is not enabled by default, we need to edit vm's xml as follow to enable it:
[root@libvirt-rhel-8 ~]# virsh dumpxml vm1

<domain type='kvm' id='18' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...
...
  <qemu:capabilities>
    <qemu:add capability='incremental-backup'/>
  </qemu:capabilities>
</domain>

Comment 3 Peter Krempa 2020-07-20 14:54:53 UTC
This is a qemu bug. Pre-requisite is that there must be bitmaps in the backing image present.

If 'cont' is issued after a migration completes successfuly the error is reported. There's no need to create snapshots during the lifetime of the VM.

The reproducer is as follows:

Create backing chain:

qemu-img create -f qcow2  /tmp/backup-test-images/backup-disk-base.qcow2 100M
qemu-img bitmap -f qcow2 /tmp/backup-test-images/backup-disk-base.qcow2 --add asdf

qemu-img create -f qcow2 -F qcow2 -b /tmp/backup-test-images/backup-disk-base.qcow2 /tmp/backup-test-images/backup-disk-overlay.qcow2
# Note that this one is not required to reproduce the bug. It's for demonstration of one of the symptoms
qemu-img bitmap -f qcow2 /tmp/backup-test-images/backup-disk-overlay.qcow2 --add bsdf

Start the VM. Note that libvirt is used just to start the VM. All migration will be executed manually.

virsh attach-disk backup-test /tmp/backup-test-images/backup-disk-overlay.qcow2 vda --subdriver qcow2 --config
virsh start backup-test

virsh qemu-monitor-command --pretty backup-test '{"execute":"stop"}'
virsh qemu-monitor-command --pretty backup-test '{"execute":"query-named-block-nodes","arguments":{"flat":true}}' > /tmp/backup-test-images/nodes-migr-pre.json

nc -l -p 12345 > /dev/null &
virsh qemu-monitor-command --pretty backup-test '{"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"tcp:localhost:12345"}}'

After migration finishes:

virsh qemu-monitor-command --pretty backup-test '{"execute":"query-named-block-nodes","arguments":{"flat":true}}' > /tmp/backup-test-images/nodes-migr-post.json
virsh qemu-monitor-command --pretty backup-test '{"execute":"cont"}'

The last 'cont' returns:

{
  "id": "libvirt-377",
  "error": {
    "class": "GenericError",
    "desc": "Could not reopen qcow2 layer: Bitmap already exists: asdf"
  }
}

Comparing the pre- and post- migration 'query-named-block-nodes' reply the '/tmp/backup-test-images/backup-disk-overlay.qcow2' image is completely missing the 'dirty-bitmaps' section after the migration, but the base image still has it.

Please note that libvirt actually needs the list of bitmaps at that point to be complete as we use it to propagate active bitmaps into the new overlay when creating a snapshot as the original reproducer shows. If that is for some reason not expected to be present at that point, please file a libvirt bug explaining why it's impossible so that we can povide a workaround.

Also note that if I try to create a snapshot before 'cont' qemu breaks even more as the original bugreport states' ('Block device libvirt-1-format is ejected' after query-named-block-nodes).

Libvirt potentially also uses this code path during regular migration (not for snapshots) if the resume of the destination fails for some reason after the migration completes. We then attempt to re-start the source.

Comment 4 Peter Krempa 2020-07-20 14:55:52 UTC
Please add the qemu version which was used in this instance. (But it doesn't matter that much as I've reproduced it also with current upstream)

Comment 5 yisun 2020-07-21 01:36:08 UTC
The qemu version of comment 0 is: qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64

Comment 7 aihua liang 2020-07-24 10:31:43 UTC
Test on qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420, can reproduce this issue.

Test Steps:
 1.Start guest with qemu cmds
    ...
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
   ...

 2.Add a persistent bitmap.
    { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}}

 3.Quit vm
    (qemu)quit

 4.Create a snapshot file
    #qemu-img create -f qcow2 /home/kvm_autotest_root/images/snapshot.qcow2 -b /home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2 20G

 5.Start guest with snapshot.qcow2 and add a persistent bitmap to it.
    ...
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
   ...

   { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}}

 6. Check bitmap info in src. -- All bitmaps info displayed
    {"execute":"query-named-block-nodes"}
     {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block105", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block043", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 69816320, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 68091904}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 70123520, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 69816320, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

 7.Start guest with -incoming and start migration from src to dst.
    ...
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -incoming tcp:0:5000
   ...
     
     {"execute":"migrate","arguments":{"uri":"tcp:0:5000"}}

 
  8. Wait until migration completed.
    {"timestamp": {"seconds": 1595585607, "microseconds": 64956}, "event": "STOP"}
    {"execute":"query-migrate"}
{"return": {"status": "completed", "setup-time": 17, "downtime": 25, "total-time": 80682, "ram": {"total": 32229892096, "postcopy-requests": 0, "dirty-sync-count": 4, "multifd-bytes": 0, "pages-per-second": 23740, "page-size": 4096, "remaining": 0, "mbps": 267.169351, "transferred": 2693554423, "duplicate": 7246988, "dirty-pages-rate": 0, "skipped": 0, "normal-bytes": 2623201280, "normal": 640430}}}

  9. Check bitmap info in src. -- snapshot bitmap detailed info not displayed
     {"execute":"query-named-block-nodes"}
     {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block105", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block043", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 70217728, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 70586368, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 70217728, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

  10. Check bitmap info in dst. -- All bitmaps info displayed
      {"execute":"query-named-block-nodes"}
      {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block138", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block029", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 70217728, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 68485120}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 70586368, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 70217728, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

 11. Quit dst vm.
     (qemu)quit

 12. Cont src vm.
     (qemu)c
Error: Could not reopen qcow2 layer: Bitmap already exists: bitmap0
     (qemu) info status 
VM status: paused (postmigrate)


Note: If migration failed from src to dst, example dst not started, bitmaps info in src will displayed correctly, details as bellow:
 Test Steps:
  1. Start guest with qemu cmds
      cmd same as above
  2. Add a persistent bitmap.
      cmd same as above

  3. Quit vm
      cmd same as above

  4. Create a snapshot file
      cmd same as above

  5. Start guest with snapshot.qcow2 and add a persistent bitmap to it.
      cmd same as above

  6. Check bitmap info in src. -- All bitmaps info displayed
      cmd and result same as above
  
  7. Do migration without dst. -- migration failed with 
     {"execute":"migrate","arguments":{"uri":"tcp:0:5000"}}
     {"execute":"query-migrate"}
{"return": {"status": "failed", "error-desc": "Failed to connect socket: Connection refused"}}

  8. Check bitmaps info in src. -- All bitmaps info displayed.
      {"execute":"query-named-block-nodes"}
      {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block103", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 39845888}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 5045620736, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 5045379072, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block050", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5045379072, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 70799360, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 69140480}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 71106560, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 70799360, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

Comment 8 Max Reitz 2020-07-24 14:18:11 UTC
So far, it looks to me (with not too much background in the bitmap migration code) like the problem is that qcow2_store_persistent_bitmaps() does not release read-only persistent bitmaps, so it does not release the backing nodes’ persistent bitmaps during migration, and then there’s a conflict when we try to reload them.

I think I’ll send a patch upstream to make qcow2_store_persistent_bitmaps() release read-only bitmaps and see what others think.  (It’s also possible that the problem is rather on the other end, i.e., when reloading the bitmaps after continuing on the source, we should just ignore all read-only bitmaps that we have loaded already.  I hope this will become clear in an upstream discussion.)

Comment 9 Max Reitz 2020-07-24 14:51:39 UTC
(In reply to Peter Krempa from comment #3)
> Comparing the pre- and post- migration 'query-named-block-nodes' reply the
> '/tmp/backup-test-images/backup-disk-overlay.qcow2' image is completely
> missing the 'dirty-bitmaps' section after the migration, but the base image
> still has it.
> 
> Please note that libvirt actually needs the list of bitmaps at that point to
> be complete as we use it to propagate active bitmaps into the new overlay
> when creating a snapshot as the original reproducer shows. If that is for
> some reason not expected to be present at that point, please file a libvirt
> bug explaining why it's impossible so that we can povide a workaround.

OK, this will become difficult.  As far as I can see from the code, it’s entirely intentional that all bitmaps are released as part of the migration process.

I would have to discuss this upstream with Vladimir from Virtuozzo first.  But was that section ever not gone?  I.e., does libvirt assume that this section is filled after migration, and that used to work in the past?  (Sure, it isn’t gone for bitmaps in the backing chain, but to me that’s the bug, actually.)

One workaround that comes to my mind is in addition to querying @dirty-bitmaps, you could go to image/format-specific/data/bitmaps, which for qcow2 images always lists all persistent bitmaps.

Comment 10 Peter Krempa 2020-07-27 09:29:13 UTC
(In reply to Max Reitz from comment #9)
> (In reply to Peter Krempa from comment #3)
> > Comparing the pre- and post- migration 'query-named-block-nodes' reply the
> > '/tmp/backup-test-images/backup-disk-overlay.qcow2' image is completely
> > missing the 'dirty-bitmaps' section after the migration, but the base image
> > still has it.
> > 
> > Please note that libvirt actually needs the list of bitmaps at that point to
> > be complete as we use it to propagate active bitmaps into the new overlay
> > when creating a snapshot as the original reproducer shows. If that is for
> > some reason not expected to be present at that point, please file a libvirt
> > bug explaining why it's impossible so that we can povide a workaround.
> 
> OK, this will become difficult.  As far as I can see from the code, it’s
> entirely intentional that all bitmaps are released as part of the migration
> process.

That is fair enough, but mentioning it somewhere might be a good idea.

> I would have to discuss this upstream with Vladimir from Virtuozzo first. 
> But was that section ever not gone?  I.e., does libvirt assume that this
> section is filled after migration, and that used to work in the past? 
> (Sure, it isn’t gone for bitmaps in the backing chain, but to me that’s the
> bug, actually.)

It might have always vanished. That's why I've mentioned that if it's intentional, libvirt needs to be fixed.

> One workaround that comes to my mind is in addition to querying
> @dirty-bitmaps, you could go to image/format-specific/data/bitmaps, which
> for qcow2 images always lists all persistent bitmaps.

I originally thought that it was deprecated along with the 'status' field of the 'dirty-bitmaps' section, but that's not true. So I can either use that, or fetch the data before kicking off the migration, to be safe from any migration-related changes. 

Thus if this is really intentional and expected, please let me know so I can fix it in libvirt.

Comment 11 Max Reitz 2020-07-27 14:04:55 UTC
OK, thanks. Then I think I’ll send my patch upstream and hope we get some discussion on what is the best way to resolve this.

Comment 12 Peter Krempa 2020-07-31 14:03:46 UTC
I've filed bug 1862472 to track that 'query-named-block-nodes' provides wrong results in regards to bitmaps after the migration.

Comment 13 Eric Blake 2020-08-03 16:06:56 UTC
Fix is upstream for 5.1:
commit fe16c7ddf82b25e54cb76936a4cdf6bf9327ece5
Author: Max Reitz <mreitz@redhat.com>
Date:   Thu Jul 30 14:02:33 2020 +0200

    qcow2: Release read-only bitmaps when inactivated
    
    During migration, we release all bitmaps after storing them on disk, as
    long as they are (1) stored on disk, (2) not read-only, and (3)
    consistent.
    
    (2) seems arbitrary, though.  The reason we do not release them is
    because we do not write them, as there is no need to; and then we just
    forget about all bitmaps that we have not written to the file.  However,
    read-only persistent bitmaps are still in the file and in sync with
    their in-memory representation, so we may as well release them just like
    any R/W bitmap that we have updated.
    
    It leads to actual problems, too: After migration, letting the source
    continue may result in an error if there were any bitmaps on read-only
    nodes (such as backing images), because those have not been released by
    bdrv_inactive_all(), but bdrv_invalidate_cache_all() attempts to reload
    them (which fails, because they are still present in memory).
    
    Signed-off-by: Max Reitz <mreitz@redhat.com>
    Message-Id: <20200730120234.49288-2-mreitz@redhat.com>
    Tested-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>
    Signed-off-by: Eric Blake <eblake@redhat.com>

Comment 14 John Ferlan 2020-08-19 14:04:05 UTC
Danilo - yet another bug that is fixed by the rebase...  Needinfo for you to update data into ON_QA

Comment 15 Danilo Cesar Lemes de Paula 2020-08-20 02:39:42 UTC
Thanks for the headsup John.

Adding TestOnly and Fixed-in field and moving to ON_QA

@aihua QA_ACK, please?

Comment 16 aihua liang 2020-08-20 11:07:43 UTC
Test on qemu-kvm-5.1.0-3.module+el8.3.0+7708+740a1315, the problem has been resolved, set bug's status to "Verified".

Test steps:
 1.Start guest with qemu cmds:
    ...
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    ...

 2.Add persistent bitmap to it.
    { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}}

 3.Quit vm
  (qemu)quit

 4.Create a snapshot chain
   qemu-img create -f qcow2 -F qcow2 /home/kvm_autotest_root/images/snapshot.qcow2 -b /home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2 20G

 5.Start src with snapshot image
   ...
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
   ...

 6.Add persistent bitmap to it
   { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0","persistent":true}}

 7.Check bitmap info in src
   {"execute":"query-named-block-nodes"}
   {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block113", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block061", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 47718400, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 44761088}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 47120384, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 47718400, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

  8.Start dst guest with -incoming
    ...
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/snapshot.qcow2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -incoming tcp:0:5000 \
    ...
 
  9.Check bitmap info in dst
    {"execute":"query-named-block-nodes"}
{"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block158", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block067", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 493883392, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 493551616, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 493883392, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

 10.Write a file in src guest:
   (guest)dd if=/dev/urandom of=test bs=1M count=400

 11.Check bitmap info in src:
    {"execute":"query-named-block-nodes"}
{"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block113", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block061", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 493883392, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 491257856}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 493551616, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 493883392, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

 12.Migrate from src to dst
    {"execute":"migrate","arguments":{"uri":"tcp:0:5000"}}
{"return": {}}
{"timestamp": {"seconds": 1597920364, "microseconds": 508824}, "event": "STOP"}

 13.Check bitmap info in dst
    {"timestamp": {"seconds": 1597920364, "microseconds": 541802}, "event": "RESUME"}

{"execute":"query-named-block-nodes"}
{"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block158", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block067", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 493883392, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 491257856}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 493686784, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 493883392, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

 14.Check bitmap info in src after migration
  {"execute":"query-named-block-nodes"}
  {"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block115", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block092", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 44044288, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 42405888, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 44044288, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}   
 
 15.Quit vm in dst
   (qemu)quit

 16.Cont vm in src
   (qemu)cont
   {"timestamp": {"seconds": 1597920494, "microseconds": 455749}, "event": "RESUME"}

 17.Check bitmap info in src
    {"execute":"query-named-block-nodes"}
{"return": [{"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block113", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1835008}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 10345517056, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "format": "file", "actual-size": 10345398272, "dirty-flag": false}, "iops_wr": 0, "ro": true, "node-name": "#block061", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 10345398272, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 21474836480, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 808456192, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "bitmaps": [{"flags": ["in-use", "auto"], "name": "bitmap0", "granularity": 65536}], "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "backing-filename": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 1, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 805175296}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}, {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 807534592, "filename": "/home/kvm_autotest_root/images/snapshot.qcow2", "format": "file", "actual-size": 808456192, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "file_image1", "backing_file_depth": 0, "drv": "file", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/kvm_autotest_root/images/snapshot.qcow2", "encryption_key_missing": false}]}

And after step17, can still run migration from src to other dst.

Comment 17 aihua liang 2020-08-20 11:19:02 UTC
Won't cover it, for it's testonly.

Comment 18 aihua liang 2020-08-21 09:16:54 UTC
Because it has code change, so will cover this case.


Note You need to log in before you can comment on or make changes to this bug.