Bug 1790492

Summary: 'dirty-bitmaps' migration capability should allow configuring target nodenames
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Peter Krempa <pkrempa>
Component: qemu-kvmAssignee: Hanna Czenczek <hreitz>
qemu-kvm sub component: Incremental Live Backup QA Contact: aihua liang <aliang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: coli, jinzhao, jsuchane, juzhang, ngu, nsoffer, virt-maint, ymankad
Version: 8.1Keywords: FutureFeature, Triaged
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 17:46:36 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1867085, 1867086    

Description Peter Krempa 2020-01-13 13:46:49 UTC 
Description of problem:
Based on discussion on qemu-devel list it's required that the 'dirty-bitmaps' migration capability is made configurable or at least does not use node-names for matching the source and destination image to copy the bitmaps to.

The rationale (described in the linked thread) is that bitmaps track changes in guest visible data so they don't really map to node-names well, in cases when
the backing chain is copied or modified at migration while keeping guest-visible state identical.

Additionally node-names are not-stable accross migration as used by libvirt.

Most of the points for this were raised in the following thread:

https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00990.html
https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00646.html
https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00601.html
Comment 1 Ademar Reis 2020-02-05 23:12:34 UTC 
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks
Comment 8 aihua liang 2020-09-10 09:18:25 UTC 
Test on qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f, bug has been fixed.

Test Env:
   kernel version:4.18.0-233.el8.x86_64
   qemu-kvm version:qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f


Scenario 1: mapping node-name and bitmap in src.
Test Steps:
  1.In src, start guest with qemu cmds:
     ...
     -blockdev node-name=file_image1,driver=file,aio=threads,filename=/mnt/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
     -device virtio-blk-pci,id=image1,drive=drive_image1,write-cache=on,bus=pcie-root-port-2,iothread=iothread0 \
     ...

  2.In dst. start guest with qemu cmds:
    ...
     -blockdev node-name=file_img1,driver=file,aio=threads,filename=/home/mirror.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_img1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_img1 \
     -device virtio-blk-pci,id=img1,drive=drive_img1,write-cache=on,bus=pcie-root-port-2,iothread=iothread0 \
     -incoming tcp:0:5000 \
     ...

  3.Expose dst image
    { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host": "10.66.144.75", "port": "3333" } } } }
    { "execute": "nbd-server-add", "arguments":{ "device": "drive_img1", "writable": true } }

  4.Do mirror from src to dst
    {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror","server":{"type":"inet","host":"10.66.144.75","port":"3333"},"export":"drive_img1"}}
    {"execute": "blockdev-mirror", "arguments": { "device": "drive_image1","target": "mirror", "sync": "full", "job-id":"j1"}}
{"timestamp": {"seconds": 1599725553, "microseconds": 761010}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
{"timestamp": {"seconds": 1599725553, "microseconds": 761175}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
{"return": {}}
{"timestamp": {"seconds": 1599725630, "microseconds": 313251}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "j1"}}
{"timestamp": {"seconds": 1599725630, "microseconds": 313352}, "event": "BLOCK_JOB_READY", "data": {"device": "j1", "len": 21475033088, "offset": 21475033088, "speed": 0, "type": "mirror"}}

  5.Set migration capabilities in both src and dst.
    {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}}

  6.Add bitmap to src image and write file in src guest and check bitmap info
    { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}}
    (src guest)#dd if=/dev/urandom of=test bs=1M count=1000
    {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/mnt/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 6699487232, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}, {"recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/mnt/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/image1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}, {"recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "type": "unknown"}]}

  7.Set pre-switchover in src and do migration from src to dst
    {"execute":"migrate-continue","arguments":{"state":"pre-switchover"}}
    {"execute":"migrate-set-parameters","arguments":{"block-bitmap-mapping":[{"node-name":"drive_image1","alias":"drive_img1","bitmaps":[{"name":"bitmap0","alias":"bitmapa"}]}]}}
    {"execute": "migrate","arguments":{"uri": "tcp:10.66.144.75:5000"}}
{"timestamp": {"seconds": 1599726335, "microseconds": 403995}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}}
{"timestamp": {"seconds": 1599726335, "microseconds": 413324}, "event": "MIGRATION_PASS", "data": {"pass": 1}}
{"timestamp": {"seconds": 1599726335, "microseconds": 413582}, "event": "MIGRATION", "data": {"status": "active"}}
{"timestamp": {"seconds": 1599726405, "microseconds": 878064}, "event": "MIGRATION_PASS", "data": {"pass": 2}}
{"timestamp": {"seconds": 1599726407, "microseconds": 180523}, "event": "MIGRATION_PASS", "data": {"pass": 3}}
{"timestamp": {"seconds": 1599726407, "microseconds": 480314}, "event": "MIGRATION_PASS", "data": {"pass": 4}}
{"timestamp": {"seconds": 1599726407, "microseconds": 486676}, "event": "STOP"}
{"timestamp": {"seconds": 1599726407, "microseconds": 488527}, "event": "MIGRATION", "data": {"status": "pre-switchover"}}

  8.Migrate continue
    {"execute":"migrate-continue","arguments":{"state":"pre-switchover"}}
{"return": {}}
{"timestamp": {"seconds": 1599726428, "microseconds": 819726}, "event": "MIGRATION", "data": {"status": "device"}}
{"timestamp": {"seconds": 1599726428, "microseconds": 820726}, "event": "MIGRATION_PASS", "data": {"pass": 5}}
{"timestamp": {"seconds": 1599726428, "microseconds": 830600}, "event": "MIGRATION", "data": {"status": "completed"}}

  9.Check status in dst
    (qemu)info status
    VM status: running

  10.Check bitmap status in dst
     {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/mirror.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 6701453312, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_img1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmapa", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 319488000}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/mirror.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/img1/virtio-backend", "dirty-bitmaps": [{"name": "bitmapa", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 319488000}], "type": "unknown"}]}


Scenario 2: mapping node-name and bitmap in dst.
Test Steps：
  1.In src, start guest with qemu cmds:
     ...
     -blockdev node-name=file_image1,driver=file,aio=threads,filename=/mnt/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
     -device virtio-blk-pci,id=image1,drive=drive_image1,write-cache=on,bus=pcie-root-port-2,iothread=iothread0 \
     ...

  2.In dst. start guest with qemu cmds:
    ...
     -blockdev node-name=file_img1,driver=file,aio=threads,filename=/home/mirror.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_img1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_img1 \
     -device virtio-blk-pci,id=img1,drive=drive_img1,write-cache=on,bus=pcie-root-port-2,iothread=iothread0 \
     -incoming tcp:0:5000 \
     ...

  3.Expose dst image
    { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host": "10.66.144.75", "port": "3333" } } } }
    { "execute": "nbd-server-add", "arguments":{ "device": "drive_img1", "writable": true } }

  4.Do mirror from src to dst
    {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror","server":{"type":"inet","host":"10.66.144.75","port":"3333"},"export":"drive_img1"}}
    {"execute": "blockdev-mirror", "arguments": { "device": "drive_image1","target": "mirror", "sync": "full", "job-id":"j1"}}
{"timestamp": {"seconds": 1599725553, "microseconds": 761010}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
{"timestamp": {"seconds": 1599725553, "microseconds": 761175}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
{"return": {}}
{"timestamp": {"seconds": 1599725630, "microseconds": 313251}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "j1"}}
{"timestamp": {"seconds": 1599725630, "microseconds": 313352}, "event": "BLOCK_JOB_READY", "data": {"device": "j1", "len": 21475033088, "offset": 21475033088, "speed": 0, "type": "mirror"}}

  5.Set migration capabilities in both src and dst.
    {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}}

  6.Add bitmap to src image and write file in src guest and check bitmap info
    { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}}
    (src guest)#dd if=/dev/urandom of=test bs=1M count=1000
    {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/mnt/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 6699487232, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}, {"recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/mnt/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/image1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}, {"recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "type": "unknown"}]}

  7.In dst, set bitmap mapping
    {"execute":"migrate-set-parameters","arguments":{"block-bitmap-mapping":[{"node-name":"drive_image1","alias":"drive_img1","bitmaps":[{"name":"bitmap0","alias":"bitmapa"}]}]}}

  8.Set pre-switchover in src and do migration from src to dst
    {"execute":"migrate-continue","arguments":{"state":"pre-switchover"}}
    {"execute": "migrate","arguments":{"uri": "tcp:10.66.144.75:5000"}}
{"timestamp": {"seconds": 1599726335, "microseconds": 403995}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}}
{"timestamp": {"seconds": 1599726335, "microseconds": 413324}, "event": "MIGRATION_PASS", "data": {"pass": 1}}
{"timestamp": {"seconds": 1599726335, "microseconds": 413582}, "event": "MIGRATION", "data": {"status": "active"}}
{"timestamp": {"seconds": 1599726405, "microseconds": 878064}, "event": "MIGRATION_PASS", "data": {"pass": 2}}
{"timestamp": {"seconds": 1599726407, "microseconds": 180523}, "event": "MIGRATION_PASS", "data": {"pass": 3}}
{"timestamp": {"seconds": 1599726407, "microseconds": 480314}, "event": "MIGRATION_PASS", "data": {"pass": 4}}
{"timestamp": {"seconds": 1599726407, "microseconds": 486676}, "event": "STOP"}
{"timestamp": {"seconds": 1599726407, "microseconds": 488527}, "event": "MIGRATION", "data": {"status": "pre-switchover"}}

  9.Migrate continue
    {"execute":"migrate-continue","arguments":{"state":"pre-switchover"}}
{"return": {}}
{"timestamp": {"seconds": 1599726428, "microseconds": 819726}, "event": "MIGRATION", "data": {"status": "device"}}
{"timestamp": {"seconds": 1599726428, "microseconds": 820726}, "event": "MIGRATION_PASS", "data": {"pass": 5}}
{"timestamp": {"seconds": 1599726428, "microseconds": 830600}, "event": "MIGRATION", "data": {"status": "completed"}}

  10.Check status in dst
    (qemu)info status
    VM status: running

  11.Check bitmap status in dst
     {"execute":"query-block"}
{"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/home/mirror.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 6701453312, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_img1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmapa", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 319488000}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/home/mirror.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/img1/virtio-backend", "dirty-bitmaps": [{"name": "bitmapa", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 319488000}], "type": "unknown"}]}
Comment 9 aihua liang 2020-09-10 10:11:51 UTC 
The "block-bitmap-mapping" works ok except an coredump issue, has tracked it by a new bug: bz1877724, so set this bug to "Verified".
Comment 12 errata-xmlrpc 2020-11-17 17:46:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137