Description of problem: qemu-img convert does not copy bitmaps from source to destination. Expected results: qemu-img can optionally copy any bitmap data from source to destination, perhaps with an optional flag.
This is required for live storage migration in RHV. When RHV is moving disk from one storage to another while a VM is running. Without this feature, after live storage migration, you will loose all the bitmaps, and the next backup will have to be a full back. Moving disks from storage is not a common operation, so having to do full backup after that is a reasonable limit for initial version of incremental backup if we cannot get this in 8.2. More details how RHV use qemu-img convert in this flow. 1. Create temporary snapshot src storage: base volume <- temporary-snapshot 2. Clone the qecow2 chain on the destination storage src storage: base volume <- temporary-snapshot <<< VM uses this now dst storage: base volume <- temporary-snapshot The volumes on the destination are empty at this point. 3. Start blockCopy() operation, mirroring the the top snapshot on the source storage to the top snapshot on the destination storage: libvirt/qemu: blockCopy: src:temporary-snapshot -> dst:tempoary-snapshost 4. Copy the rest of the chain from source storage to destination using qemu-img convert: qemu-img convert -f qcow2 -O qcow2 /path/to/source/base-volume /path/to/destination/base-volume 5. When copying the rest of the chain finish, and mirroring the temporary snapshot is done, we pivot to the the temporary snapshot on the destination storage. src: base-volume <- temporary-snapshot dst: base-volume <- temporary-snapshot <<< VM uses this now At this point the VM is using the new disk on the destination storage and RHV deletes the old disk on the source storage. 6. Start blockCommit, merging the temporary snapshot into base volume on the destination storage.
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks
Is a blanket 'qemu-img convert --bitmaps' that copies ALL bitmaps from the source to the destination without name change sufficient, or do you need finer-grained control over which bitmaps get copied and/or which destination names are created? The former is adding a simple boolean switch, the latter requires a new 'qemu-img bitmap' command and a lot more effort. Ultimately, I would love upstream to have fine-grained bitmap control via qemu-img, but it's a lot faster to get a short-term patch that just copies bitmaps from source to destination when using 'qemu-img convert' as a fancy 'cp'.
(In reply to Eric Blake from comment #5) > Is a blanket 'qemu-img convert --bitmaps' that copies ALL bitmaps from the > source to the destination without name change sufficient, Yes, and even better not having to specify --bitmaps if possible. If we do add --bitmaps option, we need a way to query if qemu-img supports this option. In the past we the equivalent of: qemu-img --help | egrep "convert.+--bitmaps" To discover similar features. I hope we have a better way now. > or do you need > finer-grained control over which bitmaps get copied and/or which destination > names are created? We must keep the bitmap names in the new image, so we don't need to renaming. Skipping some bitmaps sounds useful but I don't think it worth the effort since we will delete unneeded bitmap during future backup, and bitmaps are small compared with the image data. > The former is adding a simple boolean switch, the latter > requires a new 'qemu-img bitmap' command and a lot more effort. Ultimately, > I would love upstream to have fine-grained bitmap control via qemu-img, but > it's a lot faster to get a short-term patch that just copies bitmaps from > source to destination when using 'qemu-img convert' as a fancy 'cp'. Yes, this sounds like the right way now.
Upstream patch proposed: https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg02611.html
(In reply to Nir Soffer from comment #6) > (In reply to Eric Blake from comment #5) > > Is a blanket 'qemu-img convert --bitmaps' that copies ALL bitmaps from the > > source to the destination without name change sufficient, > > Yes, and even better not having to specify --bitmaps if possible. I just finished writing the v1 patch before seeing this feedback. In v1, adding --bitmaps is mandatory, but if others on list agree, we could change v2 to just always copy all bitmaps (my gut feel, however, is that it will remain an opt-in rather than a default action, because not everyone needs bitmaps copied over). > > If we do add --bitmaps option, we need a way to query if qemu-img supports > this option. In the past we the equivalent of: > > qemu-img --help | egrep "convert.+--bitmaps" > > To discover similar features. I hope we have a better way now. Sadly, qemu-img is not introspectible in any better manner. But this will work (the v1 patch did update the --help output). > > > or do you need > > finer-grained control over which bitmaps get copied and/or which destination > > names are created? > > We must keep the bitmap names in the new image, so we don't need to renaming. Good. The v1 patch just blindly copies all bitmaps from the source to the destination (same name, and preserving whether the bitmap is enabled or disabled). Comments on the upstream patch will be helpful.
Related to bug 1779904.
Related to bug 1780359.
Placing this back into 8.2.1 where it should be for RHEL AV. The most recent patches are still upstream as part of the series: https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg03464.html with the fix being part of patch 5/6: https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg03465.html The expectation is that after a successful review, the patches will be backported, so adding the devel_ack+ even though the patches are not downstream yet.
Aihua, could you kindly grant qa_ack ?
Upstream pull request https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg07419.html
Test on qemu-kvm-4.2.0-24.module+el8.2.1+6959+9b840e7c, the problem has been resolved, set bug's status to "Verified". Test Env: kernel version:4.18.0-193.el8.x86_64 qemu-kvm version:qemu-kvm-4.2.0-24.module+el8.2.1+6959+9b840e7c Test Steps: 1. Start guest with qemu cmds: -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=6 \ -blockdev node-name=file_data1,driver=file,aio=native,filename=/home/data.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_data1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_data1 \ -device virtio-blk-pci,id=data1,drive=drive_data1,write-cache=on,bus=pcie-root-port-4,addr=0x0,iothread=iothread1 \ 2. Add two persistent bitmaps to data disks { "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": { "node": "drive_data1", "name": "bitmap0","disabled":true,"persistent":true } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_data1", "name": "bitmap1", "persistent":true} } ] }} 3. DD file in guest (guest)#mkfs.ext4 /dev/vdb #mount /dev/vdb /mnt #cd /mnt #dd if=/dev/urandom of=test bs=1M count=1000 #md5sum test > sum1 #sync 4. Check bitmap count info ..."file": "/home/data.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/data1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap1", "recording": true, "persistent": true, "busy": false, "status": "active", "granularity": 65536, "count": 1150943232}, {"name": "bitmap0", "recording": false, "persistent": true, "busy": false, "status": "disabled", "granularity": 65536, "count": 0}] 5. Shutdown vm (qemu)system_powerdown 6. Covert image #qemu-img convert -f qcow2 -O qcow2 --bitmap /home/data.qcow2 /home/dst.qcow2 7. Start guest with dst.qcow2 -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=6 \ -blockdev node-name=file_data1,driver=file,aio=native,filename=/home/dst.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_data1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_data1 \ -device virtio-blk-pci,id=data1,drive=drive_data1,write-cache=on,bus=pcie-root-port-4,addr=0x0,iothread=iothread1 \ 8. Check file info in data disk (guest)# mount /dev/vdb /mnt #cd /mnt #cat sum1 #md5sum test After step8, md5sum value of file is the same with that recorded in sum1.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3172