Created attachment 1769034 [details] Engine VDSM libvirt and QEMU logs Description of problem: qemu-img convert command failed with the following error when used on a volume with bitmap that was taken during a backup operation. From host-0 VDSM log - qemu-img: Failed to populate bitmap 5f59b2d6-6b52-484c-ae7a-f8b43f2175a4: Bitmap \'5f59b2d6-6b52-484c-ae7a-f8b43f2175a4\' is inconsistent and cannot be used\nTry block-dirty-bitmap-remove to delete this bitmap from disk" Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Createa a VM and start a full and incremental backup. 2. Kill the QEMU process and start it again 3. Move the VM disk to another storage domain (using QEMU convert) Actual results: Disk migration failed due to inconsistent bitmap Expected results: Disk should migrated withou any issues Additional info:
Hi, Eric According to your mail reply: << It makes no sense to copy an inconsistent bitmap. Thus, live migration << and qemuing covert are both probably going to fail because of the << inability to copy a corrupt bitmap, and the solution is to delete the << bad bitmap before attempting any other operation that wants to copy << bitmaps. The only reason qemu does not automatically ignore/delete the << inconsistent bitmap is because we want to ensure the management app is << aware of the bitmap loss; the management app acknowledges it by deleting << the bad bitmap. So can you help to make a double confirmation? Then I can close it. Thanks, Aliang
Correct - the only recovery method is for the user to manually delete the broken bitmap (qemu-img bitmap --remove file.qcow2 broken_name) before migrating the file.
This means we have to check if all bitmaps are valid before the copy, and remove invalid bitmaps. It will be more useful if qemu-img covert could skip invalid bitmaps during copy. Bitmaps are only an optimization. If we don't copy a bitmap the next incremental backup will fail and we can fallback to full backup. The current situation is that entire copy fails, and recovery requires manual support.
(In reply to Nir Soffer from comment #4) > This means we have to check if all bitmaps are valid before the copy, and > remove > invalid bitmaps. > > It will be more useful if qemu-img covert could skip invalid bitmaps during > copy. > Bitmaps are only an optimization. If we don't copy a bitmap the next > incremental > backup will fail and we can fallback to full backup. > > The current situation is that entire copy fails, and recovery requires manual > support. Deleting broken bitmaps by default may be risky, but copying all working bitmaps while removing the broken ones via a command line option (maybe --skip-broken, borrowing naming from dnf) seems like it might be a nice UI addition.
(In reply to Eric Blake from comment #5) > Deleting broken bitmaps by default may be risky, but copying all working > bitmaps while removing the broken ones via a command line option (maybe > --skip-broken, borrowing naming from dnf) seems like it might be a nice UI > addition. I agree. Can get this in 8.5?
(In reply to Nir Soffer from comment #6) > (In reply to Eric Blake from comment #5) > > Deleting broken bitmaps by default may be risky, but copying all working > > bitmaps while removing the broken ones via a command line option (maybe > > --skip-broken, borrowing naming from dnf) seems like it might be a nice UI > > addition. > > I agree. Can get this in 8.5? Any updates here? I took the liberty of adjusting the summary as well as reclassifying this bug to be of low severity, low priority - let me know if there's disagreement.
(In reply to Klaus Heinrich Kiwi from comment #7) > reclassifying this bug to be of low severity, low priority - let me know if > there's disagreement. This is a regression that causes failures in basic flows in RHV. After a VM is terminated abnormally, it will have bitmaps with 'in-use' flag. Copying the vm disks to other storage will fail because of the invalid bitmaps, and there is no way to fix this without deleting the bitmaps manually, which means support case. Before adding the feature to copy bitmaps, abnormal termination did not cause any failures when copying the disks because qemu-img did not copy any bitmaps. So I don't think this is a low severity. Regarding priority, we can fix this in RHV by checking and deleting bitmaps before copying disks, but this not an easy change in RHV. So the right way to fix this seems to be adding an option to skip broken bitmaps in qemu-img convert, which RHV will always use. Adding another option to the qemu-img convert command is trivial change. I also assume that this is relatively easy to fix in qemu, a minor change in the code copying bitmaps to skip bitmaps with in-use flag. Fixing this in qemu also means that this issue will be fixed for all users, instead only for RHV. So from my point of view, it is high priority that this will be fixed in qemu and not in RHV.
(In reply to Klaus Heinrich Kiwi from comment #7) I adjusted the summary again since "improve options and defaults" does not reflect the issue well. This indeed sounds like low priority thing when the issue is failing to copy a disk.
More info - we need to support copying disks used by a vm, for example during live storage migration. In this flow we create a snapshot, mirror the snapshot to the destination disk using blockCopy job, and copy the rest of the layers using qemu-img convert. If qemu-img convert fail because a layer includes an inconsistent bitmap, the entire live storage migration will fail. Removing the bitmap before the copy requires modifying images *used* by qemu using "qemu-img bitmap --remove". This is likey not possible, and even it is possible it sounds like a really bad idea. So this must be handled by qemu-img convert.
Patch posted for upstream discussion: https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg01731.html
(In reply to Nir Soffer from comment #8) > (In reply to Klaus Heinrich Kiwi from comment #7) > > reclassifying this bug to be of low severity, low priority - let me know if > > there's disagreement. > > This is a regression that causes failures in basic flows in RHV. > > After a VM is terminated abnormally, it will have bitmaps with 'in-use' flag. > Copying the vm disks to other storage will fail because of the invalid > bitmaps, > and there is no way to fix this without deleting the bitmaps manually, which > means support case. > > Before adding the feature to copy bitmaps, abnormal termination did not cause > any failures when copying the disks because qemu-img did not copy any > bitmaps. > > So I don't think this is a low severity. > <... snip ...> > > So from my point of view, it is high priority that this will be fixed in > qemu and not in RHV. Thanks for the clarification. If this was an interface that was exposed / supported by users / LPs and is not behaving in a way that (now) can crash, I am convinced this is a high-priority regression. Luckily, seems like the upstream work is progressing.
Upstream pull request for 6.1-rc1 https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg05705.html
(In reply to Klaus Heinrich Kiwi from comment #13) Is this possible to get the fix in 8.4.z? The fix affects only qemu-img convert when using the --bitmaps option. So testing is limited to qemu-img tool.
(In reply to Nir Soffer from comment #15) > (In reply to Klaus Heinrich Kiwi from comment #13) > Is this possible to get the fix in 8.4.z? > > The fix affects only qemu-img convert when using the --bitmaps option. So > testing > is limited to qemu-img tool. I generally agree with bringing it to 8.4.z since customers could be affected otherwise. I assume the same updates needed on your end for 8.5 would be needed for 8.4.z? As for doability, I'd defer to Eric - If it's not too much (throw-away/risky) work, I'd suggest we backport it.
zstream requested per comment 15 as RHV customer w/ RHEL-AV. Eric notes patches are essentially the same as posted for 8.5.
Can we get a qa_ack+ please. I set ITM=26 since that's about the end of the non exception part of the release and the normal 2 week after DTM date. Bug has 1 more downstream ack, so it should move to on_qa soon.
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Test on qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b, the problem has been resolved. Test Env: Kernel version:4.18.0-330.el8.x86_64 qemu-kvm version:qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b Test Steps: 1.Create a qcow2 image #qemu-img create -f qcow2 /home/data.qcow2 2G 2.Add a persistent bitmap offline #qemu-img bitmap /home/data.qcow2 --add bitmap_persistent 3.Expose the persistent bitmap via qemu-nbd #qemu-nbd -t -p 10098 /home/data.qcow2 -B bitmap_persistent 4.Kill qemu-nbd #kill -9 $pid_qemu_nbd 5.Check image info of data.qcow2 # qemu-img info /home/data.qcow2 image: /home/data.qcow2 file format: qcow2 virtual size: 2 GiB (2147483648 bytes) disk size: 324 KiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false bitmaps: [0]: flags: [0]: in-use [1]: auto name: bitmap_persistent granularity: 65536 refcount bits: 16 corrupt: false extended l2: false 6.Add a new persistent bitmap to /home/data.qcow2 #qemu-img bitmap /home/data.qcow2 --add bitmap_add 7.Check image info of /home/data.qcow2 #qemu-img info /home/data.qcow2 image: /home/data.qcow2 file format: qcow2 virtual size: 2 GiB (2147483648 bytes) disk size: 452 KiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false bitmaps: [0]: flags: [0]: in-use [1]: auto name: bitmap_persistent granularity: 65536 [1]: flags: [0]: auto name: bitmap_add granularity: 65536 refcount bits: 16 corrupt: false extended l2: false 8.Convert image with all bitmaps #qemu-img convert -O qcow2 /home/data.qcow2 /home/test/data_new.qcow2 --bitmaps qemu-img: Cannot copy inconsistent bitmap 'bitmap_persistent' Try --skip-broken-bitmaps, or use 'qemu-img bitmap --remove' to delete it 9.Convert image with inconsistent bitmap skipped. #qemu-img convert -O qcow2 /home/data.qcow2 /home/test/data_new.qcow2 --bitmaps --skip-broken-bitmaps qemu-img: warning: Skipping inconsistent bitmap 'bitmap_persistent' 10.Check image info of converted image #qemu-img info /home/test/data_new.qcow2 image: /home/test/data_new.qcow2 file format: qcow2 virtual size: 2 GiB (2147483648 bytes) disk size: 324 KiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false bitmaps: [0]: flags: [0]: auto name: bitmap_add granularity: 65536 refcount bits: 16 corrupt: false extended l2: false
As comment 27, set bug's status to "Verified".
(In reply to Klaus Heinrich Kiwi from comment #16) > > As for doability, I'd defer to Eric - If it's not too much > (throw-away/risky) work, I'd suggest we backport it. z-stream was approved, clearing the needinfo flag
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4684