Created attachment 1701387 [details] libvirt vdsm and engine logs. Description of problem: When using rhv 4.4 to create live snapshot of vm, it succeeds twice but fails during vm snapshot removal on the third time. VDSM does not handle the exception thrown by libvirt. This issue of handling exceptions for vdsm is addressed in: https://bugzilla.redhat.com/show_bug.cgi?id=1857347 This bug a placeholder for the issue described in: https://bugzilla.redhat.com/show_bug.cgi?id=1857347#c8 Version-Release number of selected component (if applicable): libvirt 6.0.0-25 rhv-release-4.4.1-10-001.noarch vdsm-4.40.22-1.el8ev.x86_64 4.18.0-193.14.1.el8_2.x86_64 rhel 8.2 How reproducible: 100% Steps to Reproduce: 1. Live VM Snapshot 2. Removal of VM snapshot 3. Repeat twice On the third time exception occurs. see attached logs Actual results: exceptions are in logs Expected results: Exceptions to be handled. Additional info:
Libvirt accidentally doesn't remove the temporary bitmap after it's no longer needed when bitmaps are not present in the source image. This makes it impossible to complete any active layer block commits into the broken bitmap until the VM is restarted.
Patches proposed upstream: https://www.redhat.com/archives/libvir-list/2020-July/msg01221.html
reproduced with: libvirt-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64 [root@dell-per730-61 ~]# virsh snapshot-create-as vm2 snap1 --disk-only Domain snapshot snap1 created [root@dell-per730-61 ~]# virsh blockcommit vm2 vda --active --pivot Successfully pivoted [root@dell-per730-61 ~]# virsh snapshot-create-as vm2 snap2 --disk-only Domain snapshot snap2 created [root@dell-per730-61 ~]# virsh blockcommit vm2 vda --active --pivot error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'transaction': Bitmap already exists: libvirt-tmp-activewrite
Fixed upstream by: commit 66dc4992fa8a51c4e774d32bcf75f26b1365e998 Author: Peter Krempa <pkrempa> Date: Thu Jul 16 15:14:40 2020 +0200 qemuDomainBlockPivot: Ignore failures of creating active layer bitmap Ignore errors from creating "libvirt-tmp-activewrite" bitmap. This prevents failures of finishing blockjobs if the bitmap already exists. Note that if the bitmap exists, the worst case that can happen is that more bits are marked as dirty in the resulting merge. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit 24ac1a7c04fded2f8639478abead72bb57506162 Author: Peter Krempa <pkrempa> Date: Thu Jul 16 15:13:17 2020 +0200 qemuDomainBlockPivot: Rename 'actions' to 'bitmapactions' There are two possible 'transaction' command arguments in the function. Rename 'actions' as they deal with creating bitmaps only. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit 9d039863e2cae227cc73b7a886b40be5e5218a12 Author: Peter Krempa <pkrempa> Date: Thu Jul 16 14:46:43 2020 +0200 qemu: block: Remove 'active-write' bitmap even if there are no bitmaps to merge The 'libvirt-tmp-activewrite' bitmap is added during the 'pivot' operation of block copy and active layer block commit operations regardless of whether there are any bitmaps to merge, but was not removed unless a bitmap was merged. This meant that subsequent attempts to merge into the same image would fail. Fix it by checking whether the 'libvirt-tmp-activewrite' would be used by the code and don't skip the code which would delete it. This is a regression introduced when we switched to the new code for block commit in <20a7abc2d2d> and for block copy in <7bfff40fdfe5>. The actual bug originates from <4fa8654ece>. https://bugzilla.redhat.com/show_bug.cgi?id=1857735 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit 378e4fbc795c607485799cbd060f34483f580544 Author: Peter Krempa <pkrempa> Date: Thu Jul 16 15:19:25 2020 +0200 qemu: blockjob: Actually delete temporary bitmap on failed active commit Commit 20a7abc2d2d tried to delete the possibly leftover bitmap but neglected to call the actual monitor to do so. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit d73f107469c8882a47ffbf1a55bd0b0370d6969e Author: Peter Krempa <pkrempa> Date: Thu Jul 16 15:54:46 2020 +0200 qemu: blockjob: Don't base bitmap handling of active-layer block commit on QEMU_CAPS_BLOCKDEV_REOPEN The handler finalizing the active layer block commit doesn't actually reopen the file for active layer block commit, so the comment and check are invalid. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake>
Test with: libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64 Result: PASS Test steps: 1. create external snapshot 2. do blockcommit with --pivot --active 3. delete snapshot metadata 4. delete snapshot image file 5. repeat 1~4, but in step 2, do blockcommit --acitve, then do a blockjob --pivot 6. repeat 1~5 for 5 times [root@dell-per730-67 ~]# cat test.sh vm=$1 for i in {1..5} do snap_name=snap_$i virsh snapshot-create-as $vm $snap_name --disk-only sleep 2 virsh blockcommit $vm vda --active --pivot sleep 2 virsh snapshot-delete $vm $snap_name --metadata rm -f /var/lib/libvirt/images/jeos-27-x86_64.snap_* virsh snapshot-create-as $vm $snap_name --disk-only virsh blockcommit $vm vda --active sleep 2 virsh blockjob $vm vda --pivot virsh snapshot-delete $vm $snap_name --metadata rm -f /var/lib/libvirt/images/jeos-27-x86_64.snap_* done [root@dell-per730-67 ~]# sh test.sh vm1 Domain snapshot snap_1 created Successfully pivoted Domain snapshot snap_1 deleted Domain snapshot snap_1 created Active Block Commit started Domain snapshot snap_1 deleted Domain snapshot snap_2 created Successfully pivoted Domain snapshot snap_2 deleted Domain snapshot snap_2 created Active Block Commit started Domain snapshot snap_2 deleted Domain snapshot snap_3 created Successfully pivoted Domain snapshot snap_3 deleted Domain snapshot snap_3 created Active Block Commit started Domain snapshot snap_3 deleted Domain snapshot snap_4 created Successfully pivoted Domain snapshot snap_4 deleted Domain snapshot snap_4 created Active Block Commit started Domain snapshot snap_4 deleted Domain snapshot snap_5 created Successfully pivoted Domain snapshot snap_5 deleted Domain snapshot snap_5 created Active Block Commit started Domain snapshot snap_5 deleted
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137