Bug 1857735 - QEMU command 'transaction': Bitmap already exists: libvirt-tmp-activewrite
Summary: QEMU command 'transaction': Bitmap already exists: libvirt-tmp-activewrite
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.3
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
Depends On:
Blocks: 1857347 1857779 1861681
TreeView+ depends on / blocked
 
Reported: 2020-07-16 13:04 UTC by mlehrer
Modified: 2020-11-17 17:50 UTC (History)
12 users (show)

Fixed In Version: libvirt-6.6.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1857779 1861681 (view as bug list)
Environment:
Last Closed: 2020-11-17 17:50:17 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
libvirt vdsm and engine logs. (3.81 MB, application/gzip)
2020-07-16 13:04 UTC, mlehrer
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 5375411 0 None None None 2020-09-07 03:29:21 UTC

Description mlehrer 2020-07-16 13:04:28 UTC
Created attachment 1701387 [details]
libvirt vdsm and engine logs.

Description of problem:
When using rhv 4.4 to create live snapshot of vm, it succeeds twice but fails during vm snapshot removal on the third time.

VDSM does not handle the exception thrown by libvirt.  This issue of handling exceptions for vdsm is addressed in: https://bugzilla.redhat.com/show_bug.cgi?id=1857347  

This bug a placeholder for the issue described in:
 https://bugzilla.redhat.com/show_bug.cgi?id=1857347#c8



Version-Release number of selected component (if applicable):
libvirt 6.0.0-25
rhv-release-4.4.1-10-001.noarch
vdsm-4.40.22-1.el8ev.x86_64
4.18.0-193.14.1.el8_2.x86_64 rhel 8.2

How reproducible:
100% 

Steps to Reproduce:
1. Live VM Snapshot
2. Removal of VM snapshot
3. Repeat twice
On the third time exception occurs.

see attached logs

Actual results:
exceptions are in logs

Expected results:
Exceptions to be handled.


Additional info:

Comment 1 Peter Krempa 2020-07-16 13:09:22 UTC
Libvirt accidentally doesn't remove the temporary bitmap after it's no longer needed when bitmaps are not present in the source image. This makes it impossible to complete any active layer block commits into the broken bitmap until the VM is restarted.

Comment 4 Peter Krempa 2020-07-16 14:38:59 UTC
Patches proposed upstream: https://www.redhat.com/archives/libvir-list/2020-July/msg01221.html

Comment 5 yisun 2020-07-17 03:48:05 UTC
reproduced with:
libvirt-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64

[root@dell-per730-61 ~]# virsh snapshot-create-as vm2 snap1 --disk-only
Domain snapshot snap1 created

[root@dell-per730-61 ~]# virsh blockcommit vm2 vda --active --pivot
Successfully pivoted

[root@dell-per730-61 ~]# virsh snapshot-create-as vm2 snap2 --disk-only
Domain snapshot snap2 created

[root@dell-per730-61 ~]# virsh blockcommit vm2 vda --active --pivot
error: failed to pivot job for disk vda
error: internal error: unable to execute QEMU command 'transaction': Bitmap already exists: libvirt-tmp-activewrite

Comment 7 Peter Krempa 2020-07-21 09:34:48 UTC
Fixed upstream by:

commit 66dc4992fa8a51c4e774d32bcf75f26b1365e998 
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Thu Jul 16 15:14:40 2020 +0200

    qemuDomainBlockPivot: Ignore failures of creating active layer bitmap
    
    Ignore errors from creating "libvirt-tmp-activewrite" bitmap. This
    prevents failures of finishing blockjobs if the bitmap already exists.
    
    Note that if the bitmap exists, the worst case that can happen is that
    more bits are marked as dirty in the resulting merge.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

commit 24ac1a7c04fded2f8639478abead72bb57506162
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Thu Jul 16 15:13:17 2020 +0200

    qemuDomainBlockPivot: Rename 'actions' to 'bitmapactions'
    
    There are two possible 'transaction' command arguments in the function.
    Rename 'actions' as they deal with creating bitmaps only.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

commit 9d039863e2cae227cc73b7a886b40be5e5218a12
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Thu Jul 16 14:46:43 2020 +0200

    qemu: block: Remove 'active-write' bitmap even if there are no bitmaps to merge
    
    The 'libvirt-tmp-activewrite' bitmap is added during the 'pivot'
    operation of block copy and active layer block commit operations
    regardless of whether there are any bitmaps to merge, but was not
    removed unless a bitmap was merged. This meant that subsequent attempts
    to merge into the same image would fail.
    
    Fix it by checking whether the 'libvirt-tmp-activewrite' would be used
    by the code and don't skip the code which would delete it.
    
    This is a regression introduced when we switched to the new code for
    block commit in <20a7abc2d2d> and for block copy in <7bfff40fdfe5>. The
    actual bug originates from <4fa8654ece>.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1857735
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

commit 378e4fbc795c607485799cbd060f34483f580544
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Thu Jul 16 15:19:25 2020 +0200

    qemu: blockjob: Actually delete temporary bitmap on failed active commit
    
    Commit 20a7abc2d2d tried to delete the possibly leftover bitmap but
    neglected to call the actual monitor to do so.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

commit d73f107469c8882a47ffbf1a55bd0b0370d6969e
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Thu Jul 16 15:54:46 2020 +0200

    qemu: blockjob: Don't base bitmap handling of active-layer block commit on QEMU_CAPS_BLOCKDEV_REOPEN
    
    The handler finalizing the active layer block commit doesn't actually
    reopen the file for active layer block commit, so the comment and check
    are invalid.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

Comment 12 yisun 2020-08-11 07:15:07 UTC
Test with: libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64
Result: PASS

Test steps:
1. create external snapshot
2. do blockcommit with --pivot --active
3. delete snapshot metadata
4. delete snapshot image file
5. repeat 1~4, but in step 2, do blockcommit --acitve, then do a blockjob --pivot
6. repeat 1~5 for 5 times

[root@dell-per730-67 ~]# cat test.sh 
vm=$1
for i in {1..5}  
do
      snap_name=snap_$i	
      virsh snapshot-create-as $vm $snap_name --disk-only
      sleep 2
      virsh blockcommit $vm vda --active --pivot
      sleep 2
      virsh snapshot-delete $vm $snap_name --metadata
      rm -f /var/lib/libvirt/images/jeos-27-x86_64.snap_*
      virsh snapshot-create-as $vm $snap_name --disk-only
      virsh blockcommit $vm vda --active
      sleep 2
      virsh blockjob $vm vda --pivot 
      virsh snapshot-delete $vm $snap_name --metadata
      rm -f /var/lib/libvirt/images/jeos-27-x86_64.snap_*
done


[root@dell-per730-67 ~]# sh test.sh vm1
Domain snapshot snap_1 created

Successfully pivoted
Domain snapshot snap_1 deleted

Domain snapshot snap_1 created
Active Block Commit started

Domain snapshot snap_1 deleted

Domain snapshot snap_2 created

Successfully pivoted
Domain snapshot snap_2 deleted

Domain snapshot snap_2 created
Active Block Commit started

Domain snapshot snap_2 deleted

Domain snapshot snap_3 created

Successfully pivoted
Domain snapshot snap_3 deleted

Domain snapshot snap_3 created
Active Block Commit started

Domain snapshot snap_3 deleted

Domain snapshot snap_4 created

Successfully pivoted
Domain snapshot snap_4 deleted

Domain snapshot snap_4 created
Active Block Commit started

Domain snapshot snap_4 deleted

Domain snapshot snap_5 created

Successfully pivoted
Domain snapshot snap_5 deleted

Domain snapshot snap_5 created
Active Block Commit started

Domain snapshot snap_5 deleted

Comment 15 errata-xmlrpc 2020-11-17 17:50:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137


Note You need to log in before you can comment on or make changes to this bug.