Bug 1780705 - [blockdev enablement] ‘block commit’ failed when vm having multiple external snapshots
Summary: [blockdev enablement] ‘block commit’ failed when vm having multiple external ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
: 1780713 (view as bug list)
Depends On: 1763937
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-06 16:57 UTC by yisun
Modified: 2021-01-08 16:53 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-08 16:53:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
libvirtd.log (2.65 MB, text/plain)
2019-12-06 16:57 UTC, yisun
no flags Details

Description yisun 2019-12-06 16:57:55 UTC
Created attachment 1642672 [details]
libvirtd.log

Decription: [blockdev enablement] ‘block commit’ failed when vm having multiple external snapshots

Versions:
libvirt-5.10.0-1.module+el8.2.0+5040+bd433686.x86_64
qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64

How reproducible:
100%

Steps
1. Having a running vm 
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

2. Clear libvirtd log
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# echo "" > /var/log/libvirtd-debug.log

3. Create 4 external snapshots for vm
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# for i in {1..4}; do virsh snapshot-create-as avocado-vt-vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# virsh dumpxml avocado-vt-vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap4' index='5'/>
      <backingStore type='file' index='4'>
        <format type='qcow2'/>
        <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap3'/>
        <backingStore type='file' index='3'>
          <format type='qcow2'/>
          <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap2'/>
          <backingStore type='file' index='2'>
            <format type='qcow2'/>
            <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap1'/>
            <backingStore type='file' index='1'>
              <format type='qcow2'/>
              <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2'/>
              <backingStore/>
            </backingStore>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

4. Try to do blockcommit from top to base, error happens
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# virsh blockcommit avocado-vt-vm1 vda --wait --verbose --base vda[1] --active --pivot
error: internal error: unable to execute QEMU command 'block-commit': 'libvirt-1-format' is not in this backing file chain

5. Same error happens when do block commit from middle to middle, such as:
# virsh blockcommit avocado-vt-vm1 vda --wait --verbose --base vda[2] —top vda[3]

Expected result:
Block commit should be successful

Additional info:
Libvirtd debug log as atachment

Comment 2 Peter Krempa 2019-12-07 09:39:09 UTC
This is a manifestation of the qemu bug https://bugzilla.redhat.com/show_bug.cgi?id=1763937

The qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64 package is based on qemu-4.2-rc1 source and does NOT contain commit v4.2.0-rc1-18-gae0f57f0aa (which was pushed 18 commits after rc1):

commit ae0f57f0aa03571045a3accc0b22d91a4775e63c
Author: Kevin Wolf <kwolf>
Date:   Fri Nov 8 09:36:35 2019 +0100

    block: Remove 'backing': null from bs->{explicit_,}options
    
    bs->options and bs->explicit_options shouldn't contain any options for
    child nodes. bdrv_open_inherited() takes care to remove any options that
    match a child name after opening the image and the same is done when
    reopening.
    
    However, we miss the case of 'backing': null, which is a child option,
    but results in no child being created. This means that a 'backing': null
    remains in bs->options and bs->explicit_options.
    
    A typical use for 'backing': null is in live snapshots: blockdev-add for
    the qcow2 overlay makes sure not to open the backing file (because it is
    already opened and blockdev-snapshot will attach it). After doing a
    blockdev-snapshot, bs->options and bs->explicit_options become
    inconsistent with the actual state (bs has a backing file now, but the
    options still say null). On the next occasion that the image is
    reopened, e.g. switching it from read-write to read-only when another
    snapshot is taken, the option will take effect again and the node
    incorrectly loses its backing file.
    
    Fix bdrv_open_inherited() to remove the 'backing' option from
    bs->options and bs->explicit_options even for the case where it
    specifies that no backing file is wanted.
    
    Reported-by: Peter Krempa <pkrempa>
    Signed-off-by: Kevin Wolf <kwolf>
    Reviewed-by: Alberto Garcia <berto>
    Tested-by: Peter Krempa <pkrempa>

Please re-test it with appropriate qemu version.

An alternative test workaround is to shutdown/destroy the VM fully after creating the snapshots in step 3 and re-starting it as that fixes qemu's internal state. After that the blockjobs shoudl work.

I'll keep this BZ open until it's tested. Setting test-only and moving to QA since there's nothing to do from libvirt's PoV

Comment 3 Peter Krempa 2019-12-07 09:39:54 UTC
*** Bug 1780713 has been marked as a duplicate of this bug. ***

Comment 4 yisun 2020-02-20 04:29:33 UTC
Test result: PASS
[root@hp-dl320eg8-05 ~]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-4"
qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64
libvirt-6.0.0-5.module+el8.2.0+5765+64816f89.x86_64

[root@hp-dl320eg8-05 ~]# virsh list
 Id   Name   State
----------------------
 1    gls    running
 3    vm1    running

[root@hp-dl320eg8-05 ~]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created


[root@hp-dl320eg8-05 ~]# virsh blockcommit vm1 vda --wait --verbose --base vda[1] --active --pivot
Block commit: [100 %]
Successfully pivoted

Comment 6 Jeff Nelson 2021-01-08 16:53:32 UTC
Changing this TestOnly BZ as CLOSED CURRENTRELEASE. Please reopen if the issue is not resolved.


Note You need to log in before you can comment on or make changes to this bug.