Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1780705

Summary: [blockdev enablement] ‘block commit’ failed when vm having multiple external snapshots
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: yisun
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED CURRENTRELEASE QA Contact: yisun
Severity: high Docs Contact:
Priority: high    
Version: 8.2CC: hhan, lmen, pkrempa, smitterl, xuzhang, yisun
Target Milestone: rcKeywords: Automation, TestOnly
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-08 16:53:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1763937    
Bug Blocks:    
Attachments:
Description Flags
libvirtd.log none

Description yisun 2019-12-06 16:57:55 UTC
Created attachment 1642672 [details]
libvirtd.log

Decription: [blockdev enablement] ‘block commit’ failed when vm having multiple external snapshots

Versions:
libvirt-5.10.0-1.module+el8.2.0+5040+bd433686.x86_64
qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64

How reproducible:
100%

Steps
1. Having a running vm 
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

2. Clear libvirtd log
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# echo "" > /var/log/libvirtd-debug.log

3. Create 4 external snapshots for vm
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# for i in {1..4}; do virsh snapshot-create-as avocado-vt-vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# virsh dumpxml avocado-vt-vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap4' index='5'/>
      <backingStore type='file' index='4'>
        <format type='qcow2'/>
        <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap3'/>
        <backingStore type='file' index='3'>
          <format type='qcow2'/>
          <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap2'/>
          <backingStore type='file' index='2'>
            <format type='qcow2'/>
            <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap1'/>
            <backingStore type='file' index='1'>
              <format type='qcow2'/>
              <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2'/>
              <backingStore/>
            </backingStore>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

4. Try to do blockcommit from top to base, error happens
(.libvirt-ci-venv-ci-runtest-jUzTYn) [root@libvirt-rhel-8 ~]# virsh blockcommit avocado-vt-vm1 vda --wait --verbose --base vda[1] --active --pivot
error: internal error: unable to execute QEMU command 'block-commit': 'libvirt-1-format' is not in this backing file chain

5. Same error happens when do block commit from middle to middle, such as:
# virsh blockcommit avocado-vt-vm1 vda --wait --verbose --base vda[2] —top vda[3]

Expected result:
Block commit should be successful

Additional info:
Libvirtd debug log as atachment

Comment 2 Peter Krempa 2019-12-07 09:39:09 UTC
This is a manifestation of the qemu bug https://bugzilla.redhat.com/show_bug.cgi?id=1763937

The qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64 package is based on qemu-4.2-rc1 source and does NOT contain commit v4.2.0-rc1-18-gae0f57f0aa (which was pushed 18 commits after rc1):

commit ae0f57f0aa03571045a3accc0b22d91a4775e63c
Author: Kevin Wolf <kwolf>
Date:   Fri Nov 8 09:36:35 2019 +0100

    block: Remove 'backing': null from bs->{explicit_,}options
    
    bs->options and bs->explicit_options shouldn't contain any options for
    child nodes. bdrv_open_inherited() takes care to remove any options that
    match a child name after opening the image and the same is done when
    reopening.
    
    However, we miss the case of 'backing': null, which is a child option,
    but results in no child being created. This means that a 'backing': null
    remains in bs->options and bs->explicit_options.
    
    A typical use for 'backing': null is in live snapshots: blockdev-add for
    the qcow2 overlay makes sure not to open the backing file (because it is
    already opened and blockdev-snapshot will attach it). After doing a
    blockdev-snapshot, bs->options and bs->explicit_options become
    inconsistent with the actual state (bs has a backing file now, but the
    options still say null). On the next occasion that the image is
    reopened, e.g. switching it from read-write to read-only when another
    snapshot is taken, the option will take effect again and the node
    incorrectly loses its backing file.
    
    Fix bdrv_open_inherited() to remove the 'backing' option from
    bs->options and bs->explicit_options even for the case where it
    specifies that no backing file is wanted.
    
    Reported-by: Peter Krempa <pkrempa>
    Signed-off-by: Kevin Wolf <kwolf>
    Reviewed-by: Alberto Garcia <berto>
    Tested-by: Peter Krempa <pkrempa>

Please re-test it with appropriate qemu version.

An alternative test workaround is to shutdown/destroy the VM fully after creating the snapshots in step 3 and re-starting it as that fixes qemu's internal state. After that the blockjobs shoudl work.

I'll keep this BZ open until it's tested. Setting test-only and moving to QA since there's nothing to do from libvirt's PoV

Comment 3 Peter Krempa 2019-12-07 09:39:54 UTC
*** Bug 1780713 has been marked as a duplicate of this bug. ***

Comment 4 yisun 2020-02-20 04:29:33 UTC
Test result: PASS
[root@hp-dl320eg8-05 ~]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-4"
qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64
libvirt-6.0.0-5.module+el8.2.0+5765+64816f89.x86_64

[root@hp-dl320eg8-05 ~]# virsh list
 Id   Name   State
----------------------
 1    gls    running
 3    vm1    running

[root@hp-dl320eg8-05 ~]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created


[root@hp-dl320eg8-05 ~]# virsh blockcommit vm1 vda --wait --verbose --base vda[1] --active --pivot
Block commit: [100 %]
Successfully pivoted

Comment 6 Jeff Nelson 2021-01-08 16:53:32 UTC
Changing this TestOnly BZ as CLOSED CURRENTRELEASE. Please reopen if the issue is not resolved.