Bug 1798148 - Regression: Requested operation is not valid: format of backing image ... was not specified in the image metadata
Summary: Regression: Requested operation is not valid: format of backing image ... was...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs 1786403 1801219
TreeView+ depends on / blocked
 
Reported: 2020-02-04 16:48 UTC by Richard W.M. Jones
Modified: 2020-05-05 09:58 UTC (History)
11 users (show)

Fixed In Version: libvirt-6.0.0-8.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1801219 (view as bug list)
Environment:
Last Closed: 2020-05-05 09:57:11 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2017 None None None 2020-05-05 09:58:34 UTC

Description Richard W.M. Jones 2020-02-04 16:48:09 UTC
Description of problem:

$ qemu-img create -f qcow2 test.qcow2 1M
$ qemu-img create -f qcow2 overlay.qcow2 -b test.qcow2 
$ qemu-img info overlay.qcow2 
image: overlay.qcow2
file format: qcow2
virtual size: 1 MiB (1048576 bytes)
disk size: 196 KiB
cluster_size: 65536
backing file: test.qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

But if you add overlay.qcow2 to a guest then you can no longer
start the guest:

$ virt-install --import --name test --disk path=overlay.qcw2,format=qcow2 

ERROR    Requested operation is not valid: format of backing image 'test.qcow2' of image '/var/tmp/overlay.qcow2' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)

Note this is a regression over earlier behaviour, and also breaks
libguestfs tools such as virt-sparsify.

Version-Release number of selected component (if applicable):

libvirt-6.0.0-2.module+el8.2.0+5513+34927b6c.x86_64

How reproducible:

100%

Steps to Reproduce:
1. See above.

Comment 2 Peter Krempa 2020-02-05 13:25:47 UTC
Note that the above scenario relied on format detection which was considered unsupported and insecure for a very long time (at least since 2012, I didn't look further).

Since libvirt wasn't able to tell qemu how to open the image until -blockdev was introduced the only recourse was to use selinux to prevent opening any unwanted backing images.

That means that for a very long time, any image which didn't have a format specified was considered to be raw from libvirt's point of view. This 'accidentally' worked in some cases, e.g. when the image didn't have any backing image. Saying 'accidentally' because we didn't have a way to fix it. Additionally in cases when selinux or any other security driver was not available, the insecure format probing could still be exploited.

Since -blockdev support was introduced it became possible to control the backing files of images which libvirt started to do.

In that case it turned out, that actually assuming that a non-specified format equals to RAW which we did for a long time is dangerous as it can lead to data corruption. To avoid any possibility of it we started reporting the error you mention in this bug.

The following bugs tracked some of the problems which occured before:

https://bugzilla.redhat.com/show_bug.cgi?id=1588373
If the image which didn't have the format specified has more than one backing image, selinux will not label them properly and startup will fail.

https://bugzilla.redhat.com/show_bug.cgi?id=1790101
Not specifying the format makes the image look corrupt as the backing image was QCOW2 and libvirt assumed root.

You can also see the questions asked on the upstream list:
https://www.redhat.com/archives/libvir-list/2020-January/msg00320.html

As of the reasons above we never wanted to allow the image format probing done by qemu in the first place but that became possible only using blockdev. As of such the behaviour we have now is expected as we want to require users to certify the image format one way or another.

Note that in addition to recording the format of the image in the overlay file when creating it:

qemu-img create -f qcow2 -F qcow2 -b /path/to/backing.qcow2 /path/to/overlay.qcow2)

(-F specifies the backing file format)

It's now also possible to express the format in the VM XML as well as the fact that backing.qcow2 has no backing store:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/path/to/overlay.qcow2'/>
      <backingStore type='file'>
        <format type='qcow2'/>
        <source file='/path/to/backing.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
    </disk>

(note that you can also assume that if libvirt doesn't support the <backing> store on input, which can be also queried via the domcapabilities api [1] that the elements are ignored and the backing chain is probed)

Any new libvirt will thus reject any probed image which doesn't have the backing store specified and you can use one of the two approaches outlined above to avoid that issue.

[1] https://www.redhat.com/archives/libvirt-users/2020-January/msg00081.html
    https://libvirt.org/formatdomaincaps.html#featureBackingStoreInput

Comment 3 Richard W.M. Jones 2020-02-06 11:06:01 UTC
This change breaks at least virt-customize and virt-sparsify.  I am still
analysing the extent of the problems and how easy it will be to fix.
I don't think that making this change in the middle of RHEL 8 is a good idea,
I think this will negatively affect a lot of customers.

Comment 4 Richard W.M. Jones 2020-02-06 17:21:02 UTC
Easiest workaround seems to be to do the autodetection in libguestfs, so:
https://www.redhat.com/archives/libguestfs/2020-February/msg00013.html

Comment 9 Eric Blake 2020-02-21 12:29:17 UTC
I'm working on a qemu-img patch that will have qemu automatically record the backing format into any just-created image, where it learned the format from probing (note that in some cases, such as 'qemu-img rebase -u', the backing format is not learned).  The patch will also warn any time an implied non-raw format was written into an image (in case the user was expecting raw but forgot to specify it).  But as a qemu patch, we would need to spawn that into a separate BZ.

Comment 10 Peter Krempa 2020-02-25 15:16:47 UTC
After some upstream discussion we've agreed that we can relax the restrictions partially to avoid clients having to reinvent the image probing in a potentially vulnerable way.

Specifically we'll allow situations when the overlay does not specify the format of the backing file if:
- we can inspect the backing image (It's local storage or mapped locally.)
- the backing image does not have any further backing store (is raw or doesn't specify any)

The above means that if the backing format is omitted and the backing file is on remote storage it will be forbidden.

The discussion happened here:
https://www.redhat.com/archives/libvir-list/2020-February/msg00624.html

and the final patches are here:
https://www.redhat.com/archives/libvir-list/2020-February/msg00999.html

Comment 11 Peter Krempa 2020-02-25 20:25:58 UTC
Upstream commits pushed in order to fix the issue (including cleanups):

d552b93448 kbase: backing_chains: Clarify some aspects of image probing
ae9e6c2a2b virStorageFileGetMetadataRecurse: Allow format probing under special circumstances
3c6e6f55a5 qemu: domain: Convert detected 'iso' image format into 'raw'
c95656c995 virStorageFileGetMetadataFromFD: Remove unused 'backingFormat' argument
57df35aead virStorageFileGetMetadataFromBuf: Remove 'backingFormat' argument
fee56942e2 virStorageBackendGlusterRefreshVol: Refactor handling of backing store
264b79c63a virStorageSourceNewFromBacking: Also transfer the format
62539c5f7d util: storage: Store backing store format in virStorageSource
35d1f5bd14 virStorageSourceUpdateCapacity: Drop 'probe' argument
e5c8f6e080 tests: virstorage: Fix backing file format of created image
a570dc6767 virStorageFileGetMetadataRecurse: Remove 'cleanup' label
01adad0932 virStorageFileGetMetadataRecurse: Extract storage access
e3960f4b6d virStorageFileGetMetadataRecurse: Use virHashHasEntry instead of fake pointers
157b8722cb virStorageFileGetMetadataRecurse: Expect NULL src->path
b347e5c7dd virStorageFileGetMetadataRecurse: Shuffle around assignment of backing chain depth
84df98f29e virStorageFileGetMetadataRecurse: Remove impossible error report
181fccc2ed util: storagefile: Drop image format probing by file suffix

Comment 15 yisun 2020-04-01 13:04:19 UTC
test with libvirt-6.0.0-14.module+el8.2.0+6069+78a1cb09.x86_64
And pls note comment 10 and 11, this function is only for 1 layer backing image, and not for remote image.
result is PASS

1. Prepare a vm having a disk with path="/var/lib/libvirt/images/overlay.qcow2", and not adding backingstore info for it

[root@dell-per740xd-11 bz1798148]# virsh domblklist vm1
 Target   Source
-------------------------------------------------
 vda      /var/lib/libvirt/images/overlay.qcow2


[root@dell-per740xd-11 bz1798148]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/overlay.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>


2. Using a iso file as the overlay.qcow2's backing file, it will be detected as RAW
[root@dell-per740xd-11 bz1798148]# mkdir iso_content
[root@dell-per740xd-11 bz1798148]# dd if=/dev/urandom of=iso_content/100M bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.594126 s, 176 MB/s
[root@dell-per740xd-11 bz1798148]#

[root@dell-per740xd-11 bz1798148]# mkisofs -o /var/lib/libvirt/images/base.iso ./iso_content/

[root@dell-per740xd-11 bz1798148]# file base.iso
base.iso: ISO 9660 CD-ROM filesystem data 'CDROM'


[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 -b /var/lib/libvirt/images/base.iso /var/lib/libvirt/images/overlay.qcow2 1G
Formatting '/var/lib/libvirt/images/overlay.qcow2', fmt=qcow2 size=1073741824 backing_file=/var/lib/libvirt/images/base.iso cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@dell-per740xd-11 bz1798148]# virsh start vm1
Domain vm1 started

[root@dell-per740xd-11 bz1798148]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/overlay.qcow2' index='1'/>
      <backingStore type='file' index='2'>
        <format type='raw'/>
        <source file='/var/lib/libvirt/images/base.iso'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>

3. Using a qcow2 file as base image, it will be detected as QCOW2
[root@dell-per740xd-11 bz1798148]# virsh destroy vm1
Domain vm1 destroyed

[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 /var/lib/libvirt/images/base.qcow2 1G
Formatting '/var/lib/libvirt/images/base.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 -b /var/lib/libvirt/images/base.qcow2 /var/lib/libvirt/images/overlay.qcow2 1G
Formatting '/var/lib/libvirt/images/overlay.qcow2', fmt=qcow2 size=1073741824 backing_file=/var/lib/libvirt/images/base.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@dell-per740xd-11 bz1798148]# virsh start vm1
Domain vm1 started

[root@dell-per740xd-11 bz1798148]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/overlay.qcow2' index='1'/>
      <backingStore type='file' index='2'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/base.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>

3. Using a raw file as base image, it will be detected as RAW
[root@dell-per740xd-11 bz1798148]# virsh destroy vm1
Domain vm1 destroyed

[root@dell-per740xd-11 bz1798148]# qemu-img create -f raw /var/lib/libvirt/images/base.raw 1G
Formatting '/var/lib/libvirt/images/base.raw', fmt=raw size=1073741824

[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 -b /var/lib/libvirt/images/base.raw /var/lib/libvirt/images/overlay.qcow2 1G
Formatting '/var/lib/libvirt/images/overlay.qcow2', fmt=qcow2 size=1073741824 backing_file=/var/lib/libvirt/images/base.raw cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@dell-per740xd-11 bz1798148]# virsh start vm1
Domain vm1 started

[root@dell-per740xd-11 bz1798148]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/overlay.qcow2' index='1'/>
      <backingStore type='file' index='2'>
        <format type='raw'/>
        <source file='/var/lib/libvirt/images/base.raw'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>


4. When overlay.qcow2's backing file also having backing file, it's not supported as expected
[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 /var/lib/libvirt/images/base.qcow2 1G
Formatting '/var/lib/libvirt/images/base.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 -b /var/lib/libvirt/images/base.qcow2 /var/lib/libvirt/images/layer1.qcow2 1G
Formatting '/var/lib/libvirt/images/layer1.qcow2', fmt=qcow2 size=1073741824 backing_file=/var/lib/libvirt/images/base.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 -b /var/lib/libvirt/images/layer1.qcow2 /var/lib/libvirt/images/overlay.qcow2 1G
Formatting '/var/lib/libvirt/images/overlay.qcow2', fmt=qcow2 size=1073741824 backing_file=/var/lib/libvirt/images/layer1.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@dell-per740xd-11 bz1798148]# virsh start vm1
error: Failed to start domain vm1
error: Requested operation is not valid: format of backing image '/var/lib/libvirt/images/layer1.qcow2' of image '/var/lib/libvirt/images/overlay.qcow2' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)


5.When backing file is not local (using a iscsi lun to test), it's not supported as expected
[root@dell-per740xd-11 bz1798148]# iscsiadm -m discovery -t sendtargets -p 127.0.0.1
127.0.0.1:3260,1 iqn.2016-03.com.virttest:logical-pool.target


[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 iscsi://127.0.0.1/iqn.2016-03.com.virttest:logical-pool.target/0 100M
Formatting 'iscsi://127.0.0.1/iqn.2016-03.com.virttest:logical-pool.target/0', fmt=qcow2 size=104857600 cluster_size=65536 lazy_refcounts=off refcount_bits=16


[root@dell-per740xd-11 bz1798148]# qemu-img create -f qcow2 -b iscsi://127.0.0.1/iqn.2016-03.com.virttest:logical-pool.target/0 /var/lib/libvirt/images/overlay.qcow2 1G
Formatting '/var/lib/libvirt/images/overlay.qcow2', fmt=qcow2 size=1073741824 backing_file=iscsi://127.0.0.1/iqn.2016-03.com.virttest:logical-pool.target/0 cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@dell-per740xd-11 bz1798148]# virsh start vm1
error: Failed to start domain vm1
error: Requested operation is not valid: format of backing image 'iscsi://127.0.0.1/iqn.2016-03.com.virttest:logical-pool.target/0' of image '/var/lib/libvirt/images/overlay.qcow2' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)

Comment 17 errata-xmlrpc 2020-05-05 09:57:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.