Description of problem: Failed to create VM from image hit libvirt error "format of backing image was not specified in the image metadata" This should caused by the libvirt code change in RHEL-AV 8.2.0, more usage details please refer to Bug 1798148 comment 2 Version-Release number of selected component (if applicable): kernel-core-4.18.0-177.el8.x86_64 libvirt-daemon-kvm-6.0.0-4.module+el8.2.0+5642+838f3513.x86_64 qemu-kvm-core-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64 openstack-nova-compute-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch How reproducible: 100% Steps to Reproduce: 1. Creat flavor, image, network in OSP env 2. Try to create VM with the image failed, hit error (overcloud) [stack@dell-per730** ~]$ openstack server create --flavor m2 --image r8-qcow2 --nic net-id=0303521b-abd4-49d6-bdfd-3156589043f8 vm-r8-qcow2 Check the error message in nova_compute.log: ----------------------------------------------------------------------------------------------------- 2020-02-14 02:22:41.954 7 ERROR nova.compute.manager [instance: 1a18c83c-db95-40ac-9e50-b3cec3ec9f1a] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1166, in createWithFlags 2020-02-14 02:22:41.954 7 ERROR nova.compute.manager [instance: 1a18c83c-db95-40ac-9e50-b3cec3ec9f1a] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2020-02-14 02:22:41.954 7 ERROR nova.compute.manager [instance: 1a18c83c-db95-40ac-9e50-b3cec3ec9f1a] libvirt.libvirtError: Requested operation is not valid: format of backing image '/var/lib/nova/instances/_base/c8688b29886922382b3ccf03db5d0855756c32a6' of image '/var/lib/nova/instances/1a18c83c-db95-40ac-9e50-b3cec3ec9f1a/disk' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting) 2020-02-14 02:22:41.954 7 ERROR nova.compute.manager [instance: 1a18c83c-db95-40ac-9e50-b3cec3ec9f1a] Actual results: In step2: create VM failed Expected results: In step2: create VM successfully Additional info: Bug 1798148 - Regression: Requested operation is not valid: format of backing image ... was not specified in the image metadata
Created attachment 1663048 [details] nova-compute.log
While libvirt is going to allow image format probing in few very specific cases (see Bug 1798148) we _STRONGLY_ suggest to record the image format into the overlay if it's known to prevent having to guess. That is always safer.
Test on osp16.1, create VM from image successfully. Tested with packages: libvirt-daemon-kvm-6.0.0-16.module+el8.2.0+6139+d66dece5.x86_64 qemu-kvm-core-4.2.0-17.module+el8.2.0+6141+0f540f16.x86_64 openstack-nova-compute-20.1.2-0.20200413153449.28324e6.el8ost.noarch Steps: 1. Creat flavor, image, network in OSP env 2. Create VM with the image successfully $ openstack server create --flavor m1.small --image r8-qcow2 --nic net-id=8e4f3f6e-918a-4d0f-a4a8-1c1eb05a85af vm-r8-qcow2 $ openstack server list +--------------------------------------+-------------+--------+----------+----------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------+--------+----------+----------+--------+ | 108b560f-8cc0-4540-854d-95a7407a58d8 | vm-r8-qcow2 | BUILD | | r8-qcow2 | | +--------------------------------------+-------------+--------+----------+----------+--------+ $ openstack server list +--------------------------------------+-------------+--------+------------------------+----------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------+--------+------------------------+----------+--------+ | 108b560f-8cc0-4540-854d-95a7407a58d8 | vm-r8-qcow2 | ACTIVE | default=192.168.34.128 | r8-qcow2 | | +--------------------------------------+-------------+--------+------------------------+----------+--------+ 3. Login to nova_libvirt, can dumpxml and can login to VM console, touch file in VM
No, you should not see this failure, assuming you _do_ have the patch that provides the backing file format: https://code.engineering.redhat.com/gerrit/#/c/193812/ — libvirt: Provide the backing file format when creating qcow2 disks @ Archit: Is this reproducible?
(In reply to Kashyap Chamarthy from comment #10) > No, you should not see this failure, assuming you _do_ have the patch that > provides the backing file format: > > > https://code.engineering.redhat.com/gerrit/#/c/193812/ — libvirt: > Provide the backing file format when creating qcow2 disks > > > @ Archit: Is this reproducible? yes this is reproducible, and yes i do have the patch ()[root@compute-1 /]# cat /usr/lib/python3.6/site-packages/nova/virt/libvirt/utils.py | grep "cow_opts" cow_opts = [] cow_opts += ['backing_file=%s' % backing_file] cow_opts += ['backing_fmt=%s' % base_details.file_format] Verision: openstack-nova-compute-20.2.1-0.20200528080027.1e95025.el8ost.noarch, Logs coming in the following attachment
Created attachment 1696553 [details] test-run-console-logs
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Hi, Archit From libvirt aspective, by the command[1], overlay.qcow2 will be created and it's backing image is r8-2.qcow2, but itself don't have bootable OS. [1] # qemu-img create -f qcow2 -F qcow2 -b r8-2.qcow2 overlay.qcow2 # ls -lh total 13G -rw-r--r--. 1 root root 193K Jun 22 04:12 overlay.qcow2 -rw-r--r--. 1 root root 739M Jun 22 04:11 r8-2.qcow2 Boot an VM from overlay.qcow2 in libvirt, it'll use the backing image r8-2.qcow2, the xml is like this: <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/overlay.qcow2' index='1'/> <backingStore type='file' index='2'> <format type='qcow2'/> <source file='/var/lib/libvirt/images/r8-2.qcow2'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> Create an image in OSP from r8-2.qcow2 image id:181ea5e5-c345-4200-b205-8a6dd8d66d91, the xml is like this: <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/378cdd5d-df7c-47ec-9085-2ed4181ef6ec/disk' index='1'/> <backingStore type='file' index='2'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/3b0c44f327057dcb8b85dfd7cced735f833eb930'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> ... If you create an image in OSP from overlay.qcow2, when try to start the VM, it'll try to find the r8-2.qcow2, and given the path"/var/lib/nova/instances/_base/r8-2.qcow2", this is not existed, so you get the below error: --------------------------------------------------------------------------------------------------------- ERROR nova.compute.manager [req-cea508ad-a447-4a83-a91b-d710c2d52775 317f1c1fe439476dbfb991ec7768d23f 63a5b71d92f74e0ba553e6acca2c747c - default default] [instance: e34ba4d8-8d7b-4397-8eb8-84e03c9e5af9] Build of instance e34ba4d8-8d7b-4397-8eb8-84e03c9e5af9 aborted: Image cb2dcc77-cd19-4c57-b547-a8f87bfb7ce4 is unacceptable: fmt=qcow2 backed by: /var/lib/nova/instances/_base/r8-2.qcow2: nova.exception.BuildAbortException: Build of instance e34ba4d8-8d7b-4397-8eb8-84e03c9e5af9 aborted: Image cb2dcc77-cd19-4c57-b547-a8f87bfb7ce4 is unacceptable: fmt=qcow2 backed by: /var/lib/nova/instances/_base/r8-2.qcow2 --------------------------------------------------------------------------------------------------------- I think we can't use overlay.qcow2 as an image to boot VM directly in OSP, libvirt can detect it's backing image r8-2.qcow2, but OSP need to deal with r8-2.qcow2 first, if want to start VM from overlay.qcow2.
(In reply to chhu from comment #15) > I think we can't use overlay.qcow2 as an image to boot VM directly in OSP, > libvirt can detect it's backing image r8-2.qcow2, > but OSP need to deal with r8-2.qcow2 first, if want to start VM from > overlay.qcow2. Correct, Nova doesn't support using Glance qcow2 images that reference a backing file. The test above using one isn't valid, the initial test using a flat qcow2 file is however valid and shows the fix working. Moving the bug to VERIFIED before it's dropped from the 16.1.1 errata.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148