Bug 1776949 - Regression: unable to find any master var store for loader
Summary: Regression: unable to find any master var store for loader
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Michal Privoznik
QA Contact: zhoujunqin
URL:
Whiteboard:
Depends On:
Blocks: 1677408 1707874
TreeView+ depends on / blocked
 
Reported: 2019-11-26 16:08 UTC by Andrew Jones
Modified: 2020-08-17 10:38 UTC (History)
15 users (show)

Fixed In Version: libvirt-6.0.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-05 09:51:23 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
Screenshot-failed-to-boot-into-os (70.86 KB, image/png)
2020-02-03 03:59 UTC, zhoujunqin
no flags Details
virt-install-iso.log (17.24 KB, text/plain)
2020-02-03 10:33 UTC, zhoujunqin
no flags Details
virt-install-http.log (20.25 KB, text/plain)
2020-02-03 10:35 UTC, zhoujunqin
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2017 0 None None None 2020-05-05 09:51:57 UTC

Internal Links: 1782778

Description Andrew Jones 2019-11-26 16:08:15 UTC
Description of problem:

When attempting to install a guest on an AArch64 machine with virt-install we now get this error

ERROR    operation failed: unable to find any master var store for loader: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw

Version-Release number of selected component (if applicable):

libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7

How reproducible:

100%

Steps to Reproduce:
1. virt-install -n tmp --vcpus 8 --memory 2048 --disk size=6 --network default -l  $REPO

Actual results:

ERROR    operation failed: unable to find any master var store for loader: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw
Removing disk 'tmp.qcow2'                                                                                                                                                                                                                                          
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start tmp
otherwise, please restart your installation.

Expected results:

Guest installation starts.

Additional info:

5.6.0-6.module+el8.1.0+4244+9aa4e6bb works as expected.

Comment 1 Andrew Jones 2019-11-26 16:36:19 UTC
Gavin Shan came up with a workaround for this. One can add

 --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/AAVMF/AAVMF_VARS.fd,loader_secure=no

to the virt-install command line to avoid the error.

Comment 2 Guowen Shan 2019-11-27 02:28:00 UTC
Rerun the test case with last qemu-kvm/libvirt scratch builds, the issue is still existing:

BaseOS:    RHEL-8.2.0-20191126.n.0
qemu-kvm:  http://brew-task-repos.usersys.redhat.com/repos/scratch/drjones/qemu-kvm/4.2.0/0.el8.drjones201911191900/$basearch/
libvirt:   http://brew-task-repos.usersys.redhat.com/repos/scratch/jdenemar/libvirt/5.9.0/4.virtio_fs.b68426e149.module+el8.2.0+4836+a8e32ad7/$basearch/


[root@amd-seattle-07]# virt-install -n rhel8 --vcpus 8 --memory 4096 --disk size=6 --network bridge=br0 \
                       -l http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.2/compose/BaseOS/aarch64/os/

Starting install...
Retrieving file vmlinuz...                                                                                                        |  23 MB  00:00:00 !!! 
Retrieving file initrd.img...                                                                                                     |  56 MB  00:00:00     
Allocating 'rhel8.qcow2'                                                                                                          | 6.0 GB  00:00:00     
ERROR    operation failed: unable to find any master var store for loader: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw
Removing disk 'rhel8.qcow2'                                                                                                       |    0 B  00:00:00     
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start rhel8
otherwise, please restart your installation.

The workaround from comment#1 still helps:

[root@amd-seattle-07]# virt-install -n rhel8 --vcpus 8 --memory 4096 --disk size=6 --network bridge=br0 \
                       --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/AAVMF/AAVMF_VARS.fd,loader_secure=no \
                       -l http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.2/compose/BaseOS/aarch64/os/

Starting install...
Retrieving file vmlinuz...                                                                                                        |  23 MB  00:00:00 !!! 
Retrieving file initrd.img...                                                                                                     |  56 MB  00:00:00     
Allocating 'rhel8.qcow2'                                                                                                          | 6.0 GB  00:00:00     
Connected to domain rhel8
Escape character is ^]

Comment 3 Daniel Berrangé 2019-11-27 14:14:43 UTC
Can you say what version of the edk2-aarch64 RPMs you have installed.

Also is there any file in /usr/share/qemu/firmware that points to /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw ?

If so, does it have an nvram-template field that points to a var store file & does the file exist ?

Comment 4 Andrew Jones 2019-11-27 15:20:30 UTC
(In reply to Daniel Berrangé from comment #3)
> Can you say what version of the edk2-aarch64 RPMs you have installed.

edk2-aarch64-20190308git89910a39dcfd-6.el8.noarch

> 
> Also is there any file in /usr/share/qemu/firmware that points to
> /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw ?

60-edk2-aarch64.json

> 
> If so, does it have an nvram-template field that points to a var store file
> & does the file exist ?

        "nvram-template": {
            "filename": "/usr/share/edk2/aarch64/vars-template-pflash.raw",
            "format": "raw"
        }

# ls -lZ /usr/share/edk2/aarch64/vars-template-pflash.raw
-rw-r--r--. 1 root root system_u:object_r:usr_t:s0 67108864 Aug  5 03:46 /usr/share/edk2/aarch64/vars-template-pflash.raw


Thanks,
drew

Comment 5 Daniel Berrangé 2019-11-27 15:31:51 UTC
Can you enable some debugging in libvirtd while it is running do

$ virt-admin daemon-log-filters 1:qemu_firmware
$ virt-admin daemon-log-outputs 1:file:/var/log/libvirt/libvirtd.log

Now repeat the virt-install command, and upload the resulting logfile to this bug

Comment 7 Daniel Berrangé 2019-11-27 16:38:34 UTC
Ok, I have an idea what the problem is possibly is. Can you use  the --debug arg to virt-install & attach its output too.

Comment 9 Daniel Berrangé 2019-11-27 16:59:35 UTC
There's no firmware=efi on the <os> attribute, and virt-install is providing an explicit loader path instead.

AFAICT, this means during guest boot, we don't appear to use the firmware metadata descriptors, and instead rely on matching  against the legacy built-in firmware list.

We didn't have firmware descriptor support in libvirt in 8.0 stream, so I guess this is the cause of the regression in some way.

Comment 11 zhoujunqin 2019-12-13 09:03:40 UTC
I can also reproduce this bug on X86 platform, so I think we'd better change "Hardware" to "Unspecified" 

Version-Release number of selected component (if applicable):
libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7.x86_64
virt-manager-2.2.1-3.el8.noarch
virt-install-2.2.1-3.el8.noarch
edk2-ovmf-20190308git89910a39dcfd-6.el8.noarch
qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install a vm with Chipset: Q35 + Firmware:UEFI x86_64: /usr/share/OVMF/OVMF_CODE.secboot.fd
# virt-install  --memory 2048 --nodisk --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191206.3/compose/BaseOS/x86_64/os/ --boot uefi 
Using default --name rhel8

Starting install...
Retrieving file vmlinuz...                                                                                                                                                                  | 7.9 MB  00:00:00     
Retrieving file initrd.img...                                                                                                                                                               |  62 MB  00:00:00     
ERROR    operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start rhel8
otherwise, please restart your installation

Comment 12 Michal Privoznik 2019-12-17 09:10:07 UTC
(In reply to zhoujunqin from comment #11)
> I can also reproduce this bug on X86 platform, so I think we'd better change
> "Hardware" to "Unspecified" 
> 

Agreed, this is not arch specific. I'm working on a fix as we speak.

Comment 13 Michal Privoznik 2019-12-17 17:25:46 UTC
Patches proposed on the upstream list:

https://www.redhat.com/archives/libvir-list/2019-December/msg01076.html

Comment 15 Michal Privoznik 2020-01-07 16:11:32 UTC
I've pushed patches upstream:

8e1804f9f6 qemu_firmware: Try to autofill for old style UEFI specification
7c5264d2be src: Introduce and use virDomainDefHasOldStyleUEFI() and virDomainDefHasOldStyleROUEFI()
57f9067ca3 qemu_firmware: Introduce @want variable to qemuFirmwareMatchDomain()
50d7465f3d qemu_firmware: Pass virDomainDef into qemuFirmwareFillDomain()

v5.10.0-507-g8e1804f9f6

Comment 16 Michal Privoznik 2020-01-08 08:29:51 UTC
Switching over to RHEL-AV and moving to POST.

Comment 17 Jeremy Linton 2020-01-10 20:41:30 UTC
For those that are hitting this in the meantime (because I don't see a rawhide update yet) this can be hacked around by uncommenting the nvram stanza in /etc/libvirt/qemu.conf and renaming /usr/share/qemu/firmware.

Comment 19 zhoujunqin 2020-02-03 03:55:18 UTC
Try to verify this bug with packages, but the vm failed to boot into os.

Packages:
libvirt-6.0.0-2.module+el8.2.0+5513+34927b6c.x86_64
virt-install-2.2.1-3.el8.noarch
edk2-ovmf-20190829git37eef91017ad-6.el8.noarch
qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64


Steps:
# virt-admin daemon-log-filters 1:qemu_firmware

# virt-admin daemon-log-outputs 1:file:/var/log/libvirt/libvirtd.log

# virt-install  --memory 4096 --disk /home/test-ovmf4.img,size=10  --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/ --boot uefi 
Using default --name rhel8.2-4

....

Result:
I. vm can be installed successfully, but after click 'reboot' button, vm failed to boot into os system.
Please see screenshot for detail information.

So I think this bug issue has not been fixed fully, thanks.

Comment 20 zhoujunqin 2020-02-03 03:59:03 UTC
Created attachment 1657245 [details]
Screenshot-failed-to-boot-into-os

Comment 21 zhoujunqin 2020-02-03 04:15:57 UTC
And I also find a easy way to reproduce Comment 19 issue.

Steps:
1. Install a vm by ISO method:
# virt-install --name test-ov5 --memory 4096 --disk /home/test-ov5.img,size=10 --cdrom RHEL-8.2.0-20191219.0-x86_64-dvd1.iso --boot uefi 

...

Result: After select "Install Red Hat Enterprise Linux 8.2.0" in the first page, I will reproduce Comment 19 issue.

It blocked a lot of following testing, so I change the bug status to ASSIGNED, thanks.

Comment 22 Michal Privoznik 2020-02-03 09:36:35 UTC
(In reply to zhoujunqin from comment #21)
> And I also find a easy way to reproduce Comment 19 issue.
> 
> Steps:
> 1. Install a vm by ISO method:
> # virt-install --name test-ov5 --memory 4096 --disk
> /home/test-ov5.img,size=10 --cdrom RHEL-8.2.0-20191219.0-x86_64-dvd1.iso
> --boot uefi 
> 
> ...
> 
> Result: After select "Install Red Hat Enterprise Linux 8.2.0" in the first
> page, I will reproduce Comment 19 issue.
> 
> It blocked a lot of following testing, so I change the bug status to
> ASSIGNED, thanks.

I can't reproduce with these steps. Can you please provide domain XML from when the domain is being installed and after the reboot? I suspect the installer did not work properly.

Comment 23 zhoujunqin 2020-02-03 10:31:39 UTC
1. Using ISO to install a vm:
# virt-install --name test-ov6 --memory 4096 --disk /home/test-ov6.img,size=10 --cdrom RHEL-8.2.0-20191219.0-x86_64-dvd1.iso --boot uefi  --debug |& tee > virt-install-iso.log

2. Using http to install a vm:
# virt-install --name test-ov7 --memory 4096 --disk /home/test-ov7.img,size=10  --boot uefi --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/ --debug |& tee > virt-install-http.log

Result: Failed to boot into vm os.

Comment 24 zhoujunqin 2020-02-03 10:33:34 UTC
Created attachment 1657324 [details]
virt-install-iso.log

Comment 25 zhoujunqin 2020-02-03 10:35:54 UTC
Created attachment 1657325 [details]
virt-install-http.log

# virsh dumpxml test-ov7
...
  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

Comment 26 Michal Privoznik 2020-02-03 12:01:07 UTC
So this is the domain XML that libvirt is using for installation:

  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
    <nvram template='/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd'>/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd</nvram>
    <kernel>/var/lib/libvirt/boot/virtinst-t4ddqzw0-vmlinuz</kernel>
    <initrd>/var/lib/libvirt/boot/virtinst-7yhcjyn7-initrd.img</initrd>
    <cmdline>inst.repo=http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
    <smm state='on'/>
  </features>

and then, as you say in comment 25, the same NVRAM and loader are used after the reboot. From this I would conclude that the generated command line is the same in both cases. Can you please check that? There doesn't seem to be anything obviously wrong here.

Comment 28 zhoujunqin 2020-02-04 09:28:04 UTC
Hi mprivozn,
Thanks for your quick reply and suggestion.
As you guess, it works well with RHEL-8.1.0 released version.

Here is my testing result, please help review, thanks.

1. Test with RHEL-8.1.0 released version.
# virt-install --name test-ov9 --memory 4096 --disk /home/test-ov9.img,size=10 -l http://download.eng.pek2.redhat.com/released/rhel-6-7-8/rhel-8/RHEL-8/8.1.0/BaseOS/x86_64/os/ --boot uefi --debug |& tee > test-ov9.log
...
  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/test-ov9_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
...

Result: vm can be installed and boot up successfully, but my concern is that when we can test with latest rhel8.2 version before the its release.

2. Modify xml of vm installed with rhel8.2 latest tree.
2.1 #  virt-install --name test-ov7 --memory 4096 --disk /home/test-ov7.img,size=10  --boot uefi --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/ --debug |& tee > virt-install-http.log

Result: vm can be installed successfully but failed to reboot.

2.2 Then modify xml from using vm specified nvram file '/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd'to the '/usr/share/OVMF/OVMF_VARS.fd' 
From
...
  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
...

To:
  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
    <nvram>/usr/share/OVMF/OVMF_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

Result: Vm can boot into os successfully.

So @mprivozn, could you help tell the difference, thanks.

Comment 31 zhoujunqin 2020-02-05 07:56:29 UTC
Hi all, 
Thanks for all your details explanation.

And I have confirmed with my colleagues that for Secure boot vm installation testing, we only cover released version.
Such as, now we're in rhel8.2 testing phase, testing with rhel8.1 released version is enough.

According to Comment 28, It works well with rhel8.1 released version, but failed to boot up with rhel8.2 version, that's as expected.
So I change this bug back to ON_QA status now, and I am also sorry about re-assigning the bug back to you.

@Andrew, could you help test again with latest libvirt package on aarch64 platform, if it works for you, I will move this bug to VERIFIED status.

Comment 33 Andrew Jones 2020-02-10 16:34:09 UTC
(In reply to zhoujunqin from comment #31)
> @Andrew, could you help test again with latest libvirt package on aarch64
> platform, if it works for you, I will move this bug to VERIFIED status.

I tested libvirt-6.0.0-4.module+el8.2.0+5642+838f3513.aarch64 and virt-install was able to start the guest installation as expected (comment 0) -- the workaround from comment 1 was not necessary. Looks good to me for AArch64.

Comment 34 zhoujunqin 2020-02-11 02:32:18 UTC
(In reply to Andrew Jones from comment #33)
> (In reply to zhoujunqin from comment #31)
> > @Andrew, could you help test again with latest libvirt package on aarch64
> > platform, if it works for you, I will move this bug to VERIFIED status.
> 
> I tested libvirt-6.0.0-4.module+el8.2.0+5642+838f3513.aarch64 and
> virt-install was able to start the guest installation as expected (comment
> 0) -- the workaround from comment 1 was not necessary. Looks good to me for
> AArch64.

Hi Andrew,
Thanks for your testing.

So I move this bug from ON_QA to VERIFIED status based on Comment 28(It works well with released rhel version) and Comment 33(It also works well on AArch64 platform), thanks.

Comment 36 errata-xmlrpc 2020-05-05 09:51:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.