Description of problem: When attempting to install a guest on an AArch64 machine with virt-install we now get this error ERROR operation failed: unable to find any master var store for loader: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw Version-Release number of selected component (if applicable): libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7 How reproducible: 100% Steps to Reproduce: 1. virt-install -n tmp --vcpus 8 --memory 2048 --disk size=6 --network default -l $REPO Actual results: ERROR operation failed: unable to find any master var store for loader: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw Removing disk 'tmp.qcow2' Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start tmp otherwise, please restart your installation. Expected results: Guest installation starts. Additional info: 5.6.0-6.module+el8.1.0+4244+9aa4e6bb works as expected.
Gavin Shan came up with a workaround for this. One can add --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/AAVMF/AAVMF_VARS.fd,loader_secure=no to the virt-install command line to avoid the error.
Rerun the test case with last qemu-kvm/libvirt scratch builds, the issue is still existing: BaseOS: RHEL-8.2.0-20191126.n.0 qemu-kvm: http://brew-task-repos.usersys.redhat.com/repos/scratch/drjones/qemu-kvm/4.2.0/0.el8.drjones201911191900/$basearch/ libvirt: http://brew-task-repos.usersys.redhat.com/repos/scratch/jdenemar/libvirt/5.9.0/4.virtio_fs.b68426e149.module+el8.2.0+4836+a8e32ad7/$basearch/ [root@amd-seattle-07]# virt-install -n rhel8 --vcpus 8 --memory 4096 --disk size=6 --network bridge=br0 \ -l http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.2/compose/BaseOS/aarch64/os/ Starting install... Retrieving file vmlinuz... | 23 MB 00:00:00 !!! Retrieving file initrd.img... | 56 MB 00:00:00 Allocating 'rhel8.qcow2' | 6.0 GB 00:00:00 ERROR operation failed: unable to find any master var store for loader: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw Removing disk 'rhel8.qcow2' | 0 B 00:00:00 Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start rhel8 otherwise, please restart your installation. The workaround from comment#1 still helps: [root@amd-seattle-07]# virt-install -n rhel8 --vcpus 8 --memory 4096 --disk size=6 --network bridge=br0 \ --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/AAVMF/AAVMF_VARS.fd,loader_secure=no \ -l http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.2/compose/BaseOS/aarch64/os/ Starting install... Retrieving file vmlinuz... | 23 MB 00:00:00 !!! Retrieving file initrd.img... | 56 MB 00:00:00 Allocating 'rhel8.qcow2' | 6.0 GB 00:00:00 Connected to domain rhel8 Escape character is ^]
Can you say what version of the edk2-aarch64 RPMs you have installed. Also is there any file in /usr/share/qemu/firmware that points to /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw ? If so, does it have an nvram-template field that points to a var store file & does the file exist ?
(In reply to Daniel Berrangé from comment #3) > Can you say what version of the edk2-aarch64 RPMs you have installed. edk2-aarch64-20190308git89910a39dcfd-6.el8.noarch > > Also is there any file in /usr/share/qemu/firmware that points to > /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw ? 60-edk2-aarch64.json > > If so, does it have an nvram-template field that points to a var store file > & does the file exist ? "nvram-template": { "filename": "/usr/share/edk2/aarch64/vars-template-pflash.raw", "format": "raw" } # ls -lZ /usr/share/edk2/aarch64/vars-template-pflash.raw -rw-r--r--. 1 root root system_u:object_r:usr_t:s0 67108864 Aug 5 03:46 /usr/share/edk2/aarch64/vars-template-pflash.raw Thanks, drew
Can you enable some debugging in libvirtd while it is running do $ virt-admin daemon-log-filters 1:qemu_firmware $ virt-admin daemon-log-outputs 1:file:/var/log/libvirt/libvirtd.log Now repeat the virt-install command, and upload the resulting logfile to this bug
Ok, I have an idea what the problem is possibly is. Can you use the --debug arg to virt-install & attach its output too.
There's no firmware=efi on the <os> attribute, and virt-install is providing an explicit loader path instead. AFAICT, this means during guest boot, we don't appear to use the firmware metadata descriptors, and instead rely on matching against the legacy built-in firmware list. We didn't have firmware descriptor support in libvirt in 8.0 stream, so I guess this is the cause of the regression in some way.
I can also reproduce this bug on X86 platform, so I think we'd better change "Hardware" to "Unspecified" Version-Release number of selected component (if applicable): libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7.x86_64 virt-manager-2.2.1-3.el8.noarch virt-install-2.2.1-3.el8.noarch edk2-ovmf-20190308git89910a39dcfd-6.el8.noarch qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64 How reproducible: 100% Steps to Reproduce: 1. Install a vm with Chipset: Q35 + Firmware:UEFI x86_64: /usr/share/OVMF/OVMF_CODE.secboot.fd # virt-install --memory 2048 --nodisk --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191206.3/compose/BaseOS/x86_64/os/ --boot uefi Using default --name rhel8 Starting install... Retrieving file vmlinuz... | 7.9 MB 00:00:00 Retrieving file initrd.img... | 62 MB 00:00:00 ERROR operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start rhel8 otherwise, please restart your installation
(In reply to zhoujunqin from comment #11) > I can also reproduce this bug on X86 platform, so I think we'd better change > "Hardware" to "Unspecified" > Agreed, this is not arch specific. I'm working on a fix as we speak.
Patches proposed on the upstream list: https://www.redhat.com/archives/libvir-list/2019-December/msg01076.html
v2: https://www.redhat.com/archives/libvir-list/2020-January/msg00231.html
I've pushed patches upstream: 8e1804f9f6 qemu_firmware: Try to autofill for old style UEFI specification 7c5264d2be src: Introduce and use virDomainDefHasOldStyleUEFI() and virDomainDefHasOldStyleROUEFI() 57f9067ca3 qemu_firmware: Introduce @want variable to qemuFirmwareMatchDomain() 50d7465f3d qemu_firmware: Pass virDomainDef into qemuFirmwareFillDomain() v5.10.0-507-g8e1804f9f6
Switching over to RHEL-AV and moving to POST.
For those that are hitting this in the meantime (because I don't see a rawhide update yet) this can be hacked around by uncommenting the nvram stanza in /etc/libvirt/qemu.conf and renaming /usr/share/qemu/firmware.
Try to verify this bug with packages, but the vm failed to boot into os. Packages: libvirt-6.0.0-2.module+el8.2.0+5513+34927b6c.x86_64 virt-install-2.2.1-3.el8.noarch edk2-ovmf-20190829git37eef91017ad-6.el8.noarch qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64 Steps: # virt-admin daemon-log-filters 1:qemu_firmware # virt-admin daemon-log-outputs 1:file:/var/log/libvirt/libvirtd.log # virt-install --memory 4096 --disk /home/test-ovmf4.img,size=10 --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/ --boot uefi Using default --name rhel8.2-4 .... Result: I. vm can be installed successfully, but after click 'reboot' button, vm failed to boot into os system. Please see screenshot for detail information. So I think this bug issue has not been fixed fully, thanks.
Created attachment 1657245 [details] Screenshot-failed-to-boot-into-os
And I also find a easy way to reproduce Comment 19 issue. Steps: 1. Install a vm by ISO method: # virt-install --name test-ov5 --memory 4096 --disk /home/test-ov5.img,size=10 --cdrom RHEL-8.2.0-20191219.0-x86_64-dvd1.iso --boot uefi ... Result: After select "Install Red Hat Enterprise Linux 8.2.0" in the first page, I will reproduce Comment 19 issue. It blocked a lot of following testing, so I change the bug status to ASSIGNED, thanks.
(In reply to zhoujunqin from comment #21) > And I also find a easy way to reproduce Comment 19 issue. > > Steps: > 1. Install a vm by ISO method: > # virt-install --name test-ov5 --memory 4096 --disk > /home/test-ov5.img,size=10 --cdrom RHEL-8.2.0-20191219.0-x86_64-dvd1.iso > --boot uefi > > ... > > Result: After select "Install Red Hat Enterprise Linux 8.2.0" in the first > page, I will reproduce Comment 19 issue. > > It blocked a lot of following testing, so I change the bug status to > ASSIGNED, thanks. I can't reproduce with these steps. Can you please provide domain XML from when the domain is being installed and after the reboot? I suspect the installer did not work properly.
1. Using ISO to install a vm: # virt-install --name test-ov6 --memory 4096 --disk /home/test-ov6.img,size=10 --cdrom RHEL-8.2.0-20191219.0-x86_64-dvd1.iso --boot uefi --debug |& tee > virt-install-iso.log 2. Using http to install a vm: # virt-install --name test-ov7 --memory 4096 --disk /home/test-ov7.img,size=10 --boot uefi --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/ --debug |& tee > virt-install-http.log Result: Failed to boot into vm os.
Created attachment 1657324 [details] virt-install-iso.log
Created attachment 1657325 [details] virt-install-http.log # virsh dumpxml test-ov7 ... <os> <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type> <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd</nvram> <boot dev='hd'/> </os>
So this is the domain XML that libvirt is using for installation: <os> <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type> <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader> <nvram template='/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd'>/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd</nvram> <kernel>/var/lib/libvirt/boot/virtinst-t4ddqzw0-vmlinuz</kernel> <initrd>/var/lib/libvirt/boot/virtinst-7yhcjyn7-initrd.img</initrd> <cmdline>inst.repo=http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/</cmdline> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> <smm state='on'/> </features> and then, as you say in comment 25, the same NVRAM and loader are used after the reboot. From this I would conclude that the generated command line is the same in both cases. Can you please check that? There doesn't seem to be anything obviously wrong here.
Hi mprivozn, Thanks for your quick reply and suggestion. As you guess, it works well with RHEL-8.1.0 released version. Here is my testing result, please help review, thanks. 1. Test with RHEL-8.1.0 released version. # virt-install --name test-ov9 --memory 4096 --disk /home/test-ov9.img,size=10 -l http://download.eng.pek2.redhat.com/released/rhel-6-7-8/rhel-8/RHEL-8/8.1.0/BaseOS/x86_64/os/ --boot uefi --debug |& tee > test-ov9.log ... <os> <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type> <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/test-ov9_VARS.fd</nvram> <boot dev='hd'/> </os> ... Result: vm can be installed and boot up successfully, but my concern is that when we can test with latest rhel8.2 version before the its release. 2. Modify xml of vm installed with rhel8.2 latest tree. 2.1 # virt-install --name test-ov7 --memory 4096 --disk /home/test-ov7.img,size=10 --boot uefi --location http://download.eng.pek2.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.2.0-20191219.0/compose/BaseOS/x86_64/os/ --debug |& tee > virt-install-http.log Result: vm can be installed successfully but failed to reboot. 2.2 Then modify xml from using vm specified nvram file '/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd'to the '/usr/share/OVMF/OVMF_VARS.fd' From ... <os> <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type> <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/test-ov7_VARS.fd</nvram> <boot dev='hd'/> </os> ... To: <os> <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type> <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader> <nvram>/usr/share/OVMF/OVMF_VARS.fd</nvram> <boot dev='hd'/> </os> Result: Vm can boot into os successfully. So @mprivozn, could you help tell the difference, thanks.
Hi all, Thanks for all your details explanation. And I have confirmed with my colleagues that for Secure boot vm installation testing, we only cover released version. Such as, now we're in rhel8.2 testing phase, testing with rhel8.1 released version is enough. According to Comment 28, It works well with rhel8.1 released version, but failed to boot up with rhel8.2 version, that's as expected. So I change this bug back to ON_QA status now, and I am also sorry about re-assigning the bug back to you. @Andrew, could you help test again with latest libvirt package on aarch64 platform, if it works for you, I will move this bug to VERIFIED status.
(In reply to zhoujunqin from comment #31) > @Andrew, could you help test again with latest libvirt package on aarch64 > platform, if it works for you, I will move this bug to VERIFIED status. I tested libvirt-6.0.0-4.module+el8.2.0+5642+838f3513.aarch64 and virt-install was able to start the guest installation as expected (comment 0) -- the workaround from comment 1 was not necessary. Looks good to me for AArch64.
(In reply to Andrew Jones from comment #33) > (In reply to zhoujunqin from comment #31) > > @Andrew, could you help test again with latest libvirt package on aarch64 > > platform, if it works for you, I will move this bug to VERIFIED status. > > I tested libvirt-6.0.0-4.module+el8.2.0+5642+838f3513.aarch64 and > virt-install was able to start the guest installation as expected (comment > 0) -- the workaround from comment 1 was not necessary. Looks good to me for > AArch64. Hi Andrew, Thanks for your testing. So I move this bug from ON_QA to VERIFIED status based on Comment 28(It works well with released rhel version) and Comment 33(It also works well on AArch64 platform), thanks.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017