Bug 2241388 - Existing OVMF_VARS_4M.secboot.qcow2 UEFI VMs fail to boot with edk2 20230825-16
Summary: Existing OVMF_VARS_4M.secboot.qcow2 UEFI VMs fail to boot with edk2 20230825-16
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: edk2
Version: 39
Hardware: Unspecified
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Gerd Hoffmann
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-09-29 15:58 UTC by Adam Williamson
Modified: 2023-11-03 18:36 UTC (History)
9 users (show)

Fixed In Version: edk2-20230825-25.fc39
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-03 18:36:19 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
firmware log as requested by kraxel (70.14 KB, text/plain)
2023-09-29 15:59 UTC, Adam Williamson
no flags Details
xml configuration of affected VM (8.11 KB, application/xml)
2023-10-04 15:39 UTC, Adam Williamson
no flags Details
screencast with new vm qcow2 (1.53 MB, video/webm)
2023-10-10 08:01 UTC, laolux
no flags Details
screencast with new vm secboot.fd (1.54 MB, video/webm)
2023-10-10 08:03 UTC, laolux
no flags Details
lscpu output (3.05 KB, text/plain)
2023-10-10 13:44 UTC, laolux
no flags Details
lscpu output from second machine (3.08 KB, text/plain)
2023-10-11 01:24 UTC, laolux
no flags Details
lscpu output (3.05 KB, text/plain)
2023-10-11 01:33 UTC, Adam Williamson
no flags Details
firmware log with working COPR build (368.89 KB, text/plain)
2023-10-11 01:34 UTC, Adam Williamson
no flags Details

Description Adam Williamson 2023-09-29 15:58:48 UTC
On my F39 system, with the update from edk2 20230524-4 to 20230825-16 , two existing UEFI VMs (a Fedora one and a Windows one) both fail to start up. They do not reach the state of showing the firmware logo. One just shows a blank screen, one shows "Display output is not active."

Both use OVMF_VARS_4M.secboot.qcow2 . Gerd asked for a firmware log and gave instructions for capturing it, I'm attaching it.



Reproducible: Always

Steps to Reproduce:
1. Update edk2
2. Boot an affected VM
Actual Results:  
VM is essentially hung immediately

Expected Results:  
VM should boot normally

Comment 1 Adam Williamson 2023-09-29 15:59:18 UTC
Created attachment 1991156 [details]
firmware log as requested by kraxel

Comment 2 Gerd Hoffmann 2023-10-04 08:05:29 UTC
Hmm, the firmware has problems to figure the cpu topology via cpuid instruction.  That should not happen.

Which qemu version is this?
Can you share the VM configuration?

Comment 3 Adam Williamson 2023-10-04 15:39:02 UTC
Created attachment 1992035 [details]
xml configuration of affected VM

qemu is 8.1.1.

Comment 4 Gerd Hoffmann 2023-10-05 11:30:06 UTC
Hmm, pretty standard configuration.  Still can't reproduce that locally.

You've mentioned newly created VMs boot fine?  How does that VM configuration look like?

Can you also share 'lscpu' output of the host machine?

Comment 5 laolux 2023-10-10 08:01:28 UTC
Created attachment 1993208 [details]
screencast with new vm qcow2

I get affected by this bug, too. It also affects newly created vm's. Please see the attached screencast. I am using a fully updated Fedora 39 beta.

edk2-ovmf version: 20230825
libvirt version: 9.7.0
qemu version: 8.1.1

Comment 6 laolux 2023-10-10 08:03:28 UTC
Created attachment 1993209 [details]
screencast with new vm secboot.fd

Additionally, the OVMF_CODE.secboot.fd is also affected.

Comment 7 laolux 2023-10-10 08:22:02 UTC
Downgrading edk2-ovmf to version 20230524 from Fedora 38 fixes the problem for OVMF_CODE.secboot.fd

Comment 8 Gerd Hoffmann 2023-10-10 10:04:58 UTC
(In reply to laolux from comment #5)
> Created attachment 1993208 [details]
> screencast with new vm qcow2

That doesn't help much, I know how to create VMs, problem is
it doesn't reproduce on the hardware park I have here at hand.

(In reply to Gerd Hoffmann from comment #4)
> Can you also share 'lscpu' output of the host machine?

Anyone?

Comment 9 Gerd Hoffmann 2023-10-10 11:12:35 UTC
https://copr.fedorainfracloud.org/coprs/kraxel/edk2.testbuilds/
please try 20230825-24 (still building right now).
please attach the firmware log (even if the build works).

Comment 10 laolux 2023-10-10 13:44:08 UTC
Created attachment 1993259 [details]
lscpu output

(In reply to Gerd Hoffmann from comment #8)
> (In reply to laolux from comment #5)
> > Created attachment 1993208 [details]
> > screencast with new vm qcow2
> 
> That doesn't help much, I know how to create VMs, problem is
> it doesn't reproduce on the hardware park I have here at hand.

I don't doubt that you know what you are doing, but I am not so firm with libvirt. So I included the video to avoid the issue being user error.
The issue occurred on both my Fedora 39 test machines.

> (In reply to Gerd Hoffmann from comment #4)
> > Can you also share 'lscpu' output of the host machine?
> 
> Anyone?

Please find it attached. My other machine is currently out of reach, but it is same generation Intel CPU. Can send you the output tomorrow if needed.

(In reply to Gerd Hoffmann from comment #9)
> https://copr.fedorainfracloud.org/coprs/kraxel/edk2.testbuilds/
> please try 20230825-24 (still building right now).
> please attach the firmware log (even if the build works).

Works like a charm for me! Thanks a lot! Will try it on my other machine tomorrow.

Comment 11 laolux 2023-10-11 01:24:53 UTC
Created attachment 1993393 [details]
lscpu output from second machine

lscpu output of second affected machine attached.
On this machine as well, the copr version fixes the issue. Thanks again!

Comment 12 Adam Williamson 2023-10-11 01:32:56 UTC
sorry, it's been a bit chaotic around here :( Indeed, the COPR build fixes it here too. Attaching my lscpu output, and the firmware log.

Comment 13 Adam Williamson 2023-10-11 01:33:22 UTC
Created attachment 1993394 [details]
lscpu output

Comment 14 Adam Williamson 2023-10-11 01:34:37 UTC
Created attachment 1993395 [details]
firmware log with working COPR build

Comment 15 Gerd Hoffmann 2023-10-11 09:14:01 UTC
https://github.com/kraxel/edk2/commit/35c6cfd69232e8eb271a48c9c0b2ad4ffe05978e

New test builds with the (hopefully) final fix coming your way (same copr, version 20230825-25).
This time it is enough to know whenever the guests boot fine.

Seems to happen on latest intel hardware only, all lscpu outputs show 12gen intel processors, probably due to new cpuid leafs being added for that generation and virt-manager using cpu=host-passthrough by default.

Comment 16 Gerd Hoffmann 2023-10-11 10:08:16 UTC
https://edk2.groups.io/g/devel/message/109523

Comment 17 Adam Williamson 2023-10-11 20:43:55 UTC
yup, the -25 build boots fine too. Thanks.

Comment 18 laolux 2023-10-12 00:59:31 UTC
Can confirm, the -25 build works on both my machines. Thanks a lot!

Comment 19 Fedora Update System 2023-10-12 12:21:16 UTC
FEDORA-2023-1215290ed6 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-1215290ed6

Comment 20 Fedora Update System 2023-10-13 02:14:19 UTC
FEDORA-2023-1215290ed6 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-1215290ed6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-1215290ed6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 21 Laszlo Ersek 2023-10-25 10:10:27 UTC
(In reply to Gerd Hoffmann from comment #16)
> https://edk2.groups.io/g/devel/message/109523

Upstream commit 170d4ce8e90abb1eff03852940a69c9d17f8afe5.

Comment 22 Fedora Update System 2023-11-03 18:36:19 UTC
FEDORA-2023-1215290ed6 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.