Bug 2113005 - Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on some boards [NEEDINFO]
Summary: Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: shim
Version: 38
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://ask.fedoraproject.org/t/commo...
: 2103785 2143343 2243264 (view as bug list)
Depends On:
Blocks: BetaBlocker, F40BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2022-08-01 16:56 UTC by Gilles Buloz
Modified: 2024-02-27 22:07 UTC (History)
53 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:
bcotton: fedora_prioritized_bug-
awilliam: needinfo? (pjones)
kraxel: needinfo? (pjones)


Attachments (Terms of Use)
Additional tests (9.49 KB, text/plain)
2022-08-23 14:19 UTC, Gilles Buloz
no flags Details
with debug enabled by mok on shimx64.efi 15.6 (55.77 KB, text/plain)
2022-08-23 15:17 UTC, Gilles Buloz
no flags Details
Auto boot from custom live image with mok debug (26.30 KB, text/plain)
2022-08-23 15:29 UTC, Gilles Buloz
no flags Details
boot from bootx64 from shell on custom live image with mok debug (54.00 KB, text/plain)
2022-08-23 15:31 UTC, Gilles Buloz
no flags Details
Auto boot from official F36 live image with mok debug (boot OK; have to press ENTER in debug messages) (18.20 KB, text/plain)
2022-08-23 15:39 UTC, Gilles Buloz
no flags Details
Test of test of shimx64.efi from shim-unsigned-x64-15.6-505.x86_64.rpm with secure boot disabled (131.99 KB, text/plain)
2022-08-26 17:07 UTC, Gilles Buloz
no flags Details

Description Gilles Buloz 2022-08-01 16:56:07 UTC
Description of problem:
On some boards (?), a custom live image made with /boot/efi/EFI/BOOT/BOOTX64.EFI from shim-x64-15.6-2.x86_64 fails to boot, but replacing the BOOTX64.EFI on this image with the one from shim-x64-15.4-5.x86_64 fixes the problem.
Booting a standard Fedora 36 Live image (tested Fedora-Xfce-Live-x86_64-36-1.5.iso) on such a board works OK, but this image is using shim-x64-15.4-5.x86_64.

Version-Release number of selected component (if applicable):
shim-x64-15.6-2.x86_64

How reproducible:
Easy with the right x86_64 board
Tested with a Kontron VX3040 board (Quad core Intel Core i7-3517UE) with UEFI AMI BIOS

Steps to Reproduce:
1.Rebuild a live image (tested XFce) using livemedia-creator from latest Fedora sources
2.Try to boot the image
3.An error is displayed

Actual results:
Invalid image 
Failed to read header: Unsupported 
Failed to load image: Unsupported 
start_image() returned Unsupported

Expected results:
GRUB successfully loaded

Additional info:
As workaround I can enter the UEFI shell, enter the filesystem (map -r + fs0:), then cd EFI\BOOT, then run GRUB manually with : GRUBX64.EFI <enter>
But running BOOTX64.EFI gives the same errors than when the BIOS is trying to boot automatically from boot list (see 'Actual results")
Strangely, once booted with the workaround above and installed to disk, the disk boots successfully even with this /boot/efi/EFI/BOOT/BOOTX64.EFI from shim-x64-15.6-2.x86_64

Comment 1 Robbie Harwood 2022-08-18 21:16:59 UTC
What if you try to boot shimx64.efi from the UEFI shell?

Comment 2 Gilles Buloz 2022-08-23 08:44:42 UTC
Boot fails also with shimx64.efi :

fs0:\EFI\FEDORA> SHIMX64.EFI
Invalid image
Failed to read header: Unsupported
Failed to load image: Unsupported
start_image() returned Unsupported
start_image() returned Unsupported

fs0:\EFI\FEDORA>

Comment 3 Gilles Buloz 2022-08-23 14:19:15 UTC
Created attachment 1907142 [details]
Additional tests

Comment 4 Gilles Buloz 2022-08-23 14:23:00 UTC
Additional infos : 
- SHIMX64.EFI is not present on EFI partition of live image; only once installed to disk (under EFI\FEDORA)
- Fedora 36 official live image can boot from \EFI\BOOT\BOOTX64.EFI
- Fedora 36 custom live image can not boot from \EFI\BOOT\BOOTX64.EFI but can boot from \EFI\BOOT\GRUBX64.EFI
- Once installed to disk, the custom Fedora 36 live image can not boot from \EFI\FEDORA\SHIMX64.EFI but can boot from \EFI\BOOT\BOOTX64.EFI
See attached f36gruberrdbg_3.txt (attachment 1907142 [details])

Comment 5 Gilles Buloz 2022-08-23 14:31:48 UTC
On Fedora 36 official live image, \EFI\BOOT\BOOTX64.EFI has same checksum and size (13999 / 928592) than the one in shim-x64-15.4-5.x86_64.rpm (fedora release)
On Fedora 36 custom live image and once installed to disk, \EFI\BOOT\BOOTX64.EFI has same checksum and size  (23754 / 946712) than the one in shim-x64-15.6-2.x86_64.rpm (fedora updates)

Comment 6 Robbie Harwood 2022-08-23 14:55:40 UTC
Can you try (in a booted system that works):

mokutil --set-verbosity true
mokutil --set-fallback-verbosity true

and then reboot into the image with 15.6?  This should result in more debug output.

Comment 7 Gilles Buloz 2022-08-23 15:17:51 UTC
Created attachment 1907157 [details]
with debug enabled by mok on shimx64.efi 15.6

Comment 8 Gilles Buloz 2022-08-23 15:18:23 UTC
OK, on a system running from a disk installed from the custom live image, I ran you commands and attempted to reboot from SHIMX64.EFI (from 15.6-2). See attachment 1907157 [details]

Comment 9 Gilles Buloz 2022-08-23 15:29:35 UTC
Created attachment 1907158 [details]
Auto boot from custom live image with mok debug

Comment 10 Gilles Buloz 2022-08-23 15:31:13 UTC
Created attachment 1907159 [details]
boot from bootx64 from shell on custom live image with mok debug

Comment 11 Gilles Buloz 2022-08-23 15:39:52 UTC
Created attachment 1907160 [details]
Auto boot from official F36 live image with mok debug (boot OK; have to press ENTER in debug messages)

Comment 12 Robbie Harwood 2022-08-23 16:12:32 UTC
Thanks.

I may have a fix: https://github.com/rhboot/shim/pull/505 .  If you're able to test unsigned shims, it would be helpful if you could give that a spin.

Comment 13 Gilles Buloz 2022-08-23 16:26:14 UTC
Ok, I would be glad to help for testing, but how can I do that ? Would you send me an updated shimx64.efi ?

Comment 14 Robbie Harwood 2022-08-25 18:47:26 UTC
Well, the tricky part is signing it, not building it.  You either need your own key enrolled to sign with, or secure boot enforcement disabled.

But if it helps, and you feel like running binaries linked by quasi-strangers on the internet, you can extract built, unsigned shims from /usr/share/shim of the RPM at: https://rharwood.fedorapeople.org/shim-505/shim-unsigned-x64-15.6-505.x86_64.rpm

Comment 15 Gilles Buloz 2022-08-26 17:07:59 UTC
Created attachment 1907921 [details]
Test of test of shimx64.efi from shim-unsigned-x64-15.6-505.x86_64.rpm with secure boot disabled

I've extracted the shimx64.efi from your shim-unsigned-x64-15.6-505.x86_64.rpm test package and copied it to an installed disk under EFI/fedora/shimx64_156505.efi. I also copied version 15.6-2 under shimx64_1562.efi and 15.4-5 under shimx64_1545.efi to compare.
Then I've tried to run them manually from EFI shell with secure boto disabled :
shimx64_1545.efi : OK (grub is run)
shimx64_1562.efi : FAILS
shimx64_156505.efi : OK

Comment 16 Robbie Harwood 2022-09-01 19:49:43 UTC
Thanks for the testing.  For anyone hitting this issue before Fedora's signed shim updates: in my opinion, it is better from a security perspective to run the older, signed shim rather than disabling secureboot to run the patched, unsigned shim linked above.  

(If you're able to enroll your own keys, of course you can have the best of both worlds; and if you're just testing fixes, none of this applies anyway.)

Comment 17 Frederick Grose 2022-09-06 21:56:30 UTC
I observe the same error messages booting Fedora 37 images on an Asus Z87-PLUS motherboard.
see https://bugzilla.redhat.com/show_bug.cgi?id=2103785

Substituting the Fedora 36 
Mar 25 20:37 grubx64.efi
May  5  2021 BOOTX64.EFI
files solves the problem.

Comment 18 Adam Williamson 2022-10-25 17:45:20 UTC
*** Bug 2103785 has been marked as a duplicate of this bug. ***

Comment 19 Adam Williamson 2022-10-25 17:46:17 UTC
It's a shame we have a fix for this but are releasing F37 with it :( Sorry, I kinda took my eye off this ball a bit. Robbie, when are we scheduled to do another signed shim build?

Comment 20 Kamil Páral 2022-11-10 11:43:46 UTC
I've created a CommonBugs description here:
https://ask.fedoraproject.org/t/common-issues/28364

Can somebody please proof-read it?
(I only tried to advise workarounds usable for a general user. Approaches like using the UEFI shell can power users easily discover when reading through this bug report).

Most importantly, does this problem only affect booting install media, or does it also affect upgraded systems (F36->F37)? IOW, can a system upgrade make your system unbootable?

Comment 21 Frederick Grose 2022-11-10 12:38:42 UTC
I observe the same error messages booting the latest Rawhide .iso, Fedora-Workstation-Live-x86_64-Rawhide-20221109.n.0.iso
on my Asus Z87-PLUS motherboard.

Comment 22 Adam Williamson 2022-11-10 16:27:59 UTC
I think most likely an upgrade could break it, yes, as the affected files are part of shim-signed and typically Fedora boots with /boot/efi mounted and writeable.

It's probably worth documenting the "replace the files with ones from the older shim package" workaround too, for folks who want to do a UEFI install, or are upgrading a UEFI system.

I wish I'd thought to make this bug a higher priority :| Maybe we could've at least just downgraded to the older working shim for F37, or something, even if we couldn't get a new signed build done in time for release. I hate to do it, but I'm actually gonna throw this on the blocker list for the go/no-go, at least to let people chew it over and raise visibility.

Comment 23 Adam Williamson 2022-11-10 17:33:44 UTC
Just realized I never actually mentioned this, but my Asus H97M-E - which I believe is likely of similar vintage to Frederick's Z87-PLUS - is also affected. So the total list of affected boards so far is:

* Kontron VX3040
* Asus Z87-PLUS
* Asus H97M-E

nirik also found this forum post by another affected person: https://ask.fedoraproject.org/t/f37-invalid-image-error-while-booting/26632/3

Comment 24 Adam Williamson 2022-11-10 18:05:48 UTC
So, good news on the upgrade front. I tested, and this bug does not actually seem to affect installed systems. If you install F36 and update it, you already get the 'bad' shim, because it's in F36 (and F35) updates as well as in F37. But the system still boots fine. If you then upgrade to F37, the system still boots fine.

So it seems, for some reason, this bug only affects the live boot scenario. I don't know why that would be, but it's good news. Robbie, does that make sense to you?

Comment 25 Ben Cotton 2022-11-10 18:19:13 UTC
In today's Go/No-Go meeting, we agreed to reject this on the basis that this bug appears to be limited to specific hardware and may only affect booting live images, so it does not rise to the level of a release blocker

https://meetbot.fedoraproject.org/fedora-meeting/2022-11-10/f37-final-go_no_go-meeting.2022-11-10-17.02.log.html#l-280

Comment 26 Tomas Dolezal 2022-11-12 13:50:02 UTC
Just reporting that my main computer with following mainboard is also affected. Not only live images but also netinst (Everything "variant") has this issue.

Asus Z97-DELUXE/USB 3.1 (bought 2015/07)

Comment 27 Frederick Grose 2022-11-16 13:56:35 UTC
This workaround allowed me to boot a LiveUSB:  (from https://www.reddit.com/r/RockyLinux/comments/r49er0/comment/hs6mll2/?utm_source=share&utm_medium=web2x&context=3)

In the ESP, delete the original BOOTX64.EFI and BOOTIA32.EFI. Then 
Rename grubx64.efi and grubia32.efi to BOOTX64.EFI and BOOTIA32.EFI respectively.

Comment 28 Adam Williamson 2022-11-16 16:42:00 UTC
That's not the best idea, because it will only work with Secure Boot disabled. The best workaround is the one documented at https://ask.fedoraproject.org/t/f37-install-media-dont-boot-in-uefi-mode-on-certain-motherboards/28364 - replacing BOOTX64.EFI with the one from an older package. You don't need to worry about BOOTIA32.EFI.

Comment 29 Gianluca 2022-11-24 18:33:03 UTC
Just for information, I have an MSI H81M-E33 and that is affected too.

Comment 30 Matthew Murrian 2022-12-07 16:35:40 UTC
Also affects to Acer E5-575.

Comment 31 Dan Horák 2022-12-11 13:18:08 UTC
for the record, Lenovo ThinkCentre E73 is also affected

RHEL 9 is suffering from the same issue it seems to me.

Comment 32 Jonathan Teh 2022-12-20 23:32:09 UTC
To add to the list, this also affects Asus AM1I-A (latest AMI UEFI BIOS 0801).

Comment 33 Brandon Nielsen 2022-12-22 19:06:10 UTC
Intel DH87RL is affected.

Comment 34 rx2178 2023-01-21 10:39:33 UTC
Gigabyte GA-6LXSV is affected.  Though, I was trying to boot RHEL 9.1 (dvd and live boot images).  I worked around this by deleteing BOOTX64.efi and renaming GRUBX64.efi in its place in the efi partition.

Comment 35 Maurice 2023-02-06 17:48:33 UTC
Adding to the list:

My DELL G5 5590 with its latest UEFI version 1.22.0 is also affected. Will be installing Fedora 36 and upgrading to 37 as a workaround. Annoying to learn that this is not getting fixed.

Comment 36 Koustubh Sinkar 2023-02-15 22:10:48 UTC
ACER s1001. 

The problem that I face is similar but it has a slightly different twist with regards to the shim. The processor on my laptop is 64bit but the EFI BIOS is 32bit. I tried after replacing the 32bit shim from the centos 32bit shim but it didn’t work.

Surprisingly Fedora 31 (64 bit) is able to boot with UEFI enabled as well as disabled.

*In summary the 32 bit shim also needs to be corrected for Fedora 38*

Comment 37 Adam Williamson 2023-02-15 22:40:39 UTC
We did do special handling for 32-bit UEFI firmwares for several releases, but it's entirely possible that's either been dropped or just broken lately and nobody has noticed, because they're a very small weird category of systems. I used to have one but don't any more, for instance, so I wouldn't notice if it broke.

That would be a different bug from this one, though, so it should be filed separately.

Comment 38 RM 2023-02-25 10:26:51 UTC
Add Asus Z87-A.

Comment 39 Vitaly Zaitsev 2023-03-11 13:34:55 UTC
Can be reproduced with Fedora 38 Beta. Changing Fedora version to 38 and proposing this issue as the Final blocker.

Forums discussion: https://discussion.fedoraproject.org/t/f37-invalid-image-error-while-booting/75921

Comment 40 Fedora Blocker Bugs Application 2023-03-11 13:38:50 UTC
Proposed as a Blocker for 38-final by Fedora user xvitaly using the blocker tracking app because:

 Modern Fedora ISOs, including Fedora 38 Beta, have the following error 
on legacy hardware (tested on Haswell) when booted in UEFI mode:

```
Invalid image
Failed to read header: Unsupported
Failed to load image: Unsupported
start_image() returned Unsupported
```

A lot of users have this problem:
https://discussion.fedoraproject.org/t/f37-invalid-image-error-while-booting/75921

It needs to be fixed or documented that Fedora can no longer be installed in UEFI mode on legacy hardware.

Comment 41 František Zatloukal 2023-03-12 13:43:52 UTC
Removing RejectedBlocker as it was effective in F37 cycle. However, I expect that "this bug appears to be limited to specific hardware and may only affect booting live images, so it does not rise to the level of a release blocker" would still stand.

Comment 42 RM 2023-03-13 03:53:10 UTC
How long does the list of impacted motherboards have to grow before this rises to the level of release blocker?

Comment 43 Adam Williamson 2023-03-13 05:56:04 UTC
Since there are several options to work around it, and new builds of shim are a giant pain (we probably don't have time for one by F38 release already), I'd say pretty long.

Robbie, Peter, do we have a schedule for the next shim build?

Comment 44 Vitaly Zaitsev 2023-03-13 08:06:52 UTC
> Since there are several options to work around it, and new builds of shim are a giant pain (we probably don't have time for one by F38 release already), I'd say pretty long.

This issue was reported more than 6 months ago and still hasn't been fixed. Without a blocker, this won't be fixed in the foreseeable future.

Comment 45 Adam Williamson 2023-03-13 16:38:11 UTC
That doesn't follow. It hasn't been fixed because, as I explained, doing shim builds is a giant PITA so we don't do them often. They more or less happen on a schedule that is not exactly related to the Fedora schedule. This is because they have to go through the Secure Boot signing process.

Comment 46 Robbie Harwood 2023-03-13 17:24:54 UTC
To quote pjones: "right now we're very unlikely to do a build until v5 of this patch series is upstream in kernel: https://lore.kernel.org/lkml/cover.1671098103.git.baskov@ispras.ru/ ... (because it would require setting a compatibility flag that we're not actually compatible with)"

Comment 47 Adam Williamson 2023-03-13 17:34:57 UTC
Thanks for the info. Is there...nothing *smaller* we can do here? it's rather unfortunate that we've shipped one release that doesn't boot on 11 (so far) platforms and are on track to ship another one, when the previous build of shim *did* work fine. Can we just revert whatever change it was caused this, or is it more complicated than that?

Comment 48 Robbie Harwood 2023-03-13 18:45:47 UTC
I'm not sure what you're asking.  shim's signed - any changes to it invalidate the signature, which is why it's signed in the first place.

Comment 49 Robbie Harwood 2023-03-13 18:46:39 UTC
If any x86 kernel contributors happen to be reading: please merge the patch series already.

Comment 50 Geoffrey Marr 2023-03-13 19:20:32 UTC
Discussed during the 2023-03-13 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedBlocker (Final)" was made as a conditional violation of "All release-blocking images must boot in their supported configurations" when booting in native UEFI mode on affected hardware.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-03-13/f38-blocker-review.2023-03-13-16.00.txt

Comment 51 Adam Williamson 2023-03-13 21:29:20 UTC
I'm asking if there's something we can do to get a build with some kind of fix for this done and signed and released without waiting on upstream to merge a large kernel patch. I'm unclear on how the large upstream kernel patch and shim are related exactly, but from the point of view of this bug, the fix was apparently merged last September:

https://github.com/rhboot/shim/pull/505

but there has not been a Fedora shim build since last July, so this bug has been in Fedora all along. We released Fedora 37 with this bug (unfortunately), and if nothing changes, we'll also release Fedora 38 with this bug. That's the situation I'm hoping to avoid.

Comment 52 Robbie Harwood 2023-03-14 14:29:47 UTC
Again, one can't just build shim and have it work for secureboot.  We have to get it signed by Microsoft for that.  Microsoft's requirements now include NX support.  kernel patches for NX support exist, and have existed for months now.

If you want the bug fixed, please apply pressure to the people who can actually fix it (kernel), not us.

Comment 53 Adam Williamson 2023-03-14 15:52:00 UTC
Ah, that was new information to us. I know you have to get shim builds signed by Microsoft, but had never heard of the NX requirement.

Comment 54 Geraldo Simião 2023-03-15 02:40:02 UTC
It seems this requirement was announced on 2021, effective 11/30/2022
https://techcommunity.microsoft.com/t5/hardware-dev-center/updated-uefi-signing-requirements/ba-p/1062916

Comment 55 sachinwebgraphics 2023-03-17 04:56:30 UTC
the bug is still present in fedora beta 38th build iso's. https://bugzilla.redhat.com/show_bug.cgi?id=2143343 i have already reported this earlier.

Comment 56 Adam Williamson 2023-03-17 06:16:20 UTC
We know. That's why it's still NEW, not MODIFIED or ON_QA. You don't need to keep reporting it, thanks.

Comment 57 sachinwebgraphics 2023-03-17 07:19:11 UTC
(In reply to Adam Williamson from comment #56)
> We know. That's why it's still NEW, not MODIFIED or ON_QA. You don't need to
> keep reporting it, thanks.

Well, if its still now resolved is it bad to report and still update team? Being a responsible member and user of a opensource, reporting bugs is desired right? I was checking or not asking any ETA's here when it will be resolved.

Comment 58 Michael J Gruber 2023-03-17 09:09:13 UTC
(In reply to sachinwebgraphics from comment #57)
> (In reply to Adam Williamson from comment #56)
> > We know. That's why it's still NEW, not MODIFIED or ON_QA. You don't need to
> > keep reporting it, thanks.
> 
> Well, if its still now resolved is it bad to report and still update team?
> Being a responsible member and user of a opensource, reporting bugs is
> desired right? I was checking or not asking any ETA's here when it will be
> resolved.

Yes, reporting bugs is very much appreciated, and I'm sure Adam's "thanks" refers to that, too.

As an additional info, he explained bug status codes which are always capitalised (not to shout ...). We don't have an explicit "confirmed", but the reports above confirm it as well as a long term (because we rely on upstream) solution. That's what we know. We don't have a quick solution yet, only the mentioned workarounds.

At this point, additional reports can only help as much as point out the extent of the problem, by adding information about an affected board which has not been reported already.

Comment 59 Adam Williamson 2023-03-17 15:49:14 UTC
If the bug goes quiet for a long time, or is about to be automatically closed for inactivity, it can be useful to say 'hey, this is still happening'. But when the bug is clearly active, with 15 posts this month before yours, it's not really needed - we wouldn't be debating this and putting it on blocker lists and pinging people if we didn't already know it was still happening =)

Comment 60 Sergei LITVINENKO 2023-03-18 19:36:35 UTC
PU551LD (ASUS-NotebookSKU) is affected too

Comment 61 Adam Williamson 2023-03-27 17:27:48 UTC
*** Bug 2143343 has been marked as a duplicate of this bug. ***

Comment 62 Kamil Páral 2023-04-05 07:01:02 UTC
Nominating as a Prioritized bug. I'm aware that it will probably not make resolving this bug any faster, but I think it should be on that list.

Comment 63 Gary Buhrmaster 2023-04-07 16:42:36 UTC
(In reply to Robbie Harwood from comment #52)
> Again, one can't just build shim and have it work for secureboot.  We have
> to get it signed by Microsoft for that.  Microsoft's requirements now
> include NX support.  kernel patches for NX support exist, and have existed
> for months now.


To be slightly fair, v5 of the patch series has only been available for less than a month, and was still receiving a few review comments (mostly nits at this time last I looked).


> If you want the bug fixed, please apply pressure to the people who can
> actually fix it (kernel), not us.


Would it be possible to get a bugzilla created referencing the required enhancement that can be assigned to the kernel team[0][1], and have this bug depend on that bugzilla (rather than just a reference in the text of this bugzilla)?  That would make it clearer where the initial work needs to happen.


Thanks.



[0] If there is not already such a ticket.  I tried to do a search, but could not find such a ticket.

[1] From reports in this ticket, this also impacts the EL side of RH, so there may be ticket elsewhere.

Comment 64 Kamil Páral 2023-04-14 08:21:22 UTC
During Fedora 38 Final Go/No-Go meeting, this blocker bug has been waived under the "difficult to fix" exception, and has become a Fedora 39 Beta blocker. Please, let's do our best to fix it by then, thank you.

Comment 65 Ben Cotton 2023-04-19 14:56:27 UTC
In today's prioritized bugs meeting, we agreed to reject this as it's already an accepted F39 Beta blocker, and that's a bigger hammer
https://meetbot.fedoraproject.org/fedora-meeting-1/2023-04-19/fedora_prioritized_bugs_and_issues.2023-04-19-14.00.log.html#l-92

Comment 66 Thomas Crider 2023-04-21 21:26:17 UTC
(In reply to Kamil Páral from comment #64)
> During Fedora 38 Final Go/No-Go meeting, this blocker bug has been waived
> under the "difficult to fix" exception, and has become a Fedora 39 Beta
> blocker. Please, let's do our best to fix it by then, thank you.

I'm not sure why it's marked difficult to fix when there was already a proposed, confirmed, and merged upstream fix:

https://github.com/rhboot/shim/pull/505

Is it not possible to just bump the shim version to that of upstream containing the fix? (15.7)

Apologies if I've missed something already mentioned that otherwise blocks this.

Comment 67 Adam Williamson 2023-04-21 21:32:27 UTC
The explanation is above, see comments #51 and #52.

Comment 68 Thomas Crider 2023-04-21 21:35:06 UTC
Ah I completely missed that, thanks and again sincere apologies.

Comment 69 Adam Williamson 2023-07-23 22:45:43 UTC
Peter / Robbie, can we have an update on this for F39? It is currently an F39 Beta blocker, as that's how waiving works. Does it look like we will be able to get this done for F39 Beta? If not, what about Final?

Comment 70 comsec 2023-08-09 20:08:25 UTC
This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and 8.8 also with base install image on Fujitsu Primergy RX300 S8

Comment 71 comsec 2023-08-09 20:10:25 UTC
(In reply to comsec from comment #70)
> This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and
> 8.8 also with base install image on Fujitsu Primergy RX300 S8

RHEL7/UEFI: Boot hangs after updating shim to "shim-x64-15.6-3.el7_9"

https://access.redhat.com/solutions/7016419

Comment 72 comsec 2023-08-09 20:38:09 UTC
(In reply to comsec from comment #71)
> (In reply to comsec from comment #70)
> > This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and
> > 8.8 also with base install image on Fujitsu Primergy RX300 S8
> 
> RHEL7/UEFI: Boot hangs after updating shim to "shim-x64-15.6-3.el7_9"
> 
> https://access.redhat.com/solutions/7016419

rhel 8.6 with shim-x64-15.5-2.el8.x86_64 same instal problem

Comment 73 comsec 2023-08-10 14:34:58 UTC
(In reply to comsec from comment #71)
> (In reply to comsec from comment #70)
> > This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and
> > 8.8 also with base install image on Fujitsu Primergy RX300 S8
> 
> RHEL7/UEFI: Boot hangs after updating shim to "shim-x64-15.6-3.el7_9"
> 
> https://access.redhat.com/solutions/7016419

Rhel 8.5 finally works and seems to install just fine and it has shim-x64-15.4-2.el8_1.x86_64

So this bug on red hat 8 was introduced in shim 15.5

Comment 74 Adam Williamson 2023-08-29 00:34:02 UTC
CCing nfrayer, who I guess is looking after some bootloader stuff now Robbie's not around any more? (At least, he did the last build of grub2). Peter, Nicolas (if this is indeed your realm - apologies if not, please feel free to clear needinfo and un-CC yourself), we could really do with a status update here as we're past F39 Beta freeze.

To remind everyone of the story so far: we have a fix for this bug, but the problem - AIUI - is that we cannot get a new shim build signed by Microsoft without getting NX support added to the kernel. So we can't fix this bug until that's done. See https://bugzilla.redhat.com/show_bug.cgi?id=2113005#c52 .

Comment 75 Adam Williamson 2023-08-29 00:44:51 UTC
So, here's what I figured out myself. The promised v5 of Evgeny Baskov's patch set was posted in March and discussed a bit:

https://lore.kernel.org/lkml/8493680a-0bad-43de-a7a0-caa48e430139@uncooperative.org/#r

there are some good posts from Gerd Hoffman (kraxel) - adding him to CC too, what the hell! - and Peter explaining various aspects of this:

https://lore.kernel.org/lkml/20230315090457.6spo4f4v2l4qwdu2@sirius.home.kraxel.org/
https://lore.kernel.org/lkml/8493680a-0bad-43de-a7a0-caa48e430139@uncooperative.org/#t

and some pretty run-of-the-mill feedback along the lines of "make your commit messages better", but there was no obvious follow-up. However, there is a new recent patchset from Ard Biesheuvel called "x86/boot: Rework PE header generation":

https://lore.kernel.org/lkml/20230818134422.380032-1-ardb@kernel.org/

which is described as superseding Evgeniy's work: "This supersedes the work proposed by Evgeniy last year, which did a major rewrite of the build tool in order to clean it up, before updating it to generate the new 4k aligned image layout. As this series proves, the build tool is mostly unnecessary, and we have too many of those already."

That was posted on August 18, so it's very new. Peter, Nicolas, Gerd, anyone else who knows - how do the prospects look for this patchset? I'm assuming it's pretty late to try and land it for F39 Beta, but you never know. How about F39 Final?

Comment 76 Gerd Hoffmann 2023-08-29 06:24:09 UTC
The more important series (also by Ard) is this one:
https://lore.kernel.org/lkml/20230802154831.2147855-1-ardb@kernel.org/

Comment 77 Adam Williamson 2023-08-29 06:30:41 UTC
Thanks.

Is there some kind of *contest* to give these patch series names which give absolutely no hint that they have anything to do with nx support or secure boot signoff? :D

Comment 78 Gerd Hoffmann 2023-08-29 06:35:20 UTC
(In reply to Gerd Hoffmann from comment #76)
> The more important series (also by Ard) is this one:
Small correction after reading the cover letter again: I think we need both patch series from Ard Biesheuvel.

Comment 79 Gerd Hoffmann 2023-08-29 06:45:50 UTC
(In reply to Adam Williamson from comment #77)
> Is there some kind of *contest* to give these patch series names which give
> absolutely no hint that they have anything to do with nx support or secure
> boot signoff? :D

Dunno.  Maybe it is just an attempt to avoid feeding the secure boot bashing trolls.

Comment 80 Nicolas Frayer 2023-08-29 10:05:13 UTC
Hi Adam,

I'm not directly involved in shim NX implementation but I do know, as you mentioned in your comment, that there is also a dependency on the kernel NX patchset.
I think Peter Jones might have more information on the status.

Comment 81 Gerd Hoffmann 2023-08-29 11:44:41 UTC
Update:
The decompresser series (https://lore.kernel.org/lkml/20230802154831.2147855-1-ardb@kernel.org/) was just pulled by linus and unless it broke something horribly it should be in 6.6.
The PE header series (https://lore.kernel.org/lkml/20230818134422.380032-1-ardb@kernel.org/) will NOT make it in the 6.6 merge window, but hopefully 6.7.
Additional issue to be solved (probably downstream): SBAT and managing the generation numbers.

Comment 82 Adam Williamson 2023-08-29 15:18:56 UTC
Thanks Gerd!

Is there any prospect we could backport the patches to Fedora to make it possible to get an updated shim into F39? Or are we targeting F40 at this point?

Comment 83 Peter Robinson 2023-08-29 15:38:53 UTC
> Is there any prospect we could backport the patches to Fedora to make it
> possible to get an updated shim into F39? Or are we targeting F40 at this
> point?

The stuff that landed upstream for 6.6 should be possible, depending on when the rest lands, we obviously don't want to have to hold patches long term that will never land.

Comment 84 raravind 2023-09-04 18:10:29 UTC
Adam,
The decompresser series (v9) - https://lore.kernel.org/lkml/20230807162720.545787-1-ardb@kernel.org/ is the latest v9 patch ,I think it is upstreamed?what Gerd wrote on comment#81 is v8
rest all I'm not sure.

Comment 85 Per Sjoholm 2023-09-05 06:53:53 UTC
(In reply to Adam Williamson from comment #22)
> I think most likely an upgrade could break it, yes, as the affected files

Confirmd, updated F38 on Asus laptop.

Comment 86 Gerd Hoffmann 2023-09-12 09:30:39 UTC
> The PE header series
> (https://lore.kernel.org/lkml/20230818134422.380032-1-ardb@kernel.org/) will
> NOT make it in the 6.6 merge window, but hopefully 6.7.

v2 of the pe header series posted:
https://lore.kernel.org/all/20230912090051.4014114-17-ardb@google.com/

Comment 87 Adam Williamson 2023-09-14 18:41:29 UTC
This was discussed at today's F39 Beta Go/No-Go meeting:
https://meetbot-raw.fedoraproject.org/fedora-meeting/2023-09-14/f39-beta-go_no_go-meeting.2023-09-14-17.00.html
we agreed to again waive this blocker under the "Difficult-to-fix blocker bugs" category at https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases . As before, we cannot practically fix this until the kernel patch sets have been reviewed and accepted upstream.

This is waived to F39 Final for now. We're hoping a solution will be possible by Final.

Comment 88 Gerd Hoffmann 2023-09-18 06:34:18 UTC
(In reply to Gerd Hoffmann from comment #86)
> v2 of the pe header series posted:
> https://lore.kernel.org/all/20230912090051.4014114-17-ardb@google.com/

v3 has been posted (second half only, first half of v2 was merged already):
https://lore.kernel.org/all/20230915171623.655440-10-ardb@google.com/

Ingo Molnar applied the patches to tip:x86/boot,
so this is on track for the 6.7 merge window.

Comment 89 Adam Williamson 2023-10-03 16:20:56 UTC
So it seems like that v3 got merged.

Is anything now still waiting to be merged upstream? If not, do we possibly have time to fix this for F39 Final (which would require an update by about, say, Friday October 13), or do we need to push it to F40?

Comment 90 Adam Williamson 2023-10-03 16:25:42 UTC
well, it's merged to tip, at least.

Comment 91 Adam Williamson 2023-11-03 23:23:23 UTC
Discussed at the Fedora 39 Final go/no-go meeting - https://meetbot-raw.fedoraproject.org/fedora-meeting/2023-11-02/f39-final-go_no_go-meeting.2023-11-02-17.04.html . We agreed to waive this once more, as once more the fixes are not ready on a reasonable timeframe for the release; we are more or less at the mercy of upstream here. We are hopeful they will finally be ready for F40 Beta.

Comment 92 Gerd Hoffmann 2023-11-06 12:39:30 UTC
(In reply to Adam Williamson from comment #90)
> well, it's merged to tip, at least.

Now in 6.7-rc1.

Comment 93 Gerd Hoffmann 2023-11-06 12:42:33 UTC
(In reply to Gerd Hoffmann from comment #92)
> (In reply to Adam Williamson from comment #90)
> > well, it's merged to tip, at least.
> 
> Now in 6.7-rc1.

Well, in linus master branch, -rc1 is not tagged yet ...

Comment 94 Adam Williamson 2023-11-14 17:17:09 UTC
does that mean everything we need is now in upstream 6.7 at least, or are we still waiting on something?

Comment 95 Gerd Hoffmann 2023-11-15 11:35:28 UTC
Yes, it's upstream and meanwhile also in rawhide kernels:

kraxel@rawhide ~# pe-inspect /boot/vmlinuz-6.7.0-0.rc1.20231114git9bacdd8996c7.17.fc40.x86_64 
# file: /boot/vmlinuz-6.7.0-0.rc1.20231114git9bacdd8996c7.17.fc40.x86_64
#    section: file 0x00001000 +0x00003000  virt 0x00001000 +0x00004000  r-- (.setup)
#    section: file 0x00005000 +0x00e18000  virt 0x00005000 +0x00e18000  r-x (.text)
#    section: file 0x00e1d000 +0x00001200  virt 0x00e1d000 +0x00067000  rw- (.data)
#    section: file 0x00004000 +0x00001000  virt 0x00e84000 +0x00000008  r-- (.compat)
#    sigdata: addr 0x00e1e200 +0x00000d48
#       signature: len 0x5da, type 0x2
#          certificate
#             subject CN: Fedora Secure Boot Signer
#             issuer  CN: Fedora Secure Boot CA
#       signature: len 0x762, type 0x2
#          certificate
#             subject CN: kernel-signer
#             issuer  CN: fedoraca

Sections are nicely page-aligned, sections with different permissions do not
share pages any more, so page attributes can be used to enforce permissions.
Which is the requirement referenced in comment 46.

So kernel-wise things are green now, even though it will be a while until the
6.7 kernel lands in Fedora 38/39 updates.

The remaining work is in shim and possibly the signing infrastructure.  Not
fully sure what the exact requirements for shim are to be signed by microsoft.
I suspect one of them is that it'll boot fixed kernel only, and one option to
implement that would be to switch the kernel signer and blacklist the old one
in the new shim binary.

Peter?

Comment 96 Zbigniew Jędrzejewski-Szmek 2023-11-27 11:31:34 UTC
*** Bug 2243264 has been marked as a duplicate of this bug. ***

Comment 97 Gary Buhrmaster 2024-02-11 17:54:00 UTC
Any chance this will get resolved in time for the Fedora 40 release?

Comment 98 Adam Williamson 2024-02-11 18:11:42 UTC
Yes. I am trying to get more definite information, but there are various indications it will be. New shim-unsigned builds showed up last week, which I believe is a necessary precursor to getting the signed builds.

Comment 99 František Zatloukal 2024-02-23 10:46:34 UTC
It seems the sign request for shim 15.8 was submitted: https://github.com/rhboot/shim-review/issues/386


Note You need to log in before you can comment on or make changes to this bug.