Bug 2113005

What if you try to boot shimx64.efi from the UEFI shell?

Boot fails also with shimx64.efi :

fs0:\EFI\FEDORA> SHIMX64.EFI
Invalid image
Failed to read header: Unsupported
Failed to load image: Unsupported
start_image() returned Unsupported
start_image() returned Unsupported

fs0:\EFI\FEDORA>

Created attachment 1907142 [details]
Additional tests

Additional infos : 
- SHIMX64.EFI is not present on EFI partition of live image; only once installed to disk (under EFI\FEDORA)
- Fedora 36 official live image can boot from \EFI\BOOT\BOOTX64.EFI
- Fedora 36 custom live image can not boot from \EFI\BOOT\BOOTX64.EFI but can boot from \EFI\BOOT\GRUBX64.EFI
- Once installed to disk, the custom Fedora 36 live image can not boot from \EFI\FEDORA\SHIMX64.EFI but can boot from \EFI\BOOT\BOOTX64.EFI
See attached f36gruberrdbg_3.txt (attachment 1907142 [details])

On Fedora 36 official live image, \EFI\BOOT\BOOTX64.EFI has same checksum and size (13999 / 928592) than the one in shim-x64-15.4-5.x86_64.rpm (fedora release)
On Fedora 36 custom live image and once installed to disk, \EFI\BOOT\BOOTX64.EFI has same checksum and size  (23754 / 946712) than the one in shim-x64-15.6-2.x86_64.rpm (fedora updates)

Can you try (in a booted system that works):

mokutil --set-verbosity true
mokutil --set-fallback-verbosity true

and then reboot into the image with 15.6?  This should result in more debug output.

Created attachment 1907157 [details]
with debug enabled by mok on shimx64.efi 15.6

OK, on a system running from a disk installed from the custom live image, I ran you commands and attempted to reboot from SHIMX64.EFI (from 15.6-2). See attachment 1907157 [details]

Created attachment 1907158 [details]
Auto boot from custom live image with mok debug

Created attachment 1907159 [details]
boot from bootx64 from shell on custom live image with mok debug

Created attachment 1907160 [details]
Auto boot from official F36 live image with mok debug (boot OK; have to press ENTER in debug messages)

Thanks.

I may have a fix: https://github.com/rhboot/shim/pull/505 .  If you're able to test unsigned shims, it would be helpful if you could give that a spin.

Ok, I would be glad to help for testing, but how can I do that ? Would you send me an updated shimx64.efi ?

Well, the tricky part is signing it, not building it.  You either need your own key enrolled to sign with, or secure boot enforcement disabled.

But if it helps, and you feel like running binaries linked by quasi-strangers on the internet, you can extract built, unsigned shims from /usr/share/shim of the RPM at: https://rharwood.fedorapeople.org/shim-505/shim-unsigned-x64-15.6-505.x86_64.rpm

Created attachment 1907921 [details]
Test of test of shimx64.efi from shim-unsigned-x64-15.6-505.x86_64.rpm with secure boot disabled

I've extracted the shimx64.efi from your shim-unsigned-x64-15.6-505.x86_64.rpm test package and copied it to an installed disk under EFI/fedora/shimx64_156505.efi. I also copied version 15.6-2 under shimx64_1562.efi and 15.4-5 under shimx64_1545.efi to compare.
Then I've tried to run them manually from EFI shell with secure boto disabled :
shimx64_1545.efi : OK (grub is run)
shimx64_1562.efi : FAILS
shimx64_156505.efi : OK

Thanks for the testing.  For anyone hitting this issue before Fedora's signed shim updates: in my opinion, it is better from a security perspective to run the older, signed shim rather than disabling secureboot to run the patched, unsigned shim linked above.  

(If you're able to enroll your own keys, of course you can have the best of both worlds; and if you're just testing fixes, none of this applies anyway.)

I observe the same error messages booting Fedora 37 images on an Asus Z87-PLUS motherboard.
see https://bugzilla.redhat.com/show_bug.cgi?id=2103785

Substituting the Fedora 36 
Mar 25 20:37 grubx64.efi
May  5  2021 BOOTX64.EFI
files solves the problem.

*** Bug 2103785 has been marked as a duplicate of this bug. ***

It's a shame we have a fix for this but are releasing F37 with it :( Sorry, I kinda took my eye off this ball a bit. Robbie, when are we scheduled to do another signed shim build?

I've created a CommonBugs description here:
https://ask.fedoraproject.org/t/common-issues/28364

Can somebody please proof-read it?
(I only tried to advise workarounds usable for a general user. Approaches like using the UEFI shell can power users easily discover when reading through this bug report).

Most importantly, does this problem only affect booting install media, or does it also affect upgraded systems (F36->F37)? IOW, can a system upgrade make your system unbootable?

I observe the same error messages booting the latest Rawhide .iso, Fedora-Workstation-Live-x86_64-Rawhide-20221109.n.0.iso
on my Asus Z87-PLUS motherboard.

I think most likely an upgrade could break it, yes, as the affected files are part of shim-signed and typically Fedora boots with /boot/efi mounted and writeable.

It's probably worth documenting the "replace the files with ones from the older shim package" workaround too, for folks who want to do a UEFI install, or are upgrading a UEFI system.

I wish I'd thought to make this bug a higher priority :| Maybe we could've at least just downgraded to the older working shim for F37, or something, even if we couldn't get a new signed build done in time for release. I hate to do it, but I'm actually gonna throw this on the blocker list for the go/no-go, at least to let people chew it over and raise visibility.

Just realized I never actually mentioned this, but my Asus H97M-E - which I believe is likely of similar vintage to Frederick's Z87-PLUS - is also affected. So the total list of affected boards so far is:

* Kontron VX3040
* Asus Z87-PLUS
* Asus H97M-E

nirik also found this forum post by another affected person: https://ask.fedoraproject.org/t/f37-invalid-image-error-while-booting/26632/3

So, good news on the upgrade front. I tested, and this bug does not actually seem to affect installed systems. If you install F36 and update it, you already get the 'bad' shim, because it's in F36 (and F35) updates as well as in F37. But the system still boots fine. If you then upgrade to F37, the system still boots fine.

So it seems, for some reason, this bug only affects the live boot scenario. I don't know why that would be, but it's good news. Robbie, does that make sense to you?

In today's Go/No-Go meeting, we agreed to reject this on the basis that this bug appears to be limited to specific hardware and may only affect booting live images, so it does not rise to the level of a release blocker

https://meetbot.fedoraproject.org/fedora-meeting/2022-11-10/f37-final-go_no_go-meeting.2022-11-10-17.02.log.html#l-280

Just reporting that my main computer with following mainboard is also affected. Not only live images but also netinst (Everything "variant") has this issue.

Asus Z97-DELUXE/USB 3.1 (bought 2015/07)

This workaround allowed me to boot a LiveUSB:  (from https://www.reddit.com/r/RockyLinux/comments/r49er0/comment/hs6mll2/?utm_source=share&utm_medium=web2x&context=3)

In the ESP, delete the original BOOTX64.EFI and BOOTIA32.EFI. Then 
Rename grubx64.efi and grubia32.efi to BOOTX64.EFI and BOOTIA32.EFI respectively.

That's not the best idea, because it will only work with Secure Boot disabled. The best workaround is the one documented at https://ask.fedoraproject.org/t/f37-install-media-dont-boot-in-uefi-mode-on-certain-motherboards/28364 - replacing BOOTX64.EFI with the one from an older package. You don't need to worry about BOOTIA32.EFI.

Just for information, I have an MSI H81M-E33 and that is affected too.

Also affects to Acer E5-575.

for the record, Lenovo ThinkCentre E73 is also affected

RHEL 9 is suffering from the same issue it seems to me.

To add to the list, this also affects Asus AM1I-A (latest AMI UEFI BIOS 0801).

Intel DH87RL is affected.

Gigabyte GA-6LXSV is affected.  Though, I was trying to boot RHEL 9.1 (dvd and live boot images).  I worked around this by deleteing BOOTX64.efi and renaming GRUBX64.efi in its place in the efi partition.

Adding to the list:

My DELL G5 5590 with its latest UEFI version 1.22.0 is also affected. Will be installing Fedora 36 and upgrading to 37 as a workaround. Annoying to learn that this is not getting fixed.

ACER s1001. 

The problem that I face is similar but it has a slightly different twist with regards to the shim. The processor on my laptop is 64bit but the EFI BIOS is 32bit. I tried after replacing the 32bit shim from the centos 32bit shim but it didn’t work.

Surprisingly Fedora 31 (64 bit) is able to boot with UEFI enabled as well as disabled.

*In summary the 32 bit shim also needs to be corrected for Fedora 38*

We did do special handling for 32-bit UEFI firmwares for several releases, but it's entirely possible that's either been dropped or just broken lately and nobody has noticed, because they're a very small weird category of systems. I used to have one but don't any more, for instance, so I wouldn't notice if it broke.

That would be a different bug from this one, though, so it should be filed separately.

Add Asus Z87-A.

Can be reproduced with Fedora 38 Beta. Changing Fedora version to 38 and proposing this issue as the Final blocker.

Forums discussion: https://discussion.fedoraproject.org/t/f37-invalid-image-error-while-booting/75921

Proposed as a Blocker for 38-final by Fedora user xvitaly using the blocker tracking app because:

 Modern Fedora ISOs, including Fedora 38 Beta, have the following error 
on legacy hardware (tested on Haswell) when booted in UEFI mode:

```
Invalid image
Failed to read header: Unsupported
Failed to load image: Unsupported
start_image() returned Unsupported
```

A lot of users have this problem:
https://discussion.fedoraproject.org/t/f37-invalid-image-error-while-booting/75921

It needs to be fixed or documented that Fedora can no longer be installed in UEFI mode on legacy hardware.

Removing RejectedBlocker as it was effective in F37 cycle. However, I expect that "this bug appears to be limited to specific hardware and may only affect booting live images, so it does not rise to the level of a release blocker" would still stand.

How long does the list of impacted motherboards have to grow before this rises to the level of release blocker?

Since there are several options to work around it, and new builds of shim are a giant pain (we probably don't have time for one by F38 release already), I'd say pretty long.

Robbie, Peter, do we have a schedule for the next shim build?

> Since there are several options to work around it, and new builds of shim are a giant pain (we probably don't have time for one by F38 release already), I'd say pretty long.

This issue was reported more than 6 months ago and still hasn't been fixed. Without a blocker, this won't be fixed in the foreseeable future.

That doesn't follow. It hasn't been fixed because, as I explained, doing shim builds is a giant PITA so we don't do them often. They more or less happen on a schedule that is not exactly related to the Fedora schedule. This is because they have to go through the Secure Boot signing process.

To quote pjones: "right now we're very unlikely to do a build until v5 of this patch series is upstream in kernel: https://lore.kernel.org/lkml/cover.1671098103.git.baskov@ispras.ru/ ... (because it would require setting a compatibility flag that we're not actually compatible with)"

Thanks for the info. Is there...nothing *smaller* we can do here? it's rather unfortunate that we've shipped one release that doesn't boot on 11 (so far) platforms and are on track to ship another one, when the previous build of shim *did* work fine. Can we just revert whatever change it was caused this, or is it more complicated than that?

I'm not sure what you're asking.  shim's signed - any changes to it invalidate the signature, which is why it's signed in the first place.

If any x86 kernel contributors happen to be reading: please merge the patch series already.

Discussed during the 2023-03-13 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedBlocker (Final)" was made as a conditional violation of "All release-blocking images must boot in their supported configurations" when booting in native UEFI mode on affected hardware.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-03-13/f38-blocker-review.2023-03-13-16.00.txt

I'm asking if there's something we can do to get a build with some kind of fix for this done and signed and released without waiting on upstream to merge a large kernel patch. I'm unclear on how the large upstream kernel patch and shim are related exactly, but from the point of view of this bug, the fix was apparently merged last September:

https://github.com/rhboot/shim/pull/505

but there has not been a Fedora shim build since last July, so this bug has been in Fedora all along. We released Fedora 37 with this bug (unfortunately), and if nothing changes, we'll also release Fedora 38 with this bug. That's the situation I'm hoping to avoid.

Again, one can't just build shim and have it work for secureboot.  We have to get it signed by Microsoft for that.  Microsoft's requirements now include NX support.  kernel patches for NX support exist, and have existed for months now.

If you want the bug fixed, please apply pressure to the people who can actually fix it (kernel), not us.

Ah, that was new information to us. I know you have to get shim builds signed by Microsoft, but had never heard of the NX requirement.

It seems this requirement was announced on 2021, effective 11/30/2022
https://techcommunity.microsoft.com/t5/hardware-dev-center/updated-uefi-signing-requirements/ba-p/1062916

the bug is still present in fedora beta 38th build iso's. https://bugzilla.redhat.com/show_bug.cgi?id=2143343 i have already reported this earlier.

We know. That's why it's still NEW, not MODIFIED or ON_QA. You don't need to keep reporting it, thanks.

(In reply to Adam Williamson from comment #56)
> We know. That's why it's still NEW, not MODIFIED or ON_QA. You don't need to
> keep reporting it, thanks.

Well, if its still now resolved is it bad to report and still update team? Being a responsible member and user of a opensource, reporting bugs is desired right? I was checking or not asking any ETA's here when it will be resolved.

(In reply to sachinwebgraphics from comment #57)
> (In reply to Adam Williamson from comment #56)
> > We know. That's why it's still NEW, not MODIFIED or ON_QA. You don't need to
> > keep reporting it, thanks.
> 
> Well, if its still now resolved is it bad to report and still update team?
> Being a responsible member and user of a opensource, reporting bugs is
> desired right? I was checking or not asking any ETA's here when it will be
> resolved.

Yes, reporting bugs is very much appreciated, and I'm sure Adam's "thanks" refers to that, too.

As an additional info, he explained bug status codes which are always capitalised (not to shout ...). We don't have an explicit "confirmed", but the reports above confirm it as well as a long term (because we rely on upstream) solution. That's what we know. We don't have a quick solution yet, only the mentioned workarounds.

At this point, additional reports can only help as much as point out the extent of the problem, by adding information about an affected board which has not been reported already.

If the bug goes quiet for a long time, or is about to be automatically closed for inactivity, it can be useful to say 'hey, this is still happening'. But when the bug is clearly active, with 15 posts this month before yours, it's not really needed - we wouldn't be debating this and putting it on blocker lists and pinging people if we didn't already know it was still happening =)

PU551LD (ASUS-NotebookSKU) is affected too

*** Bug 2143343 has been marked as a duplicate of this bug. ***

Nominating as a Prioritized bug. I'm aware that it will probably not make resolving this bug any faster, but I think it should be on that list.

(In reply to Robbie Harwood from comment #52)
> Again, one can't just build shim and have it work for secureboot.  We have
> to get it signed by Microsoft for that.  Microsoft's requirements now
> include NX support.  kernel patches for NX support exist, and have existed
> for months now.


To be slightly fair, v5 of the patch series has only been available for less than a month, and was still receiving a few review comments (mostly nits at this time last I looked).


> If you want the bug fixed, please apply pressure to the people who can
> actually fix it (kernel), not us.


Would it be possible to get a bugzilla created referencing the required enhancement that can be assigned to the kernel team[0][1], and have this bug depend on that bugzilla (rather than just a reference in the text of this bugzilla)?  That would make it clearer where the initial work needs to happen.


Thanks.



[0] If there is not already such a ticket.  I tried to do a search, but could not find such a ticket.

[1] From reports in this ticket, this also impacts the EL side of RH, so there may be ticket elsewhere.

During Fedora 38 Final Go/No-Go meeting, this blocker bug has been waived under the "difficult to fix" exception, and has become a Fedora 39 Beta blocker. Please, let's do our best to fix it by then, thank you.

In today's prioritized bugs meeting, we agreed to reject this as it's already an accepted F39 Beta blocker, and that's a bigger hammer
https://meetbot.fedoraproject.org/fedora-meeting-1/2023-04-19/fedora_prioritized_bugs_and_issues.2023-04-19-14.00.log.html#l-92

(In reply to Kamil Páral from comment #64)
> During Fedora 38 Final Go/No-Go meeting, this blocker bug has been waived
> under the "difficult to fix" exception, and has become a Fedora 39 Beta
> blocker. Please, let's do our best to fix it by then, thank you.

I'm not sure why it's marked difficult to fix when there was already a proposed, confirmed, and merged upstream fix:

https://github.com/rhboot/shim/pull/505

Is it not possible to just bump the shim version to that of upstream containing the fix? (15.7)

Apologies if I've missed something already mentioned that otherwise blocks this.

The explanation is above, see comments #51 and #52.

Ah I completely missed that, thanks and again sincere apologies.

Peter / Robbie, can we have an update on this for F39? It is currently an F39 Beta blocker, as that's how waiving works. Does it look like we will be able to get this done for F39 Beta? If not, what about Final?

This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and 8.8 also with base install image on Fujitsu Primergy RX300 S8

(In reply to comsec from comment #70)
> This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and
> 8.8 also with base install image on Fujitsu Primergy RX300 S8

RHEL7/UEFI: Boot hangs after updating shim to "shim-x64-15.6-3.el7_9"

https://access.redhat.com/solutions/7016419

(In reply to comsec from comment #71)
> (In reply to comsec from comment #70)
> > This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and
> > 8.8 also with base install image on Fujitsu Primergy RX300 S8
> 
> RHEL7/UEFI: Boot hangs after updating shim to "shim-x64-15.6-3.el7_9"
> 
> https://access.redhat.com/solutions/7016419

rhel 8.6 with shim-x64-15.5-2.el8.x86_64 same instal problem

(In reply to comsec from comment #71)
> (In reply to comsec from comment #70)
> > This happens on red hat enterprise linux 7.9 and 8.x with updates, 8.7 and
> > 8.8 also with base install image on Fujitsu Primergy RX300 S8
> 
> RHEL7/UEFI: Boot hangs after updating shim to "shim-x64-15.6-3.el7_9"
> 
> https://access.redhat.com/solutions/7016419

Rhel 8.5 finally works and seems to install just fine and it has shim-x64-15.4-2.el8_1.x86_64

So this bug on red hat 8 was introduced in shim 15.5