Bug 1955416 - Lenovo ThinkPad T490, unable to boot following clean install, stuck at splash screen
Summary: Lenovo ThinkPad T490, unable to boot following clean install, stuck at splash...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: shim
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1954245 1955390 1987018 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-30 04:58 UTC by Chris Murphy
Modified: 2022-06-27 10:44 UTC (History)
35 users (show)

Fixed In Version: shim-15.6-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-17 01:19:05 UTC
Type: Bug
Embargoed:
bcotton: fedora_prioritized_bug+


Attachments (Terms of Use)
efivars (190.00 KB, application/x-tar)
2021-04-30 05:05 UTC, Seth Goldin
no flags Details
dmesg (90.29 KB, text/plain)
2021-04-30 05:06 UTC, Seth Goldin
no flags Details
Verbose output before hanging (1020.67 KB, image/jpeg)
2021-04-30 18:44 UTC, Peter Hazenberg
no flags Details
Screenshot of text appearing on screen (1.74 MB, image/jpeg)
2021-04-30 19:41 UTC, Peter Hazenberg
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github rhboot shim-review issues 241 0 None open fedora-35-20220607 2022-06-10 16:02:01 UTC
Github rhboot shim issues 437 0 None open Lenovo ThinkPads, unable to boot following clean install, stuck at (vendor) splash screen 2021-11-23 19:31:46 UTC

Description Chris Murphy 2021-04-30 04:58:51 UTC
Description of problem:

Following a clean default/automatic installation of Fedora 34, the system hangs at the Lenovo splash screen.

Booting a USB stick again, efibootmgr shows a bat guano bootorder that is instigated by shim 15.4-4, because the problem doesn't happen upon downgrading to shim 15-8, but immediately reoccurs when upgrading back to 15.4-4.

The bootorder is apparently not completely honored by the firmware, FEdora is in the 8th position and yet fallback isn't working well enough to get to that point. So there's certainly a firmware bug here, but it seems shim 15.4-4 is instigating part of this in a way that shim 15-8 wasn't.


Version-Release number of selected component (if applicable):
shim 15.4-4

How reproducible:
Always


Steps to Reproduce:
1. Install https://download.fedoraproject.org/pub/fedora/linux/releases/34/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-34-1.2.iso
2. Reboot
3.

Actual results:

Hang at Lenovo splash screen


Expected results:

Should boot


Additional info:

efibootmgr -v following boot from USB stick after failed first boot (of the installed system)

BootCurrent: 001F
Timeout: 0 seconds
BootOrder: 001F,0010,0011,0012,0013,0014,0015,0000,0019,001A,001B,001C,001D,001E,0020,0021,0022,0023
Boot0000* Fedora	HD(1,GPT,fb2c442e-2249-4bf8-a6c4-391e52174312,0x800,0x12c000)/File(\EFI\fedora\shimx64.efi)
Boot0010  Setup	FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011  Boot Menu	FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012  Diagnostic Splash Screen	FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0013  Lenovo Diagnostics	FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0014  Regulatory Information	FvFile(478c92a0-2622-42b7-a65d-5894169e4d24)
Boot0015  ThinkShield secure wipe	FvFile(3593a0d5-bd52-43a0-808e-cbff5ece2477)
Boot0016  Startup Interrupt Menu	FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0017  Rescue and Recovery	FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0018  MEBx Hot Key	FvFile(ac6fd56a-3d41-4efd-a1b9-870293811a28)
Boot0019* USB CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot001A* USB FDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot001B* NVMe0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a400)
Boot001C* NVMe1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a401)
Boot001D* ATA HDD0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f602)
Boot001E* ATA HDD1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f601)
Boot001F* USB HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)
Boot0020* PXE BOOT	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot0021* LENOVO CLOUD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,ad38ccbbf7edf04d959cf42aa74d3650)/Uri(https://download.lenovo.com/pccbbs/cdeploy/efi/boot.efi)
Boot0022  Other CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35406)
Boot0023  Other HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f606)
Boot0024* IDER BOOT CDROM	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,1)
Boot0025* IDER BOOT Floppy	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,0)
Boot0026* ATA HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)
Boot0027* ATAPI CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a354)

anaconda storage.log shows



        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr -c -w -L Fedora -d /dev/nvme0n1 -p 1 -l \EFI\fedora\shimx64.efi
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr -b 0000 -B
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr -c -w -L Fedora -d /dev/nvme0n1 -p 1 -l \EFI\fedora\shimx64.efi

Comment 1 Chris Murphy 2021-04-30 05:04:02 UTC
It was possible to assemble the installed system in chroot, downgrade to shim 15-8, and 'efibootmgr --bootorder 0000' and reboot the installed system successfully. Upon updating to shim 15.4-4 though, we're back to a failed boot even though Fedora is first in the bootorder.

efivars tar will be attached matching this nvram state:

BootCurrent: 001F
Timeout: 0 seconds
BootOrder: 0000,0019,001A,001B,001C,001D,001E,001F,0020,0021,0022,0023
Boot0000* Fedora	HD(1,GPT,fb2c442e-2249-4bf8-a6c4-391e52174312,0x800,0x12c000)/File(\EFI\fedora\shimx64.efi)
Boot0010  Setup	FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011  Boot Menu	FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012  Diagnostic Splash Screen	FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0013  Lenovo Diagnostics	FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0014  Regulatory Information	FvFile(478c92a0-2622-42b7-a65d-5894169e4d24)
Boot0015  ThinkShield secure wipe	FvFile(3593a0d5-bd52-43a0-808e-cbff5ece2477)
Boot0016  Startup Interrupt Menu	FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0017  Rescue and Recovery	FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0018  MEBx Hot Key	FvFile(ac6fd56a-3d41-4efd-a1b9-870293811a28)
Boot0019* USB CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot001A* USB FDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot001B* NVMe0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a400)
Boot001C* NVMe1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a401)
Boot001D* ATA HDD0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f602)
Boot001E* ATA HDD1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f601)
Boot001F* USB HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)
Boot0020* PXE BOOT	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot0021* LENOVO CLOUD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,ad38ccbbf7edf04d959cf42aa74d3650)/Uri(https://download.lenovo.com/pccbbs/cdeploy/efi/boot.efi)
Boot0022  Other CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35406)
Boot0023  Other HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f606)
Boot0024* IDER BOOT CDROM	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,1)
Boot0025* IDER BOOT Floppy	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,0)
Boot0026* ATA HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)
Boot0027* ATAPI CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a354)

Comment 2 Seth Goldin 2021-04-30 05:05:22 UTC
Created attachment 1777572 [details]
efivars

Comment 3 Seth Goldin 2021-04-30 05:06:33 UTC
Created attachment 1777575 [details]
dmesg

Comment 4 Chris Murphy 2021-04-30 05:09:46 UTC
*** Bug 1955390 has been marked as a duplicate of this bug. ***

Comment 5 Chris Murphy 2021-04-30 05:11:12 UTC
[    0.000000] DMI: LENOVO 20N2CTO1WW/20N2CTO1WW, BIOS N2IET94W (1.72 ) 02/18/2021

Comment 6 Peter Hazenberg 2021-04-30 18:43:34 UTC
Hi, I'm having this same issue. My laptop is slightly different but behaves the same as above.

It's a Thinkpad Yoga 370, so 2 generations older as the T490.

`dmesg | grep DMI:`
[    0.000000] DMI: LENOVO 20JJS0VK1F/20JJS0VK1F, BIOS R0HET56W (1.36 ) 08/06/2020

Downgrading the shim works here too.

Setting the boot order or creating a new boot item with efibootmgr seems to be reset after rebooting all the time, which by itself doesn't cause any problem. It does indicate that the firmware is being weird.

When I set `mokutil --set-verbosity true` and booted the new-broken shim, it showed me the attached output at the point of hanging.

I also created a video of the entire boot process here: https://www.youtube.com/watch?v=FBtazoABHYY

Comment 7 Peter Hazenberg 2021-04-30 18:44:23 UTC
Created attachment 1777882 [details]
Verbose output before hanging

Comment 8 Peter Hazenberg 2021-04-30 19:40:52 UTC
I played around with this issue some more, and I found out that with the new-broken shim in place, the problem only occurs when secure boot is disabled.

In other words. I enabled secure boot. Then I powered off the machine (seems to be required, but rebooting twice also works). Problem gone, I can now successfully boot!

To verify, re-disabled secure boot. First boot was fine, the second warm-boot did hang again. Hard poweroff, enabled secure boot again. Warm reboot hanged again, second attempt was fine again.

Meanwhile I also recorded a video of a bunch of text appearing after booting via the "ssd" option instead of the "fedora" option: https://www.youtube.com/watch?v=vqKFPMFt25Q

Relevant screenshot also attached. This text can sometimes also appear in the hanging situation if you wait for long enough.

Comment 9 Peter Hazenberg 2021-04-30 19:41:47 UTC
Created attachment 1777915 [details]
Screenshot of text appearing on screen

Comment 10 Peter Jones 2021-04-30 20:07:19 UTC
(In reply to Peter Hazenberg from comment #9)
> Created attachment 1777915 [details]
> Screenshot of text appearing on screen

This looks very much like the firmware call to HandleProtocol() (shim.c:1104) returned success but gave us back a handle that's not completely populated.  Unfortunately that print would be the best clue as to what it's even trying to do when booting the "SSD" option, so I really have no idea what's going on there.  That said, it's *probably* unrelated to the original issue.

Comment 11 Seth Goldin 2021-04-30 20:41:47 UTC
I just reported this over to the Lenovo folks to give them a heads up.

Comment 12 Seth Goldin 2021-05-01 16:07:10 UTC
Firmware 0.1.72 for this ThinkPad, from LVFS, is the latest: https://fwupd.org/lvfs/devices/com.lenovo.ThinkPadN2IETXXP.firmware

Might need to wait for an update from Lenovo.

Comment 13 Seth Goldin 2021-05-05 04:36:34 UTC
Just a thought: would it matter whether the Secure Boot mode is Standard or Custom?

Comment 14 Seth Goldin 2021-05-25 03:35:31 UTC
The workaround that I have found works best was from Chris Murphy's instructions, to download the older 15-8 shim and replace the defective components of the 15.4-4 with the older, working 15-8 components: https://www.reddit.com/r/Fedora/comments/n27212/fedora_wont_boot_after_attempting_update_to_34/gwic9d4/?utm_source=reddit&utm_medium=web2x&context=3

Would be really nice to have some sort of acknowledgement that this is even being worked on. Seems like a huge issue for what's supposed to be a flagship line of laptops for Fedora.

Comment 15 Seth Goldin 2021-06-03 03:00:38 UTC
Firmware N2IET95P, version 1.73 was released today via LVFS: https://fwupd.org/lvfs/devices/com.lenovo.ThinkPadN2IETXXP.firmware

Can anyone see if this one works out of the box with shim-x64-15.4-4?

Comment 16 woat 2021-06-08 17:53:19 UTC
I can't find it now but someone on reddit said you can get Fedora 34 to boot by entering the BIOS menu then exit discarding changes.  That worked and allowed me to boot into Fedora 34 long enough to install the old version of shim*.rpm.

It took me the whole day but I now have a working computer again :)

@sethgoldin My firmwares are fully up to date and Fedora 34 still wouldn't boot with shim-x64-15.4.4

Comment 17 Javier Martinez Canillas 2021-06-23 09:31:59 UTC
*** Bug 1954245 has been marked as a duplicate of this bug. ***

Comment 18 loposkin 2021-07-15 15:23:57 UTC
I have checked it with new firmware(1.34) on my Thinkpad T14 and it's still not working.

Comment 19 Jérémy Friche 2021-07-17 09:10:22 UTC
Hello,

I have the same issue on a Acer Aspire E1-731, both Secure Boot enabled and disabled.
The boot is stuck on Acer logo and won't go further.

I can press F12 to display the UEFI boot list, and I have two entries:
1. HDD: WDC WDxxxxxxxxxxxxxxx
2. Fedora (WDCxxxxxxxxxxxxxx)

If I select the second line, the system boots without problem.

Comment 20 Seth Goldin 2021-07-19 02:37:58 UTC
Don't think there's actually information waiting on me here. Commenting to remove needinfo request.

Comment 21 AwlsomeAlex 2021-07-25 00:32:57 UTC
Experiencing the same issue with ThinkPad P52. Dual booted with Windows 11. Had Secure Boot enabled earlier, once I disabled it I was unable to boot into a Fedora ISO or Fedora system. System will boot with Ubuntu/GParted-live.

Comment 22 ValdikSS 2021-07-31 18:39:30 UTC
Have a black screen with infinite loop (laptop runs hot) on Lenovo ThinkPad X220, and old laptop without Secure Boot support.
No prints, no anything, just black screen.
Booting grubx64 using EFI shell works fine.

Comment 23 Perry Harrington 2021-08-08 22:45:21 UTC
I can confirm that the MSI GT60 suffers this same issue.

Booting with UEFI and CSM, grub does not load.

Installed the 15-8 shim-x64 package fixed the problem.

This machine did not ship with secure boot enabled and all partitions are MBR instead of GPT.

The fix is simpler than shown in the Reddit thread:

1) Boot from fedora installer, choose recovery mode
2) Mount your root volume (option 1)
3) chroot /mnt/sysroot
4) cd /root
5) Obtain the package using curl/wget
6) yum install shim-x64...rpm
7) add the following line to /etc/dnf/dnf.conf:
exclude=shim*

No need to copy stuff around and muck up the rpm database and risk breaking things.

Comment 24 Perry Harrington 2021-08-08 22:53:34 UTC
Correction, my partition tables are GPT, not MBR.  Setting from "UEFI with CSM" to just "UEFI" did not fix the issue (there is no specific secure boot setting in the BIOS, so I assume with CSM means insecure boot and without CSM means secure boot).

Comment 25 Seth Goldin 2021-09-30 21:02:31 UTC
It's wild that this is still marked as "new," with absolutely no acknowledgement from the RH devs.

Anyway, there's a new firmware that was released. Has anyone been able to test a fresh install with this firmware? https://fwupd.org/lvfs/devices/com.lenovo.ThinkPadN2IETXXP.firmware

Comment 26 Jérémy Friche 2021-10-01 05:13:29 UTC
For me, I downloaded recently the Fedora 34 ISO and installed on my Acer Aspire E1-731 who has the issue, then updated, and there was no problem. Was this issue fixed already?

Comment 27 Seth Goldin 2021-11-11 23:21:29 UTC
Incredible. Not only has this not been acknowledged at all by any Red Hat developers, but shim-x64-15.4-5.x86_64.rpm is listed as identical to shim-x64-15.4-4.x86_64.rpm, with the version bumped up just to distinguish that it's for F35.

So I don't know why I expected anything different; but upon a clean install of F35, this issue is persisting. So I'm going to have to go replace the guts of shim-x64-15.4-5.x86_64.rpm with the actual components from 15-8 again.

Comment 28 Chris Murphy 2021-11-12 18:25:07 UTC
Can those having the problem still report `dmesg | grep DMI:`
Thanks

Comment 29 Seth Goldin 2021-11-13 00:07:34 UTC
$ dmesg | grep DMI:
[    0.000000] DMI: LENOVO 20N2CTO1WW/20N2CTO1WW, BIOS N2IET96W (1.74 ) 08/10/2021

Comment 30 woat 2021-11-13 05:45:23 UTC
sudo dmesg | grep DMI:
[    0.000000] DMI: LENOVO 20N20032US/20N20032US, BIOS N2IET96W (1.74 ) 08/10/2021

To be clear I'm on Fedora 35.  The problem started with Fedora 34 and persists into Fedora 35.  I'm able to "fix" it replacing the shim.rpm with the one from fedora 33 (shim-x64-15-8.x86_64.rpm).  My machine now has regular problems waking from sleep but I don't know if that has anything to do with this bug or the fact I'm running an older shim.

Comment 31 ValdikSS 2021-11-13 06:05:12 UTC
DMI: LENOVO 4286CTO/4286CTO, BIOS 8DET76WW (1.46 ) 06/21/2018

In my case, switching off "boot order lock" option in UEFI configuration allowed to start shim, but I ended booting grub directly anyway.

Comment 32 Daan de Wit 2021-11-15 07:59:42 UTC
$ sudo dmesg | grep DMI:
[    0.000000] MI: LENOVO 20QNCTO1WW/20QNCTO1WW, BIOS N2NET46W (1.31 ) 06/22/2021

Comment 33 Chris Murphy 2021-11-18 15:51:49 UTC
Let's get some shim debugging videos!

0. Prepare to capture video: a dark room to avoid screen glare, test with Terminal first to see if your cell phone or camera needs to have exposure and focus set manually - if you can read the text, developers can't "just figure it out". It needs to be clear.
1. Update to the failing shim
2. sudo mokutil --set-verbosity true
3. Reboot
4. Capture video of the debug scroll
5. Downgrade to working shim (boot off Fedora 33 live CD and just copy the live's shim over the new one on the system's EFI system partition)
6. sudo mokutil --set-verbosity false

Comment 34 Seth Goldin 2021-11-21 00:30:02 UTC
Just updated to new firmware today to see if anything might be fixed, but alas, it's not.

$ dmesg | grep DMI:
[    0.000000] DMI: LENOVO 20N2CTO1WW/20N2CTO1WW, BIOS N2IET97W (1.75 ) 10/09/2021

Here's a video. There's some glow, but it should be legible in HD.

I had already had those working components from the 15-8 version installed, so I reinstalled shim-x64-15.4-5.x86_64.rpm, and then tried rebooting.

You can see where it gets stuck, at that `\EFI\fedora\grubx64.efi` step.

https://www.youtube.com/watch?v=QfXtJzfFozw

Comment 35 Seth Goldin 2021-11-23 01:25:53 UTC
Another thread of users experiencing the issue: https://www.reddit.com/r/Fedora/comments/pprgol/fedora_34_and_35_wont_install_on_new_thinkpads_no/

We can call this a firmware bug and blame Lenovo, but in my opinion, if this didn't happen with 15-8 but is indeed happening with 15.4-4 and 15.4-5, then that's a bug introduced after 15-8.

Comment 36 Chris Murphy 2021-11-23 19:35:05 UTC
The two youtube videos in this issue show the hang happening at different points and don't give any clues why they've just stopped where they did. I've opened an upstream shim issue to see whether the next step is to enhance debug output.

I've also put out a call on Fedora devel@ mailing list to see if there are any T490 users who aren't having this problem. The working assumption is all T490's are affected but what if this isn't true? It's curious that shim+grub work OK when booted from install media, but not from the laptop's built-in drive following a clean install.

Comment 37 Andrei Nevedomskii 2021-12-06 10:24:51 UTC
I have Lenovo P14s (gen 2, Intel) and experiencing a similar issue. I can't boot my laptop with _DISABLED secure boot_, it gets stuck at splash screen before grub. And that happens not only when I boot from the laptop's ssd, but also from a USB drive with fedora 35 iso.
Everything boots fine with secure boot enabled.

I've noticed that because the UEFI Firmware update tool was also getting stuck at splash screen, but with _secure boot ENABLED_. I thought that maybe disabling secure boot will fix that, but then I got the issue described above.

Comment 38 Andrei Nevedomskii 2021-12-20 13:26:57 UTC
A small update: after my laptop got a firmware update I can't boot Fedora 34 and Fedora 35 anymore from a USB drive with secure boot enabled (haven't tried with secure boot disabled).

Laptop: Lenovo P14s (Gen 2, Intel with touchscreen)
UEFI Firmware version: 1.46 (N34ET46W)
Embedded Controller Firmware version: 1.37 (N34HT37W)

Ubuntu 21.10 and the latest openSUSE tumbleweed boot fine. So probably it has something to do with the combination of the firmware and the shim version used in Fedora.

Comment 39 Chris Murphy 2021-12-29 03:49:16 UTC
Proposing as a prioritized bug because: (a) it's a regression in at least shim, possibly the device firmware too (b) it affects fairly common hardware (c) other distributions aren't having this problem (yet) (d) it's taking too long to make some forward progress like even "ok we have a shim that spits out more debug info could you try this?"

Comment 40 Michel Lind 2021-12-29 04:06:50 UTC
cc Mark Pearson, can someone at Lenovo help take a look at this?

Comment 41 Mark Pearson 2021-12-30 03:20:37 UTC
Thanks Michel - I've forwarded the issue to the team in Asia for their input.
Mark

Comment 42 Mark Pearson 2022-01-04 03:15:40 UTC
Hi - I've been trying to reproduce this on my T490 to help the FW team out...and haven't been able to. Wanted to check what I was missing. Below are the steps I was trying based on the notes above:

T490
Fedora35 - I've used both shim 15.4-4 and 15.4-5, but right now I'm using 15.4-4
BIOS 1.46
Steps
 - Enable Secure Boot.
 - Boot to Fedora
 - Power down
 - Boot to F1 setup and disable secure boot
 - F10 save and exit.
 - Boot to Fedora succesfully. Reboot
 - Boot to Fedora succesfully

The FW team need some pointers as to what has changed in shim and the FW is doing 'wrong'. I had pointed them at this thread, but there hasn't (I don't think) been anything conclusive on what changed in shim that might help them focus on something. 
If I can reproduce I can do some digging but otherwise I'm a bit stuck. 

Any pointers/suggestions on next steps would be appreciated

mark

Comment 43 Seth Goldin 2022-01-04 07:35:36 UTC
Mark, can you try firmware 1.72 or above? 1.72 is the firmware where this was first noticed.

Comment 44 Daan de Wit 2022-01-04 07:36:35 UTC
Last time I tried, my P53 with BIOS version 1.34 (N2NET49W) wouldn't even boot from the F35 USB image when I picked it from the boot menu. I could only install it by changing the boot order in the BIOS to boot from USB first, and then apply the work-around of entering the BIOS on boot and exiting without saving changes. Secure Boot is disabled.

Comment 45 Mark Pearson 2022-01-05 17:02:16 UTC
I was way behind on that system :) I've updated to the latest (BIOS 1.75 and EC 1.19) and I'm still not reproducing the problem.
I'll try a fresh install next - unless anybody has pointers on a reliable set of reproduce steps.
Mark

Comment 46 Chris Murphy 2022-01-05 22:19:41 UTC
@Mark and @Seth - sounds like the two systems are not in identical states, despite the same firmware revision. Seth should maybe `sudo tar -acf space.tar /sys/firmware/efi/efivars/` for safekeeping, and then do an agreed upon "factory reset" that's intended to get a system's NVRAM in a known state.

Comment 47 Seth Goldin 2022-01-11 05:05:02 UTC
Maybe I'm missing something fundamental here.

I rebooted to the BIOS settings, hitting Enter and then F1. Then I hit F9 to "reset to defaults." Is this what would get me back to the factory default settings for the T490 firmware?

Then I booted up, and reinstalled the shim via `$ dnf reinstall shim`, so that I was truly using shim-x64-15.4-5 on F35.

Then I rebooted, and then still, as has been the case since 15.4-4 on F34, the machine hung on the Lenovo splash screen.

So I hard reset, booted to my F35 live USB, copied the three files from 15-8 back to my EFI partition, and could get back in again. So I have the guts of the old 15-8 working for me, even though dnf "thinks" I'm running 15.4-5.

$ dmesg | grep DMI:
[    0.000000] DMI: LENOVO 20N2CTO1WW/20N2CTO1WW, BIOS N2IET97W (1.75 ) 10/09/2021

I'm able to reproduce this hang at the splash screen 100% of the time. I'm not quite sure what other information I could possibly provide, but it seems like lots of folks are having this issue, so I'd love to help squash this bug once and for all.

Comment 48 Seth Goldin 2022-01-11 05:22:26 UTC
Mark, do you have LUKS enabled? I do. Perhaps that has something to do with this whole issue?

Comment 49 Chris Murphy 2022-01-11 05:56:51 UTC
>booted to my F35 live USB, copied the three files from 15-8 back to my EFI partition

Fedora 33 is the last to have shim 15-8. Fedora 34+ media have shim 15.4. But you can successfully boot either Fedora 34 or Fedora 35 install media from a USB stick?

At this stage of boot, the firmware has ostensibly only read the GPT, and the EFI file system. As it executes shim, the firmware doesn't know what other file systems are yet. But could there be something about the GPT, or FAT file system, or in NVRAM, that's resulting in confusion, only with shim 15.4? Hence the idea to wipe NVRAM to get it as stock as possible. But I'm not sure how thorough reset to defaults is when it comes to NVRAM. I guess we could compare evifars between the working and non-working hardware.

Comment 50 Mark Pearson 2022-01-11 17:51:55 UTC
Hi,

I don't have LUKS enabled - I just have a vanilla install of Fedora with defaults (and it's the only thing on the system currently). I'm hesitant to go that path - it wasn't reported on any of the other linked threads I looked at.

I did the BIOS defaults reset myself - didn't seem to make any difference. Did another fresh install and updated to the latest and everything is still working for me :( I'm stumped. I do have a prototype T490 rather than a production unit but I don't think that should make any difference. I'll ask my colleague if he can try on his unit, but he'll need to go and pick it up from the office.

Mark

Comment 51 Daan de Wit 2022-01-12 07:39:10 UTC
FWIW, I too have LUKS enabled.

Comment 52 Ben Cotton 2022-01-12 17:24:00 UTC
In today's Prioritized Bugs meeting, we agreed to accept this as a Prioritized Bug on the grounds it apparently affects multiple popular hardware vendors.
https://meetbot.fedoraproject.org/fedora-meeting-1/2022-01-12/fedora_prioritized_bugs_and_issues.2022-01-12-16.00.log.html#l-75

Comment 53 Robbie Harwood 2022-01-12 20:44:33 UTC
To put it simply: in order for forward progress to occur, someone who can test development builds of shim needs to be able to reproduce (and therefore debug) the issue.  Right now, none of us (maintainers and Lenovo) have hardware that can, and no one who can reproduce the issue has built/signed/enrolled/tested a development shim.

Comment 54 Seth Goldin 2022-01-12 20:48:25 UTC
Mark, can you try a fresh install of F35 on 1.75 with LUKS enabled on the boot drive? I suspect this is related.

Comment 55 Domantas Speciunas 2022-01-13 09:22:36 UTC
I stuck to this during upgrade from Fedora 33 to 35, my disk was and it is encrypted using LUKS.
shim downgrade and lock. 

```
 [root@x1 dnf]# cat plugins/versionlock.list 

# Added lock on Tue Dec  7 15:45:25 2021
shim-x64-0:15-8.*
```
Laptop:
SKU Number: LENOVO_MT_20KH_BU_Think_FM_ThinkPad X1 Carbon 6th

Comment 56 ValdikSS 2022-01-13 10:20:45 UTC
I have a feeling that the bug is related to UEFI variables state.
On X220, I had boot order locked in UEFI settings, and after updating to Fedora 34 shim just hang indefinitely, without any errors.
Once I disabled "lock boot order" in UEFI settings, it booted fine, and even if you lock boot order again it will work just as well. Previous shim versions worked fine with boot order locked.

Comment 57 ValdikSS 2022-01-13 10:22:09 UTC
Note that the system does not support Secure Boot and shim did not try to launch fallback .efi file when hung.

Comment 58 Chris Murphy 2022-01-13 19:05:39 UTC
(In reply to Robbie Harwood from comment #53)
> To put it simply: in order for forward progress to occur, someone who can
> test development builds of shim needs to be able to reproduce (and therefore
> debug) the issue.  Right now, none of us (maintainers and Lenovo) have
> hardware that can, and no one who can reproduce the issue has
> built/signed/enrolled/tested a development shim.

Robbie, does the development version have better debugging than the current shim in Fedora? Is there any documentation for mortal humans to follow how to do this exactly? This is the most recent Fedora doc I can find on this subject, and it's missing steps on how to do what you're asking.
https://docs.fedoraproject.org/en-US/Fedora/18/html/UEFI_Secure_Boot_Guide/index.html

This looks like it could be adapted for Fedora, but the author got stuck and there are no replies.
https://askubuntu.com/questions/1313644/how-to-build-shim-from-source-and-enroll-keys


(In reply to Seth Goldin from comment #54)
> Mark, can you try a fresh install of F35 on 1.75 with LUKS enabled on the
> boot drive? I suspect this is related.

Has the problemed laptop been given a truly clean install? e.g. blkdiscard -fv /dev/nvme0n1 and then do an automatic installation with an without LUKS? This would take about 30 minutes to do twice and see if LUKS is a factor, or for that matter some obscure anomaly with the GPT or FAT that's triggering a bug in the firmware. That's a lot more likely than LUKS which neither the firmware nor shim know anything about and will treat it like any other content on the drive it doesn't recognize: ignore it. There's literally no mechanism by which the firmware or shim could care about LUKS.

Comment 59 ValdikSS 2022-01-31 19:48:23 UTC
After 2 hours of debugging with prints, I found that fallback (fbx64.efi) fails on CopyMem inside add_to_boot_list on my Lenovo X220 in this place of fallback.c:

    CopyMem(newbootorder + 1, bootorder, sizeof (CHAR16) * option);
    CopyMem(newbootorder + option + 1, bootorder + option + 1,
        sizeof (CHAR16) * (nbootorder - option - 1));

The fix already present in upstream:
https://github.com/rhboot/shim/commit/41319e14c9144063605750baccd5bb332a3cf4fc

Kindly asking to apply this patch to finally fix this issue.

Comment 60 Robbie Harwood 2022-01-31 20:01:17 UTC
Updating shim is not an instantaneous process, but I'm glad to hear we'll have the fix available with 15.5.

Comment 61 Robbie Harwood 2022-01-31 20:02:52 UTC
(The most helpful thing for making 15.5 happen upstream is for people who can to test the RCs, both in secureboot and non-secureboot configurations.  By this I do not mean merely verifying that we have correctly used git, but actually booting it on your hardware.)

Comment 62 Mark Pearson 2022-01-31 20:12:27 UTC
Thanks ValdikSS and Robbie!

Where can I get a signed shim to test against? I'm happy to run it against a number of our platforms in various configurations.

Just wondering: there have been a couple of fairly impactful shim issues in the last 12 months (there was another issue breaking FW updates early last year) - is there something we can do at Lenovo to help with testing etc? Admittedly I personally struggled to reproduce either of the issues...but I'd like to try and help out where we can and if there's a way we can throw proposed updates at a bunch of HW to sanity check it that might be useful? Caveat - we're working on improving some internal testing resources we have to make things like this possible so it might take a while to get to the truly useful stage...but seems like a good target :) If there's any legs to this email me (markpearson at lenovo dot com) rather than dragging this bug too much sideways.

mark

Comment 63 Robbie Harwood 2022-01-31 21:07:00 UTC
> Where can I get a signed shim to test against? I'm happy to run it against a number of our platforms in various configurations.

Only the distro shims get signed by MS.  We're talking about a pre-release version from upstream, so if you want to test, you either need to enroll your own CA into the UEFI db, or disable verification for testing purposes.  (How to do this varies between vendors and machines.)

> Just wondering: there have been a couple of fairly impactful shim issues in the last 12 months (there was another issue breaking FW updates early last year) - is there something we can do at Lenovo to help with testing etc? Admittedly I personally struggled to reproduce either of the issues...but I'd like to try and help out where we can and if there's a way we can throw proposed updates at a bunch of HW to sanity check it that might be useful? Caveat - we're working on improving some internal testing resources we have to make things like this possible so it might take a while to get to the truly useful stage...but seems like a good target :) If there's any legs to this email me (markpearson at lenovo dot com) rather than dragging this bug too much sideways.

Thanks, I'll follow up via email.

Comment 64 Matteo Brancaleoni 2022-02-05 10:50:02 UTC
Same here on Lenovo p15v Gen 1 type 20TQ. Upgraded from FC33 to FC35 (disk LUKS encrypted), same issue. Tried a fresh install, same. Downgraded to shim 15-8 works as expected. Keeping a look on updates to see if fixes the issue.

Comment 65 PVasileff 2022-02-12 14:07:06 UTC
Same here on Lenovo T480 with enabled EFI + secure boot. Disk not encrypted. Everything start with upgrade from Fedora 33 -> Fedora 34 -> Fedora 35 and after reboot - stuck on Lenovo logo. 

If I restert system again, any time system stuck on Lenovo logo and does not show grub.

To success boot I need to going to bios settings and disable secure boot and trying to boot again ;) + shutdown from power buotton (long press) and enable secure boot from bios. 

After that operation I able to see grub menu. 

When I download shim-x64-15-8.x86_64.rpm and install it and exclude shim package in dnf.conf - everithing works again without problem.

That is a bug and all of us that has that problem will be happy if rh developers fix that.

Thanks...

Comment 66 Ben Cotton 2022-03-04 16:54:37 UTC
@mpe

Comment 67 Ben Cotton 2022-03-04 16:55:49 UTC
(Gah, Bugzilla, why?! Sorry for the spam, everyone)

Mark, was your testing successful?

Comment 68 Mark Pearson 2022-03-04 17:32:37 UTC
Ha - I have that issue with bugzilla all the time too.

Yes it was - I tested a bunch of platforms with no issues found. Shared the results with the RH team.

Not sure what the next steps are but I'm assuming they're looking at it.

Mark

Comment 69 woat 2022-03-04 22:31:01 UTC
src.fedoraproject.org is no longer hosting the working Fedora 33 shim (shim-x64-15-8.x86_64.rpm). So I've uploaded it in case anyone needs it but forgot to back it up. I realize there's a security concern with downloading from me, a random internet person. Hopefully someone else can show us an official way to get the rpm.

sha256sum 0ff0897d7fd548a435345944ad3faece662ae8fdbf8c7e625991ce3ccfd54eef
https://mega.nz/file/JZYWGRDY#R8rGHeys4VLbAbN9Mrmfy6jRF3YTE_R3nwsn6kH8Bwk

Comment 70 Robbie Harwood 2022-03-04 22:51:11 UTC
Yeah so don't do that.

Built rpms don't live on src.fedoraproject.org.  If you want shim-x64-15-8.x86_64.rpm you can get it from koji: https://koji.fedoraproject.org/koji/buildinfo?buildID=1149359

Comment 71 Seth Goldin 2022-03-10 05:54:36 UTC
I'm still not seeing any new package with the fix: https://koji.fedoraproject.org/koji/packageinfo?buildOrder=-completion_time&packageID=14502&tagOrder=name&tagStart=0#buildlist

Couldn't this patch just be backported? https://github.com/rhboot/shim/commit/41319e14c9144063605750baccd5bb332a3cf4fc

What's the hold-up?

Comment 72 Chris Murphy 2022-03-11 01:02:28 UTC
shim-unsigned 15.5 was just built today, this is not signed for UEFI Secure Boot though. But it is the first step in a multistep process to get a signed shim packaged per https://github.com/rhboot/shim-review
https://koji.fedoraproject.org/koji/buildinfo?buildID=1932202

Once it's posted for review, I figure it's a few weeks for it to get signed. In the meantime it should be possible to sign it yourself and enroll the key with mokutil.

Comment 73 Andrei Nevedomskii 2022-04-04 08:46:03 UTC
Hello!

Are there any updates on this? Looks like it hasn't been posted for review yet

Comment 74 Ben Cotton 2022-04-08 20:11:25 UTC
I'm checking with the maintainer to see if a timeline is available.

Comment 75 Ben Cotton 2022-04-20 14:34:20 UTC
Currently, the maintainers expect to send the updated shim for signing in early June.

Comment 76 mattias.eriksson 2022-04-21 13:53:18 UTC
What is the reason for waiting until June before sending it to be signed? 

For someone who are not that familiar with the shim development, it seems a bit strange that this bug is marked as a prioritized bug on the F36 blockers page, the unsigned binaries are built, and yet it will not be sent for signing until after 3 month after the binaries were built. 
I'm tracking this bug since the current shim also prevents firmware upgrades on some Lenovo laptops and have been so for about a year (which can cause security issues), so there are more than one reason for this to be prioritized.

Comment 77 Ben Cotton 2022-04-21 14:53:36 UTC
(In reply to mattias.eriksson from comment #76)
> What is the reason for waiting until June before sending it to be signed? 

There's other work going into it that I can't comment on publicly, unfortunately. shim is an unusual package due to the signing process, so it's generally better to batch up the work and go through the process once instead of several times in short succession.

Comment 78 Jan Vesely 2022-04-21 22:12:24 UTC
Hi,
What's the impact on f36 use/upgrade path for Lenovo users?
The bug is reported against f34, is it safe to assume that systems that are running f35 are OK to upgrade?
what about systems that successfully upgraded f34->f35?

Comment 79 Marek Greško 2022-04-28 17:43:29 UTC
*** Bug 1987018 has been marked as a duplicate of this bug. ***

Comment 80 Seth Goldin 2022-05-11 03:14:58 UTC
I would also like to know the answer to Jan's question. If I upgrade or clean install Fedora 36, is the flawed shim-x64-15.4-5.x86_64.rpm still shipping with it, and will we all have to go through the convoluted workaround yet again?

Comment 81 Marek Greško 2022-05-11 11:03:09 UTC
The problems started from F34. It seems not to be fixed in F36 yet. But if you are running F34 or F35 without workarounds you should be safe to upgrade. If you are affected the workaround is possible also for F36.

Comment 82 Ben Cotton 2022-05-12 16:57:05 UTC
This message is a reminder that Fedora Linux 34 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '34'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 34 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 83 woat 2022-05-15 17:43:58 UTC
I tried booting Fedora 36 on my T490. I can confirm the bug is still present. Sadly my BIOS now won't remember the boot order I set more than one boot before resetting making it impossible to even use the workaround.

Comment 84 Seth Goldin 2022-05-16 14:53:20 UTC
Hi Peter, are you able to update this for F36 as well? Still present in 36, and I don't want the bug to close before it is.

Comment 85 Matteo Brancaleoni 2022-05-20 15:32:46 UTC
Maybe Peter or Ben can updated the version to FC35/36 ?

Is still present on both FC35 and FC36 and should not be lost until the update is released.

Comment 86 Chris Murphy 2022-05-20 20:14:16 UTC
This bug is in Rawhide so setting it to rawhide. I expect it needs to be fixed there first before we'd see it get to F35. I see a shim review request for rhel 9 back in March, but I don't see one for Fedora, so I don't know what or when then plan is.

Comment 87 Fedora Update System 2022-06-15 16:32:10 UTC
FEDORA-2022-98830efc68 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-98830efc68

Comment 88 Peter Jones 2022-06-15 16:43:02 UTC
f35, f36, and rawhide eventually all need the same build, so test with the one above, and I'll work with rel-eng on tagging when it's okay to do so.

Comment 89 Fedora Update System 2022-06-16 02:14:55 UTC
FEDORA-2022-98830efc68 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-98830efc68`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-98830efc68

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 90 Fedora Update System 2022-06-17 01:19:05 UTC
FEDORA-2022-98830efc68 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 91 ValdikSS 2022-06-19 15:40:03 UTC
(In reply to Fedora Update System from comment #90)
> FEDORA-2022-98830efc68 has been pushed to the Fedora 35 stable repository.
> If problem still persists, please make note of it in this bug report.

This shim (shim-x64-15.6-1.x86_64) has a regression, it does not allow to work with third-party enrolled certificates or binary file hashes. It always shows security violation screen, regardless what is enrolled into moklist.

Comment 92 Chris Murphy 2022-06-20 14:40:37 UTC
(In reply to ValdikSS from comment #91)
> This shim (shim-x64-15.6-1.x86_64) has a regression, it does not allow to
> work with third-party enrolled certificates or binary file hashes. It always
> shows security violation screen, regardless what is enrolled into moklist.

Please file a new bug with exact reproduce steps. I think it's pretty urgent we determine if this is reproducible because it's planned to go stable soon. This would be a really bad regression if folks can't use their own signed EFI programs or kernel modules, so I'm not sure 2 weeks is enough testing let alone 5 days. I think we ought to keep it in Rawhide for a while longer before moving it back to Fedora 35/36.

Could the original posters with the problem *this* bug is about please test this new shim in koji and confirm your problem is fixed?

Comment 93 Chris Murphy 2022-06-20 17:26:00 UTC
FWIW I can't reproduce the problem mentioned in c91

$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -sha256 -days 365
$ openssl x509 -in cert.pem -inform PEM -out test_cert.cer -outform DER
$ sudo mokutil --import test_cert.cer
(reboot, I can enroll the key, following key enrollment I can boot, nothing unexpected appears including a security violation screen; deleting the enrolled key works as expected too)

Comment 94 Andrei Nevedomskii 2022-06-20 17:27:34 UTC
The new shim version works just fine for me. I was finally able to install firmware updates with no issues.

I also have self-signed kernel modules and haven't faced the issue described by VladikSS. Didn't notice any security violation screen.

Comment 95 ValdikSS 2022-06-20 19:13:49 UTC
(In reply to Chris Murphy from comment #92)
> Please file a new bug with exact reproduce steps. I think it's pretty urgent
> we determine if this is reproducible because it's planned to go stable soon.

Sorry, it seems this regression was introduced not at this version, but in shim-x64-15.4-1.x86_64, it's the first version that does not load third-party files with security violation. shim-x64-15-8.x86_64 works as intended.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=2099380

Comment 96 Matteo Brancaleoni 2022-06-27 10:44:40 UTC
Updated here on FC35 and works ok, thanks!

Also with self signed modules no issues, everything continues running ok. Finally I can remove shim from versionlocked dnf list ;)


Note You need to log in before you can comment on or make changes to this bug.