Bug 2138019 - kernel-5.14.0-17X.el9.x86_64 / Not bootable (EFI) after Firmware update
Summary: kernel-5.14.0-17X.el9.x86_64 / Not bootable (EFI) after Firmware update
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: distribution
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Brian Stinson
QA Contact: Brian Stinson
URL:
Whiteboard:
: 2135315 2136892 2138077 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-10-26 21:56 UTC by Leon Fauster
Modified: 2022-11-08 09:54 UTC (History)
8 users (show)

Fixed In Version: kernel-5.14.0-183.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-137698 0 None None None 2022-10-26 22:06:27 UTC

Description Leon Fauster 2022-10-26 21:56:28 UTC
Description of problem:
--------------------------------

kernel-5.14.0 of the release series 17x do not boot while kernel-5.14.0 of the release series 16x do boot.

I was hoping that a new release of the kernel package would solve my issue but it didn't.

My issue started after updating my bios/firmware in combination with (at that time)
the kernel 5.14.0-171.el9.x86_64.


I updated the BIOS/Firmware of a DELL notebook from 1.8 to 1.9. and after it the latest C9S

kernel-5.14.0-171.el9.x86_64

Update: and this applies also to ... now:

kernel-5.14.0-174.el9.x86_64
kernel-5.14.0-176.el9.x86_64

can't be booted anymore (secure boot on) but the two older ones do boot:

kernel-5.14.0-165.el9.x86_64
kernel-5.14.0-168.el9.x86_64

UPDATE: kernel-5.14.0-167.el9.x86_64 also bootable

(in the meantime the firmware was update to 1.10 without solving this issues)

The grub error message when trying to boot kernel-5.14.0-171.el9.x86_64, 174- and 176-release
looks like:

error: ../../grub-core/kern/efi/sb.c:183:bad shim signature.
error: ../../grub-core/loader/i386/efi/linux.c:259:you need to load the kernel first.

I wonder how this happens. The firmware is classified as bug-fix update.

Not sure if DBX list was update. fwupdmgr shows "Current version: 83"
If so, it does not make sense that older kernels can be used to boot the system.
So, a big question mark how to solve this issue? Any hints ...?


# sha256sum /boot/efi/EFI/BOOT/BOOTX64.EFI
3ae459e79408b5287ce70c5b86ddcc92c243c7442d6769a330390598b7a351b1 /boot/efi/EFI/BOOT/BOOTX64.EFI



Version-Release number of selected component (if applicable):
--------------------------------
kernel-5.14.0-171.el9.x86_64
kernel-5.14.0-174.el9.x86_64
kernel-5.14.0-176.el9.x86_64


Actual results:
--------------------------------
error: ../../grub-core/kern/efi/sb.c:183:bad shim signature.
error: ../../grub-core/loader/i386/efi/linux.c:259:you need to load the kernel first.


Expected results:
--------------------------------
booting the system as the older kernels do. Works with
kernel-5.14.0-167.el9.x86_64
kernel-5.14.0-165.el9.x86_64
kernel-5.14.0-168.el9.x86_64


Additional info:
--------------------------------

A current DBX update will be blocked:

# LANG=C.UTF8 fwupdmgr update 
Devices with no available firmware updates: 
 • TPM 2.0
 • UEFI Device Firmware
 • UEFI Device Firmware
Devices with the latest available firmware version:
 • BC711 NVMe SK hynix 512GB
 • System Firmware
╔══════════════════════════════════════════════════════════════════════════════╗
║ Upgrade UEFI dbx from 83 to 217?                                             ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ This updates the dbx to the latest release from Microsoft which adds         ║
║ insecure versions of grub and shim to the list of forbidden signatures due   ║
║ to multiple discovered security updates.                                     ║
║                                                                              ║
║ Before installing the update, fwupd will check for any affected executables  ║
║ in the ESP and will refuse to update if it finds any boot binaries signed    ║
║ with any of the forbidden signatures.If the installation fails, you will     ║
║ need to update shim and grub packages before the update can be deployed.     ║
║                                                                              ║
║ Once you have installed this dbx update, any DVD or USB installer images     ║
║ signed with the old signatures may not work correctly.You may have to        ║
║ temporarily turn off secure boot when using recovery or installation media,  ║
║ if new images have not been made available by your distribution.             ║
║                                                                              ║
║ UEFI dbx and all connected devices may not be usable while updating.         ║
╚══════════════════════════════════════════════════════════════════════════════╝

Perform operation? [Y|n]: 
Downloading…             [***************************************]
Downloading…             [***************************************]
Decompressing…           [***************************************]
Decompressing…           [***************************************]
Authenticating…          [***************************************]
Authenticating…          [***************************************]
Restarting device…       [***************************************]
Writing…                 [***************************************]
Decompressing…           [***************************************]
Blocked executable in the ESP, ensure grub and shim are up to date: /boot/efi/EFI/BOOT/BOOTX64.EFI Authenticode checksum [cb994b400590b66cbf55fc663555caf0d4f1ce267464d0452c2361e05ee1cd50] is present in dbx

Comment 1 Leon Fauster 2022-10-27 23:36:06 UTC
TLDR:

/boot/vmlinuz-5.14.0-16* 

versus 

/boot/vmlinuz-5.14.0-17*

shows 

The signer's common name is CentOS Secure Boot Signing 201

versus

The signer's common name is Red Hat Test Certificate


############################################################


# ls -1 /boot/vmlinuz-5.14.0-16* |xargs -n1 pesign --show-signature -i 
---------------------------------------------
certificate address is 0x7fba079ec0e8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is CentOS Secure Boot Signing 201
The signer's email address is security
Signing time: Thu Sep 22, 2022
There were certs or crls included.
---------------------------------------------
---------------------------------------------
certificate address is 0x7f7ce4d0bd28
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is CentOS Secure Boot Signing 201
The signer's email address is security
Signing time: Fri Sep 23, 2022
There were certs or crls included.
---------------------------------------------




# ls -1 /boot/vmlinuz-5.14.0-17* |xargs -n1 pesign --show-signature -i 
---------------------------------------------
certificate address is 0x7efe96b739c8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Red Hat Test Certificate
No signer email address.
Signing time: Sat Oct 01, 2022
There were certs or crls included.
---------------------------------------------
---------------------------------------------
certificate address is 0x7f65df7ed9c8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Red Hat Test Certificate
No signer email address.
Signing time: Fri Oct 07, 2022
There were certs or crls included.
---------------------------------------------
---------------------------------------------
certificate address is 0x7fe45a19e728
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Red Hat Test Certificate
No signer email address.
Signing time: Wed Oct 12, 2022
There were certs or crls included.
---------------------------------------------
---------------------------------------------
certificate address is 0x7fd435357ee8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Red Hat Test Certificate
No signer email address.
Signing time: Mon Oct 17, 2022
There were certs or crls included.
---------------------------------------------


# rpm -q kernel|sort
kernel-5.14.0-167.el9.x86_64
kernel-5.14.0-168.el9.x86_64
kernel-5.14.0-171.el9.x86_64
kernel-5.14.0-174.el9.x86_64
kernel-5.14.0-176.el9.x86_64
kernel-5.14.0-177.el9.x86_64

Comment 2 Brian Stinson 2022-10-28 15:40:48 UTC
We had a misconfiguration on our kernel signing builders that we think caused this. We have reapplied the proper configuration which will be picked up on the next kernel build.

Comment 3 Leon Fauster 2022-10-28 16:07:15 UTC
Great, that the cause was found. My firmware update was then just a coincidence ...

Comment 4 Brian Stinson 2022-11-01 14:38:59 UTC
*** Bug 2138077 has been marked as a duplicate of this bug. ***

Comment 5 Brian Stinson 2022-11-01 15:19:47 UTC
*** Bug 2136892 has been marked as a duplicate of this bug. ***

Comment 6 Brian Stinson 2022-11-01 15:21:20 UTC
*** Bug 2135315 has been marked as a duplicate of this bug. ***

Comment 7 Timothée Ravier 2022-11-08 09:54:30 UTC
Looks like it's working for us with kernel-5.14.0-183.el9.x86_64 so I think this can be closed / moved to QA now.


Note You need to log in before you can comment on or make changes to this bug.