Bug 2040307

Summary: grub2 2.06-13.el9 update breaks RHEL 9 secureboot
Product: Red Hat Enterprise Linux 9 Reporter: Oliver Ilian <oliver>
Component: grub2Assignee: Bootloader engineering team <bootloader-eng-team>
Status: CLOSED CURRENTRELEASE QA Contact: Release Test Team <release-test-team>
Severity: high Docs Contact:
Priority: high    
Version: 9.0CC: rharwood, sbarcomb
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-17 21:27:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2037084    
Bug Blocks:    

Description Oliver Ilian 2022-01-13 12:39:10 UTC
Description of problem:
After updating an RHEL 9 Beta install (UEFI / SecureBoot enabled) with the latest content from Satellite, the machine is no longer booting

Version-Release number of selected component (if applicable):
grub2: 2.06-13.el9
kernel-5.14.0-1.7.1.el9.x86_64.rpm

How reproducible:
always

Steps to Reproduce:
1. Enable UEFI and SecureBoot
2. Install an RHEL 9 Beta system (I used rhel-baseos-9.0-beta-0-x86_64-dvd.iso)
3. subscribe system to Satellite with updated RHEL 9 beta repos
4. update grub2* via yum

==================================================================================================
 Package                  Arch      Version          Repository                            Size
==================================================================================================
Upgrading:
 grub2-common             noarch    1:2.06-13.el9    rhel-9-for-x86_64-baseos-beta-rpms    936 k
 grub2-efi-x64            x86_64    1:2.06-13.el9    rhel-9-for-x86_64-baseos-beta-rpms    1.3 M
 grub2-tools              x86_64    1:2.06-13.el9    rhel-9-for-x86_64-baseos-beta-rpms    1.9 M
 grub2-tools-minimal      x86_64    1:2.06-13.el9    rhel-9-for-x86_64-baseos-beta-rpms    609 k
Installing dependencies:
 grub2-tools-efi          x86_64    1:2.06-13.el9    rhel-9-for-x86_64-baseos-beta-rpms    543 k
 grub2-tools-extra        x86_64    1:2.06-13.el9    rhel-9-for-x86_64-baseos-beta-rpms    847 k

Transaction Summary
==================================================================================================
Install  2 Packages
Upgrade  4 Packages

Total download size: 6.1 M

Actual results:
- after the grub2 update, the system is no longer rebooting and shows the message right after boot:
     Verification failed: (0x1A) Security Violation

Expected results:
System boots in secure boot


Additional info:
- I tested the issue in a KVM machine. A customer also sees the issue in VCenter.
- Disabling SecureBoot allows the system to boot again

Comment 1 Robbie Harwood 2022-01-13 14:47:26 UTC
-13 is a version that is known to not be signed with the right keys.  Please retest with the latest version - i.e., >= -16.

Comment 2 Oliver Ilian 2022-01-13 16:16:40 UTC
Thanks for the info. -16 indeed works.

Do you know when -16 will be available, so it can be synced into the customer's Satellite server?

Also.. I think that might be a kernel topic, but after subscribing directly to subscription.rhsm.redhat.com and updating I get kernel 5.14.0-39.el9. I manually install grub2 *-16 and on reboot, the new kernel is not booting (the previous kernel works)

Error:

error:../,,/grub-core/kern/efi/sb.c:150:bad shim signature
error:../,,/grub-core/loader/i386/efi/linux.c:208:you need to load the kernel first.
Press any key to continue

I checked for a newer shim version, but could not find any (currently shim-x64-15-16.el8

Thanks for the feedback

Comment 3 Robbie Harwood 2022-01-13 19:41:06 UTC
> Do you know when -16 will be available, so it can be synced into the customer's Satellite server?

You'd need to ask a release engineering type person that - from engineering's perspective, -16 is ready to go for GA, and -11 was the last version slated for Beta.

> Also.. I think that might be a kernel topic, but after subscribing directly to subscription.rhsm.redhat.com and updating I get kernel 5.14.0-39.el9. I manually install grub2 *-16 and on reboot, the new kernel is not booting (the previous kernel works)

Interesting, does kernel >= 5.14.0-40.el9 work?  (With grub >= 2.06-16.el9)

Comment 4 Oliver Ilian 2022-01-14 08:22:09 UTC
(In reply to Robbie Harwood from comment #3)
> > Do you know when -16 will be available, so it can be synced into the customer's Satellite server?
> 
> You'd need to ask a release engineering type person that - from
> engineering's perspective, -16 is ready to go for GA, and -11 was the last
> version slated for Beta.

Thanks.. Understood. Now I am wondering if -11 was the last version meant for the beta, how did -12 made it into the beta

> 
> > Also.. I think that might be a kernel topic, but after subscribing directly to subscription.rhsm.redhat.com and updating I get kernel 5.14.0-39.el9. I manually install grub2 *-16 and on reboot, the new kernel is not booting (the previous kernel works)
> 
> Interesting, does kernel >= 5.14.0-40.el9 work?  (With grub >= 2.06-16.el9)

Nope, does not work with 5.14.0-40.el9 or 5.14.0-42.el9

Comment 5 Robbie Harwood 2022-01-14 19:14:32 UTC
> Thanks.. Understood. Now I am wondering if -11 was the last version meant for the beta, how did -12 made it into the beta

Unsure, but based on the kernel versions you're seeing, I suspect you're not actually running beta - they aren't targeted for beta either.

In any case, I think this needs to wait on https://bugzilla.redhat.com/show_bug.cgi?id=2037084