Bug 1861977

Summary: RHSA-2020:3216 grub2 security update renders system unbootable
Product: Red Hat Enterprise Linux 8 Reporter: are-support
Component: shimAssignee: Bootloader engineering team <bootloader-eng-team>
Status: ASSIGNED --- QA Contact: Release Test Team <release-test-team>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.2CC: ajb, akborder, anazmy, aperotti, apm118, asamir, awilliam, beat, b, berend.de.schouwer, bhoefer, bodik, bubrown, cdonnell, chref, Colin.Simpson, cperry, crxssi, cyberrider, cye, daniell1, dev, dmaley, dornelas, dwojewod, ealcaniz, emcnabb, farid_ghoreyshi, fedoraproject, fiezzi, fmartine, francesco.defrancesco, fweimer, gbailey, glaubitz, hakon.gislason, hartsjc, h.hoogendoorn, jbittner, jcsible, jeff, jpazdziora, jscheibe, jstancek, jstodola, jsuchane, jwedgeco, jwkwan2000, kas, knoha, kwalker, lee.jnk, ludwigit, mark, mbanas, mbenatto, mburman, mfalz, mgarciac, mhofmann, miabbott, mironov.ivan, mpoole, murray.j.blakeman, mvanderw, ngompa13, nk, obockows, ofalk, perobins, peter, petr, phil, phil.randal, pjones, prjagtap, ptalbert, pvlasin, rboza89, redhat-bugzilla, redhat, rmaity, rmetrich, robert.scheck, rstrode, sbonazzo, smstewa4, sonny, sreber, steevithak, stefw, tbblake, therman, thoger, toracat, tru, twaugh, vkadlcik, w.deborger, wmealing, wolfy, x.wolverine.x, yuokada
Target Milestone: rcKeywords: ZStream
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1862230 1862231 1862232 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1862230, 1862231, 1862232    
Attachments:
Description Flags
shimx64.efi none

Description are-support 2020-07-30 05:15:35 UTC
Description of problem:
Applying the RHSA-2020:3216 grub2 security update and the RHSA-2020:3218 kernel security and bug fix update on a fresh "minimal" installation of RHEL 8.2 renders the system unbootable.

How reproducible:
Consistently

Steps to Reproduce:
1. Install RHEL 8.2 "minimal" version from Binary DVD iso downloaded on 7/29/2020 on system running in EFI mode
2. Apply current updates as of 7/29/2020 with "yum update"
3. Reboot system

Actual results:
System hangs after POST and the grub menu never loads

Expected results:
System should display kernel version selection menu and then boot to RHEL

Additional info:
Reproducible on several consecutive clean installs of RHEL 8.2 and on one existing RHEL 8.2 system.

Comment 1 Sandro Bonazzola 2020-07-30 06:38:30 UTC
Hit the same issue right now on my work laptop

Comment 3 Wade Mealing 2020-07-30 07:15:32 UTC
You may be able to work around this by disabling secure boot in the BIOS.    Not ideal.. for sure but if you can't see grub.. sounds like something is up in the secure boot chain.

Comment 6 Sandro Bonazzola 2020-07-30 08:33:03 UTC
I booted my system with the installation ISO in troubleshoot mode, chrooted to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed grub2-install.
Now I can go past grub but got stuck to emergency shell failing swtching root.

Comment 7 Sandro Bonazzola 2020-07-30 09:22:06 UTC
(In reply to Sandro Bonazzola from comment #6)
> I booted my system with the installation ISO in troubleshoot mode, chrooted
> to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> grub2-install.
> Now I can go past grub but got stuck to emergency shell failing swtching
> root.

solved this by actually copying the grubenv file to /boot/grub2 instead of relying on the symlink to efi.

Comment 8 Clifford Perry 2020-07-30 09:36:23 UTC
Hi, 
If you are Red Hat customers, please do open a support case with Red Hat Support. This will allow our support engineers to gather debugging data that may help to understand what is happening in these individual situations. 

Thanks,
Cliff

Comment 11 Renaud Métrich 2020-07-30 09:56:46 UTC
See also BZ 1862045 on RHEL7.

Happens on HPE hardware with secure boot disabled.

Solution is to downgrade grub2 / shim / mokutil

Comment 12 Renaud Métrich 2020-07-30 10:21:03 UTC
Seen on HPE ProLiant XL230k Gen10 with secure boot disabled.

Comment 15 Jaroslav Suchanek 2020-07-30 12:09:44 UTC
I hit it as well and resolved by

1) booting to rhel-8 (centos-8) troubleshooting mode
2) chrooted to /mnt/sysimage
3) setup networking
4) yum downgrade shim-x64 grub2\*
5) reboot

Switching boot mode to legacy in bios does not help.

Comment 16 Javier Martinez Canillas 2020-07-30 12:14:02 UTC
(In reply to Jaroslav Suchanek from comment #15)
> I hit it as well and resolved by
> 
> 1) booting to rhel-8 (centos-8) troubleshooting mode
> 2) chrooted to /mnt/sysimage
> 3) setup networking
> 4) yum downgrade shim-x64 grub2\*
> 5) reboot
> 
> Switching boot mode to legacy in bios does not help.

Thanks a lot for the information. Could you please try to downgrade them individually
to confirm which package is causing the issue? Since if I understood correctly the
problem happens when Secure Boot is disabled, that should be possible.

Comment 17 Rohan Maity 2020-07-30 12:42:04 UTC
Solution tells to protect yum from upgrading
I got new laptop from RHEL8.2 have not yum upgrade system. But I might need to run `yum update` in order to install some dependencies

would this solution of protecting yum from upgrading is enough for me ?



Protect yum from upgrading the packages by adding the following entry in /etc/yum.conf
Raw
exclude=grub2* shim* mokutil

Comment 21 Christof Efkemann 2020-07-30 14:24:29 UTC
The problem is in grub2-efi-x64, specifically in the grubx64.efi. You can have everything else up-to-date but use grubx64.efi from the previous RPM, and the system can boot again (assuming that SecureBoot is turned off, of course).

Comment 23 Peter Jones 2020-07-30 14:50:47 UTC
(In reply to Christof Efkemann from comment #21)
> The problem is in grub2-efi-x64, specifically in the grubx64.efi. You can
> have everything else up-to-date but use grubx64.efi from the previous RPM,
> and the system can boot again (assuming that SecureBoot is turned off, of
> course).

Just so I've got clear data - which version is "the previous RPM" for you?

Comment 24 Christof Efkemann 2020-07-30 14:55:53 UTC
(In reply to Peter Jones from comment #23)
> Just so I've got clear data - which version is "the previous RPM" for you?

The previously installed RPM was grub2-efi-x64-1:2.02-81.el8.x86_64

Comment 28 Christof Efkemann 2020-07-30 15:23:31 UTC
One additional observation:
Like I said, we were able to boot the new kernel (4.18.0-193.14.2.el8_2.x86_64) with the grubx64.efi from the 2.02-81.el8.x86_64 RPM.
However, this seems to work only when bypassing shim altogether.

I. e.:
UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
UEFI -> grubx64.efi (2.02-87) -> *crash*
UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*

Comment 29 Javier Martinez Canillas 2020-07-30 15:45:38 UTC
(In reply to Christof Efkemann from comment #28)
> One additional observation:
> Like I said, we were able to boot the new kernel
> (4.18.0-193.14.2.el8_2.x86_64) with the grubx64.efi from the
> 2.02-81.el8.x86_64 RPM.
> However, this seems to work only when bypassing shim altogether.
> 
> I. e.:
> UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
> UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
> UEFI -> grubx64.efi (2.02-87) -> *crash*
> UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*

This is with shim-x64-15-14.el8_2.x86_64 right? What happens if you downgrade the shim package?

Comment 31 Christof Efkemann 2020-07-30 16:42:01 UTC
(In reply to Javier Martinez Canillas from comment #29)
> (In reply to Christof Efkemann from comment #28)
> > One additional observation:
> > Like I said, we were able to boot the new kernel
> > (4.18.0-193.14.2.el8_2.x86_64) with the grubx64.efi from the
> > 2.02-81.el8.x86_64 RPM.
> > However, this seems to work only when bypassing shim altogether.
> > 
> > I. e.:
> > UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
> > UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
> > UEFI -> grubx64.efi (2.02-87) -> *crash*
> > UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*
> 
> This is with shim-x64-15-14.el8_2.x86_64 right? What happens if you
> downgrade the shim package?

No, that's with shim-x64-15-13.el8.x86_64.

However, after writing the last comment, I got some doubts about my previous observations -- being unable to boot grubx64.efi version 2.02-87 without shim resulting in a crash. Turns out, when I tried to reproduce this crash just now, I wasn't able to. So the revised list looks like this:

UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
UEFI -> grubx64.efi (2.02-87) -> kernel -> *success*
UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*

Therefore, my initial assessment (grubx64.efi is the culprit) clearly seems wrong -- the problem seems to lie with shim-x64-15-13.el8.x86_64.

(NB: shim-x64-15-14.el8_2.x86_64 isn't available on my system -- is that from AppStream?)

Comment 32 Javier Martinez Canillas 2020-07-30 18:32:23 UTC
Created attachment 1702984 [details]
shimx64.efi

Please test with the shim EFI binary attached.

It has to be copied to /boot/efi/EFI/redhat/

Comment 36 Ludwig 2020-07-30 19:52:53 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

This worked for me on CentOS 8.

Comment 37 Jaroslav Suchanek 2020-07-30 20:25:01 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

Works well on lenovo t470s with rhel-8.2.

Comment 38 Christof Efkemann 2020-07-30 20:27:11 UTC
(In reply to Ludwig from comment #36)
> (In reply to Javier Martinez Canillas from comment #32)
> > Created attachment 1702984 [details]
> > shimx64.efi
> > 
> > Please test with the shim EFI binary attached.
> > 
> > It has to be copied to /boot/efi/EFI/redhat/
> 
> This worked for me on CentOS 8.

Same here, works on CentOS 8.2 (of course, I didn't try SecureBoot - I assume that the signature check would fail in this case (RHEL shim/CentOS GRUB)).

Comment 40 BlairS 2020-07-31 02:27:41 UTC
Thank you Sandro Bonazzola for that tip on grubenv.  Also tried to fix this from the troubleshooter (centos 8) and had the same switch root fail/emergency console.

Banging my head all day but at least I'll sleep good tonight.  Tomorrow I'll see if there are fixed packages.  Turning off updates for now.

Cheers

Comment 42 Nerijus 2020-07-31 06:54:32 UTC
(In reply to Sandro Bonazzola from comment #6)
> I booted my system with the installation ISO in troubleshoot mode, chrooted
> to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> grub2-install.
> Now I can go past grub but got stuck to emergency shell failing swtching
> root.

Did the same, but disabled secure boot in BIOS. Ended up with a system that boots to `grub>` prompt. Entering `normal` at it resumes usual OS boot with image menu, etc.

Comment 44 Robert Scheck 2020-07-31 11:43:04 UTC
For us (non-secure-boot environment), replacing /boot/efi/EFI/redhat/shimx64.efi (from shim-x64-15-14.el8_2.x86_64) by /boot/efi/EFI/redhat/grubx64.efi (from grub2-efi-x64-2.02-87.el8_2.x86_64) lead us back to a bootable system. Opened case 02717116 at the Red Hat customer portal anyway, but I guess it's related to this RHBZ.

Comment 45 Scott Marshall 2020-07-31 12:04:38 UTC
(In reply to Wade Mealing from comment #3)
> You may be able to work around this by disabling secure boot in the BIOS.   
> Not ideal.. for sure but if you can't see grub.. sounds like something is up
> in the secure boot chain.

My UEFI configuration already has SecureBoot disabled.
The CentOS 7 server (CentOS 7.8.2003) locked up at the point it was loading the shimx64.efi module; grub never even got a chance to display its menu.

Fortunately, I had another box with the prior GRUB2 EFI modules as I'd not updated it yet, mainly because I was distrustful of new shim modules!

After booting the dead host from USB/ISO rescue and mounting the boot/efi file system, I copied all the old efi modules to their correct locations.
No change to grubenv.

Now I'm running with the older grub2 efi modules and the latest kernel (3.10.0-1128.18.2.el7)

Comment 47 Jan "Yenya" Kasprzak 2020-07-31 12:36:05 UTC
I can confirm that CentOS 7 is also broken by the latest grub/shim update. With
grub2-2.02-0.86.el7.centos.x86_64 and shim-x64-15-7.el7_9.x86_64 it does not boot for me,
grub2-2.02-0.81.el7.centos.x86_64 and shim-x64-15-2.el7.centos.x86_64 the system is booting OK.

Comment 48 Bertus 2020-07-31 12:53:01 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

Yesterday I spend several hours ending up booting in recovery mode and perform a roll back to previous version as described in https://access.redhat.com/solutions/3486741, today I tried the grub update again to find out the systems did fail again to boot.

On CentOS 8.2 copied the EFI file into /boot/efi/EFI/centos/ and the server is back to normal. Thanks for this fix.

Comment 49 crxssi 2020-07-31 14:54:37 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

I am on an HP ML350G10 & CENTOS8.2.  Same problem as everyone else- after the update and reboot, the server froze on the "HP Enterprise" logo.  I booted rescue media and was unable to rollback the affected packages as suggested by https://access.redhat.com/solutions/5272311  I just get a bunch of "lowest version already installed, cannot downgrade it."  And I was unable to locate a previous version (or even name of) shimx64-15-13.el8.x86_64.

That is when I decided to just give your shimx64.efi a try. I renamed the broken one, then copied the supplied shimx64.efi to /boot/efi/EFI/centos/ and rebooted the system.  Grub came up normally.  Choose newest kernel, late in the boot it complained about "Warning -- SELinux targeted policy relabel is required", it worked on that a while, then it rebooted itself again.  After that reboot, I have 4.18.0-193.14.2.el8_2.x86_64 up and running again, apparently normal.

Not sure where you got the file, but it works great. Thank you!  Now we just wait for updated packages (once they figure out what went wrong).  For sure I will be more prepared for the next attempt...

Comment 50 Peter Ajamian 2020-07-31 15:22:16 UTC
This issue just hit mainstream:
https://linux.slashdot.org/story/20/07/31/158212/red-hat-security-update-renders-systems-unbootable

Comment 51 godzillante 2020-07-31 21:34:12 UTC
Can confirm on a Centos 7 server.
I had to boot with a recovery USB stick and then downgraded grub, shim and also mokutil as yum dependency.

Then the system rebooted without problems

Comment 52 Steevithak 2020-08-01 02:24:10 UTC
I did a "yum update" this afternoon on our CentOS 8.2 server and it seems to be bricked now. I get the Dell logo and then a black screen, no grub no CentOS. I see some people above saying they've fixed it by replacing a "shim" file - can anyone point me to detailed instructions on how to do this with a non-bootable server? Is there a bootable CD or flash stick image that has the fix on it?

Comment 53 James K 2020-08-01 02:49:20 UTC
(In reply to Nerijus from comment #42)
> (In reply to Sandro Bonazzola from comment #6)
> > I booted my system with the installation ISO in troubleshoot mode, chrooted
> > to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> > grub2-install.
> > Now I can go past grub but got stuck to emergency shell failing swtching
> > root.
> 
> Did the same, but disabled secure boot in BIOS. Ended up with a system that
> boots to `grub>` prompt. Entering `normal` at it resumes usual OS boot with
> image menu, etc.

I am seeing the same thing.  How did you fix it?

Comment 54 Steevithak 2020-08-01 03:14:43 UTC
Update: I found this page at RedHat that is supposed to solve the problem:

https://access.redhat.com/solutions/5272311

I tried applying to my CentOS 8.2 but ran into a couple of problems:

1. All the steps are in clear text except Step 2, which is critical to the solution. Step 2 is pay-walled and you have pay a support fee to access it which seems like bit of a jerk move for a critical issue like this. But I am using CentOS and don't pay for support so I guess I can't complain too much. Without step 2, you can't easily start networking after booting to the troubleshooting mode of the install DVD. I googled and found enough info bring up the ethernet interface, give it an ip, and set a route, so I got minimal networking. So far so good.

2. The solution as described in the above document says to fix the problem by using "yum downgrade shim\* grub2\* mokutil" but that doesn't work on my CentOS 8.2, I just get a yum error saying I already have the oldest version and there's nothing to downgrade to.

If anyone has successfully fixed this on a CentOS 8.2. and could post a step-by-step guide for what they did, it would be appreciated. I have a feeling it's going to be long night... :(

Comment 55 Colin.Simpson 2020-08-01 03:35:42 UTC
Centos 8.2 said earlier versions of the packages weren't available when I tried the yum downgrade from the chrooted environment.
I just rpm -qa shim-x64\* grub2\* and noted the packages I had. 

Then manually downloaded the previous versions from:


http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/

Then set an ip in the chroot and scp'd these packages onto this box. Used rpm -Uvh --force to install them. I then rebooted, and came up , it did and SELinux relabel and rebooted again.then all fine.

https://lists.centos.org/pipermail/centos-devel/2020-July/055953.html 

Other people seem to be doing the same, there maybe a neater way someone knows.

Comment 56 Wouter De Borger 2020-08-01 09:55:57 UTC
Work around that gets the system to boot (doesn't fix it, just allowed me to boot, without secure boot)

1. drop to boot menu, then to uefi shell (hold F11 during boot on my system)
2. navigate to the efi folder  (uefi shell is just like dos, for me the commands where:)

---
FS0:
cd EFI 
cd CENTOS
--- 
3. directly load grubx86.efi

---
grubx64.efi
---

Comment 57 Clifford Perry 2020-08-01 14:46:09 UTC
I just wanted to post a brief note that updated shim packages are now available and can be used in conjunction with previously released grub2, fwupd, and fwupdate packages. Both https://access.redhat.com/security/vulnerabilities/grub2bootloader and https://access.redhat.com/solutions/5272311 have been updated with links to the new shim package bugfix errata.

Comment 58 Thomas Stephen Lee 2020-08-01 18:17:36 UTC
Red Hat fixes are available.

https://access.redhat.com/errata/RHBA-2020:3265
https://access.redhat.com/errata/RHBA-2020:3262

wait for CentOS.

Comment 59 James K 2020-08-01 21:17:31 UTC
(In reply to Steevithak from comment #54)
> Update: I found this page at RedHat that is supposed to solve the problem:
> 
> https://access.redhat.com/solutions/5272311
> 
> I tried applying to my CentOS 8.2 but ran into a couple of problems:
> 
> 1. All the steps are in clear text except Step 2, which is critical to the
> solution. Step 2 is pay-walled and you have pay a support fee to access it
> which seems like bit of a jerk move for a critical issue like this. But I am
> using CentOS and don't pay for support so I guess I can't complain too much.
> Without step 2, you can't easily start networking after booting to the
> troubleshooting mode of the install DVD. I googled and found enough info
> bring up the ethernet interface, give it an ip, and set a route, so I got
> minimal networking. So far so good.
> 
> 2. The solution as described in the above document says to fix the problem
> by using "yum downgrade shim\* grub2\* mokutil" but that doesn't work on my
> CentOS 8.2, I just get a yum error saying I already have the oldest version
> and there's nothing to downgrade to.
> 
> If anyone has successfully fixed this on a CentOS 8.2. and could post a
> step-by-step guide for what they did, it would be appreciated. I have a
> feeling it's going to be long night... :(

I am not sure if you fixed the problem.  I had same issue when I tried to downgraded shim\* etc and it did not help at all.  What I did is follow the other thread that provided a shimx64.efi attachment.  I have to save it to a usb from another computer. Once I booted into the trouble shoot and recovery command prompt, I have to mount usb and copied the shim file to the /boot/efi/EFI/centos.  This method was tested by few people.  One issue I have now is instead of seeing the boot option, it started with 'Grub>' prompt and I have to type 'normal' to enter the boot option.  I am still trying to figure out how to fix this one.

Comment 60 Charlweed Hymerfan 2020-08-01 22:52:56 UTC
An alternate work-around to get bootable again is to get a recovery/live image that has a writable grub.conf, and insert only the boot entries from the broken system's grub.conf into the recovery system's grub.conf. This avoids many issues that must otherwise be resolved before one can work on a chrooted system.

For example, we were blocked by the fact that previous CentOS 7 recovery images and live USBs were too old to load some kernel modules needed to connect and downgrade, and recent CentOS 8 images could not load others. Surprisingly, the VFAT module was one of these.

Comment 65 Logan 2020-08-02 20:53:25 UTC
For CentOS 8 just install the new shim-x64-15-15.el8_2.x86_64.rpm (http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/shim-x64-15-15.el8_2.x86_64.rpm). Works like a charm.

Comment 66 Logan 2020-08-02 21:00:50 UTC
yum list grub2\* shim\*

Installed Packages
grub2-common.noarch                                                                                   1:2.02-87.el8_2                                                                         @BaseOS
grub2-efi-x64.x86_64                                                                                  1:2.02-87.el8_2                                                                         @BaseOS
grub2-pc.x86_64                                                                                       1:2.02-87.el8_2                                                                         @BaseOS
grub2-pc-modules.noarch                                                                               1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools.x86_64                                                                                    1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools-efi.x86_64                                                                                1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools-extra.x86_64                                                                              1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools-minimal.x86_64                                                                            1:2.02-87.el8_2                                                                         @BaseOS
shim-x64.x86_64                                                                                       15-15.el8_2                                                                             @@commandline

Comment 67 Farid Ghoreyshi 2020-08-03 12:55:32 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/


Hi
I encountered this bug just a few days ago trying a clean install of CentOS 8.2 and did a yum update. I was like (WTF :/ What just happened) [Kidding !] and tried rescuing grub and yet no luck but thanks to this attached file I was able to successfully boot the OSs I have installed. Thanks again. Hope this fixture gets released on the stream so anyone who is experiencing this issue can fix it without having any difficulty.


Worked On Lenovo G50-80 with bios version "b0cna0ww"

Comment 68 crxssi 2020-08-03 21:29:30 UTC
(In reply to Logan from comment #65)
> For CentOS 8 just install the new shim-x64-15-15.el8_2.x86_64.rpm
> (http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/shim-
> x64-15-15.el8_2.x86_64.rpm). Works like a charm.

Confirmed, I was able to do nothing but a "yum update" and got the package on my dead, then hacked-to-work system.  It replaced the shimx64.efi file I downloaded from this bug report.  I rebooted and it is fine and up-to-date.  According to the time stamp, the file was updated on the mirrors on Saturday (Aug 01, 2020).  Fast!

I assume the same was/is true for RHEL repos.  If so, I propose the bug report can now be resolved/closed.

Comment 69 are-support 2020-08-03 22:50:33 UTC
I can confirm that a RHEL 8.2 system with shim-x64-15-15.el8_2.x86_64.rpm installed and a RHEL 7.8 system with shim-x64-15-8.el7_8.x86_64.rpm installed both successfully reboot now.

Comment 70 R. Boza 2020-08-04 17:02:06 UTC
Hi,
I noticed that after updating to:
grub2 1:2.02-87
shim 15-15 


mokutil --list-enrolled would output:
MokListRT is empty

This prevents kmod-drivers such as kmod-nvidia from loading in 8.2 and 7.8 and boot up doesn't complete.

This is 100% reproducible for me at least. Fresh system install with either centos 7.8 or centos 8.2 with secure boot enabled. Using kmod-nvidia drivers from ELRepo on a Quadro K4200. It worked fine before this bug, I can see the keys before applying the update.

(In reply to Clifford Perry from comment #57)
> I just wanted to post a brief note that updated shim packages are now
> available and can be used in conjunction with previously released grub2,
> fwupd, and fwupdate packages. Both
> https://access.redhat.com/security/vulnerabilities/grub2bootloader and
> https://access.redhat.com/solutions/5272311 have been updated with links to
> the new shim package bugfix errata.

Comment 71 Akemi Yagi 2020-08-04 18:43:30 UTC
I wonder what happens if you try to install the ELRepo's SB key again.

Comment 72 R. Boza 2020-08-04 18:56:29 UTC
(In reply to Akemi Yagi from comment #71)
> I wonder what happens if you try to install the ELRepo's SB key again.

I tried that, and what happens is that it lets me enroll the key again even though it actually already exists but mokutil --list-enrolled continues to output MokListRT is empty. However upon trying on a fresh 8.2 before the bug, I see the key was duplicated. So not only the original key was always there, it duplicated the key without acknowledging it already existed.

Comment 73 Akemi Yagi 2020-08-04 19:05:19 UTC
(In reply to R. Boza from comment #72)
> (In reply to Akemi Yagi from comment #71)
> > I wonder what happens if you try to install the ELRepo's SB key again.
> 
> I tried that, and what happens is that it lets me enroll the key again even
> though it actually already exists but mokutil --list-enrolled continues to
> output MokListRT is empty. However upon trying on a fresh 8.2 before the
> bug, I see the key was duplicated. So not only the original key was always
> there, it duplicated the key without acknowledging it already existed.

Strange indeed. What do you see with a "keyctl list %:.system_keyring" ?

Comment 74 R. Boza 2020-08-04 19:40:12 UTC
(In reply to Akemi Yagi from comment #73)
> (In reply to R. Boza from comment #72)
> > (In reply to Akemi Yagi from comment #71)
> > > I wonder what happens if you try to install the ELRepo's SB key again.
> > 
> > I tried that, and what happens is that it lets me enroll the key again even
> > though it actually already exists but mokutil --list-enrolled continues to
> > output MokListRT is empty. However upon trying on a fresh 8.2 before the
> > bug, I see the key was duplicated. So not only the original key was always
> > there, it duplicated the key without acknowledging it already existed.
> 
> Strange indeed. What do you see with a "keyctl list %:.system_keyring" ?

I was only able to test from 8.2: (it looks like keyctl list %:.system_keyring is now keyctl list %:.builtin_trusted_keys. https://bugzilla.redhat.com/show_bug.cgi?id=1509714 documentation error reported for Fedora 26+ fixed now)
3 keys in keyring:
Centos Linux kernel
Centos Linux kpatch
Centos Linux Driver update

So it is missing ELRepo, and a few other keys that would normally be there. I think something in the update is preventing those keys from being read.

Comment 75 R. Boza 2020-08-04 20:03:08 UTC
(In reply to R. Boza from comment #74)
> (In reply to Akemi Yagi from comment #73)
> > (In reply to R. Boza from comment #72)
> > > (In reply to Akemi Yagi from comment #71)
> > > > I wonder what happens if you try to install the ELRepo's SB key again.
> > > 
> > > I tried that, and what happens is that it lets me enroll the key again even
> > > though it actually already exists but mokutil --list-enrolled continues to
> > > output MokListRT is empty. However upon trying on a fresh 8.2 before the
> > > bug, I see the key was duplicated. So not only the original key was always
> > > there, it duplicated the key without acknowledging it already existed.
> > 
> > Strange indeed. What do you see with a "keyctl list %:.system_keyring" ?
> 
> I was only able to test from 8.2: (it looks like keyctl list
> %:.system_keyring is now keyctl list %:.builtin_trusted_keys.
> https://bugzilla.redhat.com/show_bug.cgi?id=1509714 documentation error
> reported for Fedora 26+ fixed now)
> 3 keys in keyring:
> Centos Linux kernel
> Centos Linux kpatch
> Centos Linux Driver update
> 
> So it is missing ELRepo, and a few other keys that would normally be there.
> I think something in the update is preventing those keys from being read.

The output matches the one of a system with UEFI and Secure Boot disabled. However secureboot is enabled and I have doubled checked via BIOS settings and mokutil --sb-state.

it should be something like:
...asymmetric: Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87...
...asymmetric: Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29...
...asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed...
...asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e...
...asymmetric: Red Hat Enterprise Linux kernel signing key: 4249689eefc77e95880b...
...asymmetric: Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b7...
...ElRepo signing key...

Comment 76 Akemi Yagi 2020-08-04 20:18:17 UTC
I believe it is supposed to display:

asymmetric: The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key: f365ad3481a7b20e3427b61b2a26635b83fe427b

This is something that needs investigation. Thank you for your input.

Comment 77 Kyle Walker 2020-08-04 21:26:44 UTC
Thank you for the above report! I have opened the following bug to address the MOK list failure.

    1866107 – Following 1861977, the MOK list is inaccessible with "Couldn't get UEFI MokListRT" visible in the logs
    https://bugzilla.redhat.com/show_bug.cgi?id=1866107

Comment 78 Stephen Steward 2020-08-05 18:28:44 UTC
This bug has also affected my RHEL 8.2 system. After POST, the GRUB menu is not displayed. I have SecureBoot disabled by default in my UEFI BIOS settings.

Comment 79 Adam Williamson 2020-08-05 20:22:58 UTC
The bug can affect you whether or not SB is enabled. Please follow the steps in https://access.redhat.com/solutions/5272311 to recover now.