Bug 1883609 - Secure Boot fails to boot F33 Beta image
Summary: Secure Boot fails to boot F33 Beta image
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: shim
Version: 34
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://fedoraproject.org/wiki/Common...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-29 17:07 UTC by Alex Thomas
Modified: 2021-03-15 02:07 UTC (History)
26 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-15 02:02:21 UTC
Type: Bug
Embargoed:
bcotton: fedora_prioritized_bug+


Attachments (Terms of Use)
SecureBoot error (274.30 KB, image/jpeg)
2020-10-01 14:26 UTC, Kamil Páral
no flags Details
SB config (483.88 KB, image/jpeg)
2020-10-01 15:03 UTC, Kamil Páral
no flags Details
efivars from install F32 with SB on (24.50 KB, application/gzip)
2020-10-01 15:47 UTC, Kamil Páral
no flags Details
efivars from install F32 with SB off (23.31 KB, application/gzip)
2020-10-01 15:48 UTC, Kamil Páral
no flags Details
livecd-iso-to-disk commands (2.64 KB, text/plain)
2020-10-06 00:03 UTC, Colin Misare
no flags Details
T450s-fwupdmgr-devices.txt (4.01 KB, text/plain)
2020-10-07 15:41 UTC, Kamil Páral
no flags Details

Description Alex Thomas 2020-09-29 17:07:18 UTC
Description of problem:
Booting Fedora 33 beta ( KDE ) on Thinkpad T440p with Secure Boot turned on. Get error of " Secure Boot Image failed to verify with *ACCESS DENIED* "

Version-Release number of selected component (if applicable):
Fedora 33 Beta

How reproducible:
Always

Steps to Reproduce:
1. Create Fedora 33 Beta boot usb drive
2. Attempt to boot Fedora 33 Beta usb stick on laptop with Secure Boot enabled
3. Get error of " Secure Boot Image failed to verify with *ACCESS DENIED* "

Actual results:

Get error of " Secure Boot Image failed to verify with *ACCESS DENIED* "

Expected results:

Boot to live image of Fedora 33 Beta
Additional info:

Comment 1 Chris Murphy 2020-09-30 00:18:04 UTC
This image? Fedora-KDE-Live-x86_64-33_Beta-1.3.iso

>1. Create Fedora 33 Beta boot usb drive

How are you creating it? dd or Fedora Media Writer or other?

>3. Get error of " Secure Boot Image failed to verify with *ACCESS DENIED* "

I'm not seeing this message in source for shim or grub2. I'm not sure what's up...

Comment 2 Alex Thomas 2020-09-30 12:17:41 UTC
I am downloading it, and using Gnome Disks to write it to the usb stick. I will try the media writer. The error message is from the firmware on the laptop. For some reason it is not seeing whatever part of the image that is signed for Secure Boot as being valid.

Bugzilla required a component and Grub2 was the closest thing I could find on the list. If I chose wrong, I am sorry.

Comment 3 Alex Thomas 2020-10-01 12:59:00 UTC
Ok I reset the Secure Boot keys to factory defaults and it now works.

Comment 4 Neal Gompa 2020-10-01 13:16:41 UTC
This is a problem with shim, re-assigning to that component.

Comment 5 Fedora Blocker Bugs Application 2020-10-01 13:18:37 UTC
Proposed as a Blocker for 33-final by Fedora user ngompa using the blocker tracking app because:

 This violates the criteria "Release-blocking images must boot" because we are now seeing computers that reject our UEFI cert on shim to actually boot the live media or the final installed system.

Comment 6 Kamil Páral 2020-10-01 14:26:20 UTC
Created attachment 1718189 [details]
SecureBoot error

I can confirm the same problem on Thinkpad T450s. With SecureBoot on, I see the same error message, see the screenshot. If I turn SB off, I see only black screen when I try to boot (i.e. I can't boot it; I didn't try BIOS mode yet). This is F33 Workstation Beta.

The same stick boots completely fine on Thinkpad T480s with SB on.

Comment 7 Kamil Páral 2020-10-01 14:28:00 UTC
(In reply to Chris Murphy from comment #1)
> I'm not seeing this message in source for shim or grub2.

It is likely a firmware message.

Comment 8 Peter Jones 2020-10-01 14:37:10 UTC
Do we know how did dbx get updated on the machine?  That's the operative question here, and will be key to guiding our response.

Comment 10 Kamil Páral 2020-10-01 14:47:00 UTC
(In reply to Peter Jones from comment #8)
> Do we know how did dbx get updated on the machine?  That's the operative
> question here, and will be key to guiding our response.

Peter, I can help out with debugging on my Thinkpad T450s (see comment 6), but I don't understand your reply. What information can I provide to help you debug this? I have no idea what "dbx" is.

Comment 11 Kamil Páral 2020-10-01 15:03:35 UTC
Created attachment 1718193 [details]
SB config

I tried to use "Restore Factory Keys" in SB configuration on my T450s. That did not resolve the issue, F33 Beta stick still produces the same error.

However, on the same system, I can boot my installed F32 system just fine, and in journal I see "secureboot: Secure boot enabled" message.

Comment 12 Peter Jones 2020-10-01 15:13:54 UTC
(In reply to Kamil Páral from comment #10)
> (In reply to Peter Jones from comment #8)
> > Do we know how did dbx get updated on the machine?  That's the operative
> > question here, and will be key to guiding our response.
> 
> Peter, I can help out with debugging on my Thinkpad T450s (see comment 6),
> but I don't understand your reply. What information can I provide to help
> you debug this? I have no idea what "dbx" is.

Can you do:

1) set it up to the failing state
2) go in the firmware and disable secure boot
3) boot in to the OS and do:
   # tar cjf efivar.tar.bz2 /sys/firmware/efi/efivars/
4) attach that here

That may or may not actually show us what data is being used by the firmware, depending on how they implement disabling SB.

Comment 13 Peter Jones 2020-10-01 15:18:54 UTC
Also, in case it's not what my first suspicion was - which boot image failed, and if you've seen this on an installed box, what kernel, shim and grub2 packages were installed?

Comment 14 Kamil Páral 2020-10-01 15:34:25 UTC
I had firmware 1.36, I updated to latest 1.37. That did not help. But I found out a very curious thing. If I power off the laptop, power it on, and try to boot from the USB stick (F33 Beta), I get the Secure Boot "Access denied" error. However, if I boot my installed F32 first, and *reboot*, I can then boot from the USB stick successfully! So the installed F32 somehow changes the runtime state of the firmware to allow the stick to boot, while a cold laptop start denies that.

I'll attach the required logs shortly.

Comment 15 Kamil Páral 2020-10-01 15:47:52 UTC
Created attachment 1718206 [details]
efivars from install F32 with SB on

These are efivars from the installed F32 with SB on (no issues with booting).

Comment 16 Kamil Páral 2020-10-01 15:48:21 UTC
Created attachment 1718207 [details]
efivars from install F32 with SB off

These are efivars from the installed F32 with SB off (no issues with booting).

Comment 17 Kamil Páral 2020-10-01 15:52:27 UTC
The image failing to boot is this one:
https://dl.fedoraproject.org/pub/alt/stage/33_Beta-1.3/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-33_Beta-1.3.iso

It contains:
grub2-common-2.04-30.fc33.noarch
grub2-efi-ia32-2.04-30.fc33.x86_64
grub2-efi-ia32-cdboot-2.04-30.fc33.x86_64
grub2-efi-x64-2.04-30.fc33.x86_64
grub2-efi-x64-cdboot-2.04-30.fc33.x86_64
grub2-pc-2.04-30.fc33.x86_64
grub2-pc-modules-2.04-30.fc33.noarch
grub2-tools-2.04-30.fc33.x86_64
grub2-tools-efi-2.04-30.fc33.x86_64
grub2-tools-extra-2.04-30.fc33.x86_64
grub2-tools-minimal-2.04-30.fc33.x86_64
kernel-5.8.6-301.fc33.x86_64
kernel-core-5.8.6-301.fc33.x86_64
kernel-headers-5.8.6-300.fc33.x86_64
kernel-modules-5.8.6-301.fc33.x86_64
kernel-modules-extra-5.8.6-301.fc33.x86_64
shim-ia32-15-8.x86_64
shim-x64-15-8.x86_64

However, I'm not sure how the image is created, if some other tools were not used to create its bootloader.

I haven't seen the issue on my installed F32 (and I can't easily install F33 on that system to test).

Comment 18 Kamil Páral 2020-10-01 16:00:41 UTC
Here is another case of people encountering these issues:
https://twitter.com/chadmccullough/status/1311356891596029952

There is also this FESCo ticket discussing this topic:
https://pagure.io/fesco/issue/2479

Comment 19 Peter Jones 2020-10-01 16:18:33 UTC
(In reply to Kamil Páral from comment #16)
> Created attachment 1718207 [details]
> efivars from install F32 with SB off
> 
> These are efivars from the installed F32 with SB off (no issues with
> booting).

Which firmware setting exactly did you change here?  That looks like somehow it switched it to be /unsupported/ rather than /disabled/, which is extra weird.  Normally I would not expect just disabling it to do that.

Anyway, that removed all the settings, so it doesn't tell us anything directly about the issue.

Comment 20 Peter Jones 2020-10-01 16:20:01 UTC
(In reply to Kamil Páral from comment #18)
> Here is another case of people encountering these issues:
> https://twitter.com/chadmccullough/status/1311356891596029952

As of yet we do not know if this is the same issue.

Comment 21 Peter Jones 2020-10-01 16:30:11 UTC
(In reply to Kamil Páral from comment #17)
> The image failing to boot is this one:
> https://dl.fedoraproject.org/pub/alt/stage/33_Beta-1.3/Workstation/x86_64/
> iso/Fedora-Workstation-Live-x86_64-33_Beta-1.3.iso

That image has the exact same signed 1st stage bootloader binary as F32 ISOs did.  Something else must have been changed on the device in question.

Comment 22 Colin Misare 2020-10-02 07:44:14 UTC
I believe I am seeing this as well. I am unable to boot either of the F32 or F33-Beta Live Workstation media when Secure Boot is enabled on my machine, and receive the error message when trying:

Operating System Loader signature found in SecureBoot exclusion database ('dbx').

In my case this seems to be due to dual-booting with an Ubuntu install, which recently updated a package called secureboot-db (https://changelogs.ubuntu.com/changelogs/pool/main/s/secureboot-db/secureboot-db_1.6~20.04.1/changelog is the only reference I can find but my machine didn't see the update until Sep 25) that updated my machine with the July 29 2020 revocation list found at https://uefi.org/revocationlistfile.

Disabling Secure Boot allows either of the F32 or F33-Beta media to boot.

Comment 23 Lukas Ruzicka 2020-10-02 07:54:49 UTC
I have a spare T460s and I gave it a try. 

1. Downloaded the Beta 1.3 Workstation image.
2. Switched on the Secure Boot -> it also automatically toggled Boot to "UEFI Only" and CMS? to "off". 
3. With the Secured Boot switched on, I was able to boot the previously installed F33 Beta with no issues.
4. I was able to boot the F33 Beta ISO from a usb stick with no issues.

If you tell me how, I would gladly provide the technical data from firmware etc, but I have never done it before, so I need a bump to where do I look.

Comment 24 Kamil Páral 2020-10-02 08:21:24 UTC
(In reply to Peter Jones from comment #19)
> Which firmware setting exactly did you change here?  That looks like somehow
> it switched it to be /unsupported/ rather than /disabled/, which is extra
> weird.  Normally I would not expect just disabling it to do that.

If you look at attachment 1718193 [details], I only changed the first line called "Secure Boot" from "Enabled" to "Disabled".


(In reply to Peter Jones from comment #21)
> That image has the exact same signed 1st stage bootloader binary as F32 ISOs
> did.  Something else must have been changed on the device in question.

How sure you are about this? I can successfully boot F32 Workstation Live [1] on that laptop. And if I burn F33 Beta Workstation Live [2] on the same USB stick and of course don't change anything on that laptop's firmware, I get "Access Denied" SB error. So something has to be different between those images, they behave differently on the same hardware and I can replicate this 100%.

[1] https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-32-1.6.iso
[2] https://dl.fedoraproject.org/pub/alt/stage/33_Beta-1.3/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-33_Beta-1.3.iso

Comment 25 Kamil Páral 2020-10-02 08:25:32 UTC
(In reply to Lukas Ruzicka from comment #23)
> 3. With the Secured Boot switched on, I was able to boot the previously
> installed F33 Beta with no issues.
> 4. I was able to boot the F33 Beta ISO from a usb stick with no issues.

Please note that just enabling SB in UEFI doesn't necessarily mean it's enabled. You need to verify in the booted system using "journalctl -b | grep -i secureboot". On my desktop PC, I had to reset all the SB keys to default first, because SB truly got enabled (before that, firmware reported enabled but Fedora reported disabled).

> If you tell me how, I would gladly provide the technical data from firmware
> etc, but I have never done it before, so I need a bump to where do I look.

I think we don't need reports from machines where everything works as expected (Peter can correct me if I'm wrong). I also have 2 machines at home where F33 Beta boots with SB on just fine.

Comment 26 Kamil Páral 2020-10-02 08:27:10 UTC
> I had to reset all the SB keys to default first, because SB truly got enabled
                                                   ^^ before

Comment 27 Peter Robinson 2020-10-02 10:35:01 UTC
(In reply to Colin Misare from comment #22)
> I believe I am seeing this as well. I am unable to boot either of the F32 or
> F33-Beta Live Workstation media when Secure Boot is enabled on my machine,
> and receive the error message when trying:

Please report the hardware this is running on. This goes for anyone adding a "me too" comment.

Comment 28 Colin Misare 2020-10-02 11:15:34 UTC
(In reply to Peter Robinson from comment #27)
> (In reply to Colin Misare from comment #22)
> > I believe I am seeing this as well. I am unable to boot either of the F32 or
> > F33-Beta Live Workstation media when Secure Boot is enabled on my machine,
> > and receive the error message when trying:
> 
> Please report the hardware this is running on. This goes for anyone adding a
> "me too" comment.

My apologies. My machine is a Dell XPS 15 9570 laptop.

Comment 29 Chris Murphy 2020-10-02 16:53:01 UTC
(In reply to Kamil Páral from comment #24)
> (In reply to Peter Jones from comment #21)
> > That image has the exact same signed 1st stage bootloader binary as F32 ISOs
> > did.  Something else must have been changed on the device in question.
> 
> How sure you are about this?

100% - shim-x64-15-8.x86_64 is in every image since 2018. Not only are the keys the same, the binary is identical because it hasn't even been rebuilt.

>And if I burn F33 Beta Workstation Live [2] on the same
> USB stick and of course don't change anything on that laptop's firmware, I
> get "Access Denied" SB error. So something has to be different between those
> images, they behave differently on the same hardware and I can replicate
> this 100%.

grubx64.efi is different, but I don't know if a way to tell whether the firmware is tripping before or after shimx64.efi

Comment 30 Chris Murphy 2020-10-05 17:43:44 UTC
Could someone who has run into this bug, take the same USB stick and same failing ISO image, and re-image using livecd-iso-to-disk? I'm pretty sure the flags to use are '--noverify --efi --format' -- Adam says there might be a testcase with the commands to use. I think the --reset-mbr flag isn't necessary in this case, since we only need a strictly EFI supporting media.

Comment 31 Chris Murphy 2020-10-05 17:51:29 UTC
By the way, just to be sure, it's a good idea to use Fedora Media Writer's reset option. Or alternatively 'dd if=/dev/zero of=/dev/sdX bs=1M count=3 oflag=direct' should be adequate.

Comment 32 Geoffrey Marr 2020-10-05 20:07:42 UTC
Discussed during the 2020-10-05 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as, while this is a worrying bug, it's also very mysterious at present. We don't know what's going on or how many systems it might affect or if it's even a Fedora bug at all (it may be a Lenovo firmware bug). We'll try to gather more information before making a decision

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-10-05/f33-blocker-review.2020-10-05-16.00.txt

Comment 33 Colin Misare 2020-10-06 00:03:44 UTC
Created attachment 1719199 [details]
livecd-iso-to-disk commands

Comment 34 Colin Misare 2020-10-06 00:13:29 UTC
(In reply to Chris Murphy from comment #30)
> Could someone who has run into this bug, take the same USB stick and same
> failing ISO image, and re-image using livecd-iso-to-disk? I'm pretty sure
> the flags to use are '--noverify --efi --format' -- Adam says there might be
> a testcase with the commands to use. I think the --reset-mbr flag isn't
> necessary in this case, since we only need a strictly EFI supporting media.

I've attached my commands and output to create both F32 and F33-Beta live disks using livecd-iso-to-disk. Both of these disks fail to boot on my Dell XPS 15 9570 laptop with SecureBoot enabled, showing the same error message for both:

Operating System Loader signature found in SecureBoot exclusion database ('dbx').

Disabling SecureBoot in the firmware on my laptop allows both of the created disks to boot successfully.

I also came across https://www.reddit.com/r/Fedora/comments/j5e5p1/ where it seems another user may have encountered this issue in a different way. The user there says they have a dual boot setup with F33-Beta and Windows 10, and after running an update in Windows 10 they now receive a "Selected boot image did not Authenticate" message when trying to boot Fedora subsequently. A few comments down the user mentions disabling SecureBoot and then being able to successfully boot into Fedora.

Let me know if there is any additional information I can provide, and thanks again for all of the help.

Comment 35 Alex Thomas 2020-10-06 04:45:29 UTC
Looking like it is something with something else updating the dbx which then blocks fedora 33 beta.iso from booting. After I reset the secure boot keys to default, booted just fine. Still issue with packages in Fedora 33 and the build environment I needed, so dropped back to ubuntu 20.04 and sure enough, after it updated, had same issue. So it is recreated. 

The following is the information I think if being requested, when after reseting SB keys, I do get it to boot.

grub2-common-2.04-30.fc33.noarch
grub2-efi-ia32-2.04-30.fc33.x86_64
grub2-efi-ia32-cdboot-2.04-30.fc33.x86_64
grub2-efi-x64-2.04-30.fc33.x86_64
grub2-efi-x64-cdboot-2.04-30.fc33.x86_64
grub2-pc-2.04-30.fc33.x86_64
grub2-pc-modules-2.04-30.fc33.noarch
grub2-tools-2.04-30.fc33.x86_64
grub2-tools-efi-2.04-30.fc33.x86_64
grub2-tools-extra-2.04-30.fc33.x86_64
grub2-tools-minimal-2.04-30.fc33.x86_64
kernel-5.8.6-301.fc33.x86_64
kernel-core-5.8.6-301.fc33.x86_64
kernel-headers-5.8.6-300.fc33.x86_64
kernel-modules-5.8.6-301.fc33.x86_64
kernel-modules-extra-5.8.6-301.fc33.x86_64
shim-ia32-15-8.x86_64
shim-x64-15-8.x86_64

Comment 36 Kamil Páral 2020-10-06 08:27:57 UTC
Sigh, I spent several more hours debugging this and I'm going crazy and really hate the firmware in my T450s:

1. When I said I can boot F32 Live just fine, I was wrong. At least today, I can't, at all. 'Access Denied' SB error.
2. I must be careful to do a cold boot every time, because if I reboot into my installed system (grub is enough), all the SB errors disappear and I can boot any image, until poweroff.
3. The poweroff must last some unspecified time (voltage decrease?), otherwise it behaves like a reboot (see above), and that creates a lovely random factor into all my testing. Perhaps that was the reason why F32 'worked' for me before.
4. If I "Restore Factory Keys" in SB config in UEFI, it changes nothing, Access Denied.
5. However, if I "Clear All SB Keys" immediately followed by "Restore Factory Keys", I'm able to boot any image (even F33) *once*, and then I get Access Denied on subsequent attempts. Facepalm.

6. Today with a careful approach considering the above, I can't boot any USB stick in SB mode - neither F33 nor F32 nor F31 - created by the Fedora Media Writer (dd). Access Denied.
7. I tried livecd-iso-to-disk approach on F33 and F32 images, created from F33 host, and none of them boots (Access Denied).
8. I tried the same from F32 host, again nothing boots (Access Denied). I even downgraded lorax to the oldest version available in F32 main repo, no change.

I'm really inclined to just mark the T450's SB implementation as utterly broken, document it and go on. Of course we still need to try to figure out the root cause for the other affected devices and find a bug if there's one on our side, but at least on my device I'm getting convinced that the Lenovo firmware is simply broken beyond repair.

Comment 37 Chris Murphy 2020-10-06 15:35:06 UTC
Can everyone having this problem:

1. Run the following command
$ fwupdmgr get-devices
2. Look in the UEFI dbx section
3. Report the "Current version" you find there
4. Report all other OS's you're booting and their versions

e.g. for me the answer is:
77, Windows 10 (not updated in months)

Comment 38 Colin Misare 2020-10-06 18:40:44 UTC
(In reply to Chris Murphy from comment #37)
> Can everyone having this problem:
> 
> 1. Run the following command
> $ fwupdmgr get-devices
> 2. Look in the UEFI dbx section
> 3. Report the "Current version" you find there
> 4. Report all other OS's you're booting and their versions
> 
> e.g. for me the answer is:
> 77, Windows 10 (not updated in months)

238, Ubuntu 20.04 (up to date)

Comment 39 Kamil Páral 2020-10-07 15:41:57 UTC
Created attachment 1719787 [details]
T450s-fwupdmgr-devices.txt

(In reply to Chris Murphy from comment #37)
> Can everyone having this problem:
> 
> 1. Run the following command
> $ fwupdmgr get-devices
> 2. Look in the UEFI dbx section

I don't see any dbx section. See attachment.

> 3. Report the "Current version" you find there
> 4. Report all other OS's you're booting and their versions

Windows 10, but I boot it twice a year, and I hadn't booted it for a long time when I first reported this issue. When I played with SB, I then booted into it, but nothing changed, and resetting all the keys and restoring factory defaults still "works" the same, per my latest report.

Comment 40 Miro Hrončok 2020-10-07 17:11:24 UTC
FTR this has been accepted as a FESCo blocker. "The solution might not be necessarily technical."

Comment 41 Chris Murphy 2020-10-07 19:33:27 UTC
(In reply to Kamil Páral from comment #39)
> I don't see any dbx section. See attachment.

Hmm, how about 'dbxtool -l' ? If you get a list, only the first number of the last line is relevant.

Comment 42 Kamil Páral 2020-10-08 08:37:18 UTC
$ dbxtool -l
Could not get dbx variable: No such file or directory
$ sudo dbxtool -l
Could not get dbx variable: No such file or directory

Comment 43 Matthew Miller 2020-10-08 16:12:06 UTC
(In reply to Miro Hrončok from comment #40)
> FTR this has been accepted as a FESCo blocker. "The solution might not be
> necessarily technical."

That's 

https://pagure.io/fesco/issue/2479

right?

Comment 44 Chris Murphy 2020-10-08 18:57:58 UTC
(In reply to Matthew Miller from comment #43)
> That's 
> 
> https://pagure.io/fesco/issue/2479
> 
> right?

Correct.

Comment 45 Kamil Páral 2020-10-09 11:41:55 UTC
> <cmurf> kparal what do you get for 'ls -l /sys/firmware/efi/efivars | fpaste'

See comment 15 and comment 16.

> <cmurf> and is secure boot enabled?

$ journalctl -b | grep secureboot
Oct 09 11:53:52 dryad kernel: secureboot: Secure boot enabled
Oct 09 11:53:52 dryad kernel: secureboot: Secure boot enabled

> <cmurf> 'mokutil --sb-state'

$ mokutil --sb-state 
SecureBoot enabled

Comment 46 Adam Williamson 2020-10-12 17:48:36 UTC
Miro, Matthew, Chris - I thought we'd agreed the FESCo issue was not in fact anything to do with kparal's bug here? kparal's issues do not seem to be to do with key revocation.

Comment 47 Adam Williamson 2020-10-12 17:55:41 UTC
Oh, okay, per https://pagure.io/fesco/issue/2479#comment-694022 it sounds like at least Colin is actually running into the key revocation problem, and I think https://bugzilla.redhat.com/show_bug.cgi?id=1883609#c35 suggests Alex may be as well, right?

But Kamil's problem seems a lot...weirder and we're not really sure what's going on with his case yet.

So...should we ask Kamil to file a separate bug, and treat this one as actually being for the problem of the keys that signed our shim getting revoked by a dbx update that's applied...somehow (system firmware update, update of another OS)? If I'm understanding that correctly.

Comment 48 Bojan Smojver 2020-10-13 09:28:51 UTC
In case it matters...

I created a bootable USB stick of Fedora 33 beta and booted it on my T450s. The secure boot was enabled and dmesg and jouranlctl report the same.

Comment 49 Bojan Smojver 2020-10-13 10:38:20 UTC
Note that there is just one OS on my T450s: F32. There used to be Windows many years ago, but that was since removed. BIOS/UEFI version is 1.37 and applied by cutting an update USB stick from the ISO image Lenovo distributes on their site.

To my knowledge and unless F32 is doing something I am unaware of, no revocation of anything related to secure boot has ever been applied to the machine.

Comment 50 Kamil Páral 2020-10-13 11:22:24 UTC
(In reply to Bojan Smojver from comment #48)
> I created a bootable USB stick of Fedora 33 beta and booted it on my T450s.
> The secure boot was enabled and dmesg and jouranlctl report the same.

Bojan, have you done that after a cold boot? I.e. shut down the laptop completely, wait 30 seconds, and then boot right into the USB stick (press F11 during post). In comment 36 I described how rebooting into the USB stick from an installed F32 worked for me as well, but not from a cold boot.

Comment 51 Bojan Smojver 2020-10-13 11:23:30 UTC
Let me try...

Comment 52 Bojan Smojver 2020-10-13 11:29:17 UTC
Yeah, same. Boots fine. Secure boot reported by the kernel.

Comment 53 Michael Catanzaro 2020-10-13 13:49:04 UTC
So this is a blocker bug, that means it needs to be solved immediately (like today/tomorrow) or the release slips. Peter, do you have a plan here? If the shim has to be signed by Microsoft, it sounds like there is extremely little time remaining here?

Comment 54 Adam Williamson 2020-10-13 16:46:49 UTC
FWIW, since FESCo declared this a blocker and said the solution "does not necessarily have to be technical", I'm sort of figuring responsibility for deciding what actually needs to be done to "resolve the blocker" is shared between FESCo and pjones. But yes, someone needs to decide what constitutes "resolving the blocker", quickly. We don't have many other showstoppers ATM so we might be able to get an RC, aside from this issue.

Comment 55 Adam Williamson 2020-10-13 17:06:23 UTC
Some news from pjones: it seems the plan here is to include a re-signed shim in F33. He's aiming to submit it for signing this afternoon and we hope it will get signed in "not more than a couple of days".

Go/No-Go is scheduled for Thursday, to meet that schedule we really need an RC today or by the latest tomorrow, so if the new shim isn't signed very quickly we're looking at a slip of some kind here, whether that's a soft slip by pushing Go/No-Go a bit and compressing the schedule between then and Tuesday, or actually slipping the release date.

Ben, FESCo folks, what's your vision for that?

Comment 56 Ben Cotton 2020-10-13 18:11:27 UTC
Given that Friday is a day off for Red Hat employees globally and most of the people involved fall into that category, I'm inclined to think a soft slip is off the table. 

My inclination is to request an RC today or tomorrow and start testing against that. If we can get a new RC that includes the signed shim, we can proceed with the Go/No-Go. If not, we shift to Target Date #1. 

(I'm intentionally ignoring other blockers at the moment. Unless most of them get fixed today, it's probably a moot point anyway. If we're left with just this one and 1861700, I'm more inclined to proceed and decide how we feel on Thursday.)

Comment 57 Neal Gompa 2020-10-13 22:20:29 UTC
With my FESCo hat on, I'm inclined to suggest that if this is the only remaining blocker, then we could _wait_ for the first candidate to be created on Friday or Saturday, which would permit us to test without things slipping by having the new shim in the ISO.

Alternatively, we could test with everything as-is and ignore the shim issue, and just make sure we spin a new candidate with just that added in and verify *that* particular problem is fixed and still not slip.

Comment 58 Adam Williamson 2020-10-13 23:52:00 UTC
"we could _wait_ for the first candidate to be created on Friday or Saturday, which would permit us to test without things slipping by having the new shim in the ISO."

The problem there is that the time between Thursday and Tuesday isn't just there for grins, it's for releng and other teams (docs, websites etc) to get all the stuff that needs to happen between sign-off and actual release done. Asking them to do it between Sunday and Tuesday instead of Thursday and Tuesday is a big ask.

Comment 59 Kevin Fenzi 2020-10-14 01:38:18 UTC
Also, due to shim's position, we can't just test without it, add it and assume all is well. At least all the booting tests will need redoing to make sure it's ok, etc. 

I think we are just going to need to wait and see when we get the signed shim and act accordingly.

Comment 60 Vitaly 2020-10-15 09:44:08 UTC
Microsoft has revoked signatures of shim and Grub 2 due to the following CVEs:

* CVE-2020-7205;
* CVE-2020-15707;
* CVE-2020-15706;
* CVE-2020-15705;
* CVE-2020-14311;
* CVE-2020-14310;
* CVE-2020-14309;
* CVE-2020-14308;
* CVE-2020-10713.

Shim must be updated to the latest release signed with a new signature.

See also:

https://fwupd.org/lvfs/devices/org.linuxfoundation.dbx.ia32.firmware
https://fwupd.org/lvfs/devices/org.linuxfoundation.dbx.x64.firmware

For dual-boot users, Windows will install these updates automatically.

Comment 61 Adam Williamson 2020-10-15 23:32:11 UTC
Vitaly: that's not quite it, AIUI. Microsoft has not actually made Windows do the revocations yet. There is a general industry-wide consensus to do them next year...but Ubuntu has jumped the gun and started doing it already. So if you install a recent Ubuntu on a system, it does the revocations, and Fedora (and other things whose bootloaders haven't been re-signed yet, I guess) don't boot any more. See comments 22 and 35. Because Ubuntu has jumped the gun on this we're having to rush through re-signing shim when we previously had a plan to do it together with a new release.

Comment 62 Vitaly 2020-10-16 08:47:02 UTC
Got it. Thanks.

But fwupd will ship DBX updates soon too (already in testing).

Comment 63 André 2020-10-16 13:34:01 UTC
Fedora silverblue 33 doesn't boot with secure boot enabled and the latest dbx UEFI Revocation List File database update by using: https://uefi.org/revocationlistfile

Comment 64 Michael Catanzaro 2020-10-20 13:12:09 UTC
Any update, Peter? (Remember any additional release slip will be a two week slip, since we will not release on election day.)

Comment 65 Adam Williamson 2020-10-20 15:55:41 UTC
We talked to Peter in #fedora-devel yesterday; he was having issues with building shim on Fedora. Log extracts:

<bcotton> pjones: any update on the shim signing? we're discussing in #fedora-blocker-review right now
<pjones> bcotton: I'm still working on it; just did another test build a bit ago.
<adamw> pjones: what are the chances of getting it in bodhi by tomorrow or wednesday
<pjones> adamw: very low
<pjones> adamw: not zero, but a lot of good luck would be involved.
<pjones> adamw: Eighth_Doctor: current status is that if I build in f31/f32/f33 I get a pkcs7 verification error in openssl; if I build on rhel 8 I do not.
<pjones> so... trying to figure out wtf gcc is doing differently.
<pjones> awesomely the debuginfo is also about 1/3 the size with fedora's gcc.
<Eighth_Doctor> pjones: is there a build log we can look at?
<pjones> sure
<pjones> Eighth_Doctor: https://pjones.fedorapeople.org/f33.build.log <-- broked https://pjones.fedorapeople.org/el8.build.log <-- not broked
<pjones> if it helps, PKCS7_verify, which is the function that asserts the error, is in pkcs7_smime.c
<Eighth_Doctor> pjones: shim has a vendored openssl, doesn't it?
<pjones> yes.
<pjones> yeah also it's statically linked and doesn't have libc APIs available
<Eighth_Doctor> pjones: any chance you could upload a SRPM somewhere for me to take a look at it?
<pjones> Eighth_Doctor: you know, on account of generally not trying to push broken things to dist-git
<pjones> Eighth_Doctor: http://pjones.fedorapeople.org/shim-unsigned-x64-15.2-1.fc31.src.rpm
<pjones> Eighth_Doctor: okay, I think I see part of what's happening, working on isolating it better.
<Eighth_Doctor> 06:38:24> pjones: any progress on that bug?
<Eighth_Doctor> 06:38:29> with shim builds on F33
<pjones> 06:39:08> I think I know where the problem is, working on narrowing down /what/ the problem is now, which will lead to a solution.

Those last three messages are from 2 hrs 25 minutes ago as I write this comment. Remember that once pjones gets a successful build it still needs to actually get signed (AIUI), given that, chances of signing off a candidate with a re-signed shim on Thursday are basically zero at this point.

Comment 66 Kamil Páral 2020-10-21 15:09:34 UTC
Moving back to a proposed blocker based on FESCo retracting their Blocker decision:
https://pagure.io/fedora-qa/blocker-review/issue/135#comment-697231

Comment 67 Javier Martinez Canillas 2020-10-22 10:52:16 UTC
I want to point out that Ubuntu has reverted their DBX updates due this:

https://github.com/rhboot/shim-review/issues/120#issuecomment-713438405

https://launchpad.net/ubuntu/groovy/+source/secureboot-db/+changelog

Comment 68 Kamil Páral 2020-10-22 11:12:22 UTC
Javier, thanks for a helpful comment. What does this mean for existing PCs which already applied the dbx update before? Will the rollback "fix" those machines so that they can boot Fedora with its current shim again? Or does the rollback only help with PCs which were not updated before (with secureboot-db 1.6 or 1.7)?

Comment 69 Javier Martinez Canillas 2020-10-22 11:30:24 UTC
(In reply to Kamil Páral from comment #68)
> Javier, thanks for a helpful comment. What does this mean for existing PCs
> which already applied the dbx update before? Will the rollback "fix" those
> machines so that they can boot Fedora with its current shim again? Or does
> the rollback only help with PCs which were not updated before (with
> secureboot-db 1.6 or 1.7)?

Peter can correct me if I'm wrong on this, but my understanding is that DBX
updates can't be reverted. So it will be the latter, help users that didn't
install the secureboot-db > 1.5 updates yet.

Users that already installed will have to either disable Secure Boot or use
the BIOS setup to reset the DBX database to a factory default. After that,
installing the new secureboot-db 1.8 version won't deny Fedora's shim anymore. 

Also, the secureboot-db was rolled-out for Ubuntu 20.10 (Groovy Gorilla) and
to Ubuntu 20.04 (Focal Fossa) only to some extend, but since then the update
has been demoted from "update" to "proposed". You can check this info here:

https://launchpad.net/ubuntu/+source/secureboot-db/+publishinghistory

All this is to say that we shouldn't expect this to become a bigger issue with
more people affected.

Comment 70 Javier Martinez Canillas 2020-10-22 11:35:40 UTC
(In reply to Javier Martinez Canillas from comment #69)
> Also, the secureboot-db was rolled-out for Ubuntu 20.10 (Groovy Gorilla) and
> to Ubuntu 20.04 (Focal Fossa) only to some extend, but since then the update
> has been demoted from "update" to "proposed". You can check this info here:

I meant here the secureboot-db 1.6 that contained the Boot Hole DBX updates
that caused the Fedora shim to be denied. Sorry that I wasn't clear.

Comment 71 Chris Murphy 2020-10-22 19:41:55 UTC
Colin reports in c28 that he has a Dell laptop that's affected. If it's due to the Ubuntu dbx update, the last Q&A item in this support doc might help.

https://www.dell.com/support/article/en-bo/sln322287/additional-information-regarding-the-boothole-grub-vulnerability?lang=en

The gist is, this describes a process for clearing the Secure Boot databases, all of them. Every firmware implementation has a different interface for this. Some have an explicit option, like kparal's, to clear Secure Boot keys. It's also possible a "factory reset" will do this, but might go further and clear all NVRAM variables.

Comment 72 Chris Murphy 2020-10-22 19:58:16 UTC
Since the Dell sequence above doesn't suggest merely doing a "factory reset" I'm not certain all implementations will clear Secure Boot databases. But the Windows Hardware Compatibility Spec says:

\\\
If the firmware is reset to factory defaults, then any customized Secure Boot variables are also factory reset. If the
firmware settings are reset to factory defaults, all custom-set variables shall be erased and the OEM PKpub shall be re-
established along with the original, manufacturer-provisioned signature databases.
\\\

The dbx exists as in EFI variable, but again, I'm not certain a reset will remove the offending dbx. However, attempting to clear the Secure Boot databases is a much better option than "disable Secure Boot".

Comment 73 Adam Williamson 2020-10-22 21:42:08 UTC
To clear up blocker status here: after FESCo retracted it as a FESCo blocker - see https://bugzilla.redhat.com/show_bug.cgi?id=1883609#c66 - this was voted on under the normal criteria process in the Go/No-Go meeting today:

https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2020-10-22/f33-final-go_no_go-meeting.2020-10-22-17.00.log.html

and rejected as a blocker more or less on the grounds that the majority of voters didn't think enough people would run into it before Fedora 34 release, and that we can potentially release a subset or full set of rebuilt/updated images at some point during the 33 cycle if it is considered necessary.

Our current best understanding is that Ubuntu was shipping the DBX update to users (whether all or some subset) at some point but has now stopped doing that, and Microsoft will not ship the DBX update until Q2 2021.

Comment 74 Ben Cotton 2020-11-04 18:22:09 UTC
Accepted as a Fedora Prioritized Bug

Comment 75 Fedora Blocker Bugs Application 2021-03-08 19:55:13 UTC
Proposed as a Blocker for 34-final by Fedora user bcotton using the blocker tracking app because:

 Will violate the "Release-blocking images must boot" criterion once the updated DBX file is published. I believe we are expecting this after the F34 GA but before F35, so this is our last chance to fix the installer media.

Comment 76 Chris Murphy 2021-03-08 23:25:21 UTC
There's two issues: the original key revocation due to BootHole, and need for new signed shim; but there's also a replacement for DBX revocation that's generation based, due to the DBX taking up way too much space in NVRAM. And it possibly means a new GRUB as well. If it's going to happen in Fedora 34, we might want to have a plan for a test day where we ask as many folks as possible to download a nightly with the change and simply boot it. It'd be better if they can do an installation, but in theory the edge cases related to bootloader + firmware confusion should materialize just from booting the installer media with UEFI Secure Boot enabled.

Comment 77 Chris Murphy 2021-03-14 00:03:45 UTC
I spoke to Javier on IRC today, short summary is: there will be a new versions of shim (15.3) and grub2 (2.06-rc1) for Fedora 34 final, and they will be signed with new keys. This isn't landing in beta. Expect to see builds in Rawhide and Fedora 34 by Tuesday. Exact details for Fedora 32/33 forthcoming.

A test day for this is being discussed. https://pagure.io/fedora-qa/issue/663

Comment 78 Chris Murphy 2021-03-15 02:02:21 UTC
Agreed to close this bug (too long, multiple issues, some of it fixed itself) and move the remaining issue to new bug 1938630.

Comment 79 Chris Murphy 2021-03-15 02:07:03 UTC
I've transferred these over to the new bug.


Note You need to log in before you can comment on or make changes to this bug.