Bug 2127995 - BOOTX64.EFI on f36 Silverblue needs update to be able to upgrade UEFI dbx
Summary: BOOTX64.EFI on f36 Silverblue needs update to be able to upgrade UEFI dbx
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm-ostree
Version: 36
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Colin Walters
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-19 14:40 UTC by Armin Warda
Modified: 2022-11-25 17:03 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FC-612 0 None None None 2022-09-19 18:32:03 UTC

Description Armin Warda 2022-09-19 14:40:11 UTC
Description of problem:

UEFI firmware update blocked due to BOOTX64.EFI (grub2 2.06) Authenticode checksum present in dbx, probably affected by CVE-2020-10713,CVE-2020-14308,CVE-2020-14309,CVE-2020-14310,CVE-2020-14311,CVE-2020-15705,CVE-2020-15706 and CVE-2020-15707

Version-Release number of selected component (if applicable):

Fedora 36 Silverblue
fedora/36/x86_64/silverblue 36.20220919.0
grub 2.06

Steps to Reproduce:

1. sudo fwupdmgr --force refresh
2. sudo fwupdmgr update

Actual results:

blocked executable in the ESP, ensure grub and shim are up to date: /boot/efi/EFI/BOOT/BOOTX64.EFI Authenticode checksum [0ce02100f67c7ef85f4eed368f02bf7092380a3c23ca91fd7f19430d94b00c19] is present in dbx

Expected results:

rpm-ostree update should update BOOTX64.EFI
UEFI Device Firmware dbx can be upgraded

Additional info:

Lenovo ThinkPad X1 Carbon 7th Gen.

https://blogs.gnome.org/hughsie/2020/08/17/updating-secure-boot-dbx-with-fwupd-and-the-lvfs/

awarda@fedora:~$ sudo fwupdmgr --force refresh
Updating lvfs
Downloading…             [***************************************]
Downloading…             [***************************************]
Successfully downloaded new metadata: 9 local devices supported
awarda@fedora:~$ sudo fwupdmgr update
Devices with no available firmware updates: 
 • USB2.0 Hub
 • USB2.0 Hub
 • USB3.1 Hub
 • USB3.1 Hub
 • ThinkPad Thunderbolt 3 Dock
 • ThinkPad Thunderbolt 3 Dock USB Audio
 • UEFI Device Firmware
Devices with the latest available firmware version:
 • Embedded Controller
 • Intel Management Engine
 • MZVLB256HBHQ-000L7
 • Prometheus
 • Prometheus IOTA Config
 • System Firmware
 • Thunderbolt host controller
 • UEFI Device Firmware
╔══════════════════════════════════════════════════════════════════════════════╗
║ Upgrade UEFI dbx from 77 to 217?                                             ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ This updates the dbx to the latest release from Microsoft which adds         ║
║ insecure versions of grub and shim to the list of forbidden signatures due   ║
║ to multiple discovered security updates.                                     ║
║                                                                              ║
║ UEFI dbx and all connected devices may not be usable while updating.         ║
╚══════════════════════════════════════════════════════════════════════════════╝

Perform operation? [Y|n]: y
Downloading…             [***************************************]
Decompressing…           [***************************************]
Authenticating…          [***************************************]
Waiting…                 [***************************************]
Writing…                 [***************************************]
Decompressing…           [                                       ]
Blocked executable in the ESP, ensure grub and shim are up to date: /boot/efi/EFI/BOOT/BOOTX64.EFI Authenticode checksum [0ce02100f67c7ef85f4eed368f02bf7092380a3c23ca91fd7f19430d94b00c19] is present in dbx

awarda@fedora:~$ cat /etc/redhat-release 
Fedora release 36 (Thirty Six)

awarda@fedora:~$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220919.0 (2022-09-19T00:44:56Z)
               BaseCommit: a36d2542999f5e374772e1b34abeee25e102a771be4905a564f118feb21033e4
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: f33-backgrounds-base.noarch f33-backgrounds-extras-base.noarch
                           f33-backgrounds-extras-gnome.noarch f33-backgrounds-gnome.noarch gnome-tweaks
                           google-chrome-stable igt-gpu-tools libguestfs-tools libvirt libvirt-daemon-config-network
                           libvirt-daemon-kvm libvirt-nss python3-libguestfs qemu-kvm virt-install virt-manager
                           virt-top virt-viewer

  fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220916.0 (2022-09-16T00:55:03Z)
               BaseCommit: 39c6553ee9ddb322305db3f695b2448184931cd62eac1d5be5dc7166a35fe32c
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: f33-backgrounds-base.noarch f33-backgrounds-extras-base.noarch
                           f33-backgrounds-extras-gnome.noarch f33-backgrounds-gnome.noarch gnome-tweaks
                           google-chrome-stable igt-gpu-tools libguestfs-tools libvirt libvirt-daemon-config-network
                           libvirt-daemon-kvm libvirt-nss python3-libguestfs qemu-kvm virt-install virt-manager
                           virt-top virt-viewer

awarda@fedora:~$ grub2-set-default --version
grub2-set-default (GRUB) 2.06

Comment 1 Robbie Harwood 2022-09-19 18:30:23 UTC
grub2 isn't responsible for updating DBX - fwupd is.  Nor is BOOTX64.EFI, which is part of shim.

> blocked executable in the ESP, ensure grub and shim are up to date: /boot/efi/EFI/BOOT/BOOTX64.EFI Authenticode checksum [0ce02100f67c7ef85f4eed368f02bf7092380a3c23ca91fd7f19430d94b00c19] is present in dbx

Did you ensure that grub and shim are up to date?  What fwupd is telling you is that if you apply the DBX update, your system will be unbootable because you have a version of shim that is outdated installed on your system.  You need to update it.

> awarda@fedora:~$ grub2-set-default --version
> grub2-set-default (GRUB) 2.06

Like most tools, this tells you the upstream release, not the actual component version.  It's also not the right package.  A more useful command would be something like `rpm -qv shim-x64`, or if you don't know the package it comes from, `rpm -qf /boot/efi/EFI/BOOT/BOOTX64.EFI`, both of which yield a result like "shim-x64-15.6-2.x86_64".

Comment 2 Armin Warda 2022-09-19 19:52:31 UTC
There is no rpm package manager in Silverblue, this fedora variant is ostree based. 
Thus grub2 & shim would have to be updated with an atomic rpm-ostree update:

warda@fedora:~$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220919.0 (2022-09-19T00:44:56Z)
               BaseCommit: a36d2542999f5e374772e1b34abeee25e102a771be4905a564f118feb21033e4
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: f33-backgrounds-base.noarch f33-backgrounds-extras-base.noarch
                           f33-backgrounds-extras-gnome.noarch f33-backgrounds-gnome.noarch gnome-tweaks
                           google-chrome-stable igt-gpu-tools libguestfs-tools libvirt libvirt-daemon-config-network
                           libvirt-daemon-kvm libvirt-nss python3-libguestfs qemu-kvm virt-install virt-manager
                           virt-top virt-viewer

awarda@fedora:~$ rpm-ostree db list a36d25 |grep -E "shim|grub"
 grub2-common-1:2.06-53.fc36.noarch
 grub2-efi-ia32-1:2.06-53.fc36.x86_64
 grub2-efi-x64-1:2.06-53.fc36.x86_64
 grub2-pc-1:2.06-53.fc36.x86_64
 grub2-pc-modules-1:2.06-53.fc36.noarch
 grub2-tools-1:2.06-53.fc36.x86_64
 grub2-tools-minimal-1:2.06-53.fc36.x86_64
 grubby-8.40-67.fc36.x86_64
 ostree-grub2-2022.5-2.fc36.x86_64
 shim-ia32-15.6-2.x86_64
 shim-x64-15.6-2.x86_64

Comment 3 Timothée Ravier 2022-09-19 20:16:33 UTC
This is https://github.com/fedora-silverblue/issue-tracker/issues/120. Workaround is to update manually from the content of /usr/lib/ostree-boot/.
Hopefully soon (F38) we will have bootupd for Silverblue (see issue above).

Comment 5 Armin Warda 2022-09-22 06:07:08 UTC
Timothée, are you proposing this?

sudo cp /usr/lib/ostree-boot/efi/EFI/BOOT/*64* /boot/efi/EFI/BOOT/

---

awarda@fedora:~$ sudo ls -la /usr/lib/ostree-boot/efi/EFI/BOOT/
total 1820
drwx------.  1 root root     84  1. Jan 1970  .
drwxr-xr-x.  1 root root     20  1. Jan 1970  ..
-rwx------.  7 root root 742064  1. Jan 1970  BOOTIA32.EFI
-rwx------. 10 root root 946712  1. Jan 1970  BOOTX64.EFI
-rwx------.  4 root root  70776  1. Jan 1970  fbia32.efi
-rwx------.  4 root root  90280  1. Jan 1970  fbx64.efi

awarda@fedora:~$ sudo rpm -qf /usr/lib/ostree-boot/efi/EFI/BOOT/BOOTX64.EFI
file /usr/lib/ostree-boot/efi/EFI/BOOT/BOOTX64.EFI is not owned by any package

awarda@fedora:~$ sudo ls -la /boot/efi/EFI/BOOT/
total 1544
drwx------. 2 root root    4096  1. Jan 1980  .
drwx------. 4 root root    4096  1. Jan 1980  ..
-rwx------. 1 root root 1210776  1. Jan 1980  BOOTX64.EFI
-rwx------. 1 root root  357248  1. Jan 1980  fbx64.efi

awarda@fedora:~$ sudo rpm -qf /boot/efi/EFI/BOOT/BOOTX64.EFI
shim-x64-15.6-2.x86_64

Comment 6 David Jaša 2022-09-22 10:22:37 UTC
I created a 'fedora-new' directory as a copy of /usr/lib/ostree-boot/efi/EFI/fedora from which I deleted ia32 files, ending up here:

# ls fedora*
fedora:
BOOTX64.CSV  fonts  fw  fwupdx64.efi  grub.cfg  grub.cfg.old  grubenv  grubx64.efi  mmx64.efi  shim.efi  shimx64.efi  shimx64-fedora.efi

fedora-new:
BOOTX64.CSV  grubx64.efi  mmx64.efi  shim.efi  shimx64.efi

so apart from fonts/ and fw/ subdirectories and grub configs, new dir is missing fwupdx64.efi and shimx64-fedora.efi. Will the system be able to boot 1) with old versions of them, or 2) without them?

(In reply to Timothée Ravier from comment #3)
> Hopefully soon (F38) we will have bootupd for Silverblue (see issue above).

Given that:
* this is now equivalent of leaving CVEs unfixed in three active branches (IIUC 36, 37 and pre-bootupd 38)
* the workaround is neither straightforward nor obvious even for people with some experience
* SB is frequently deployed as hard-to-break system that is trusted to update itself safely so users who can not manage their computers are secure as far as system allow
* this issue is first of its kind in 10 years
we need some sort of hotfix sooner than later...

Comment 7 Robbie Harwood 2022-09-22 17:57:21 UTC
Fedora 38 is way too late.

The DBX update is a security update.  It prohibits booting known-vulnerable systems.  The shim update to 15.6 is also CVE fix.

Leaving released fedoras without a path to apply security updates isn't workable - they need to have these applied.

Comment 8 Colin Walters 2022-09-23 20:15:31 UTC
Bigger picture, a problem is that we've let Fedora Silverblue, IoT and CoreOS drift.  We deployed bootupd on FCOS some time ago for this, and as of recently it's been added to F38SB, but not IoT.

It's trivial to backport to earlier branches, I started that at
https://pagure.io/workstation-ostree-config/pull-request/303

(Hmm, actually it looks like the earlier PR was missing some wiring up at the build side, specifically we need this bit https://github.com/coreos/fedora-coreos-config/blob/e9ea2831f41bfb5456312774ba79649b1d2c25e1/manifests/bootupd.yaml#L6 )

Once it's shipped there we'll do a bit of testing and document how to turn it on.

When we're talking about security though, bear in mind that this is a consequence of us making *every other upgrade* transactional and safe.  I've lost track of how many people I've seen have their laptop run out of battery or a kernel panics (or more fun, something takes down the desktop during live updates which kills the terminal running traditional yum/dnf) and then they have a half-updated broken system and they learn to fear operating system updates - and avoid doing it often.

The reason bootupd isn't on by default is because we haven't proven out transactional bootloader updates.  Which means doing so particularly on a desktop system should have a screen like one sees on other operating systems that says e.g. "Please don't turn off your computer" if we were doing it correctly.  (But better of course to make it transactional!)

This bug got filed against rpm-ostree, but it's really more of a bug against "image based updates", of which rpm-ostree is just one mechanism.  (We've tried to avoid tightly tying together rpm-ostree and bootupd too, although they clearly could use better integration even)

I'd say again going back to the top the root issue is drift between our various image-based systems, and I've got some work in the background to address that.

Comment 9 Jens Petersen 2022-11-02 03:00:36 UTC
Thanks, Colin, for explaining the direction in detail

I have to agree with others that it is getting increasingly uncomfortable
still not being able to apply firmware updates on Silverblue.

(In reply to Colin Walters from comment #8)
> Once it's shipped there we'll do a bit of testing and document how to turn
> it on.

How close are we now to be able to do this for F37SB users? :)

Comment 10 Boricua 2022-11-21 18:04:47 UTC
I share this info in case is useful. A few days ago, my Intel motherboard died and I chose as replacement the ASRock B550M PG Riptide, with AMD Ryzen 7 5700G with Radeon. This is Fedora Silverblue 37. Everything worked OK, except that I was hit by this same bug.

Comment 11 Armin Warda 2022-11-24 09:06:17 UTC
Do I interpret the efibootmgr output below correctly, that only these three files are accessed from the local drive during boot, and that BOOTX64.EFI is not accessed at all during boot of my Fedora 37 Silverbue?

awarda@fedora:~$ efibootmgr -v|grep File|grep HD
Boot0000* Red Hat Enterprise Linux HD(1,GPT,26a6453f-d383-47f6-a1d3-35ad8bc99513,0x800,0x64000)/File(\EFI\redhat\shimx64.efi)
Boot0001* Fedora HD(1,GPT,a774f63b-e5e4-4a3c-9712-a6785d68cfa1,0x800,0x12c000)/File(\EFI\fedora\shimx64.efi)
Boot0002* Linux-Firmware-Updater HD(1,GPT,a774f63b-e5e4-4a3c-9712-a6785d68cfa1,0x800,0x12c000)/File(\EFI\fedora\fwupdx64.efi)

Is the \EFI\redhat\ a potential left-over from my previous, deleted RHEL installation on this disk?

It looks like my Fedora 37 Silverblue and the Firmware Updater use the directory \EFI\fedora\ instead.

Thus it should be safe for me to delete /boot/efi/EFI/BOOT/BOOTX64.EFI in my Fedora 37 Silverblue?

Suggestion: instead of complaining about outdated, unused EFI files, fwupdmgr chould try to determine which EFI files are in (multi-) boot sequences and only complain about EFI files that are actually used.

Comment 12 Adam Williamson 2022-11-24 18:29:59 UTC
While people work on fixing this, can anyone confirm that Armin's suggestion from https://bugzilla.redhat.com/show_bug.cgi?id=2127995#c5 is good as a workaround? I'm 99% sure it is, but it would be good to have confirmation. We should at least document this on Common Bugs, as it's likely more and more users will be affected by it as time goes on and vendors update their revocation lists.

Comment 13 Adam Williamson 2022-11-25 07:30:02 UTC
I see there's also a workaround suggested in the silverblue issue: https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-1177515110 . Not sure which one would be better to document.

Comment 14 Armin Warda 2022-11-25 11:27:55 UTC
Looking around in my f37 Silverblue, I found the ostree-grub2-2022.6-1.fc37.x86_64 RPM installed. This seems to be taking care of updating GRUB in an OSTREE (vs. regular RPM) based Fedora system via /etc/grub.d/15_ostree. 

Could we do something similar for shim, fwupd, and the other EFI stuff, automatically updating the stuff in /boot/efi/EFI/ from /usr/lib/ostree-boot/efi/EFI?

Comment 15 Adam Williamson 2022-11-25 17:03:03 UTC
Well, that's more or less what switching to bootupd will achieve. bootupd is replacing ostree-grub2 , and will solve this problem and maybe some others.


Note You need to log in before you can comment on or make changes to this bug.