1. Please describe the problem: I get the following error from grub when trying to boot: ``` error: ../../grub-core/loader/i386/efi/linux.c:307:Chd0.gpt2/ostree/fedora-da43c8b4ba977e49d0cc9c0ae4eff847bf933c397e8bbaa8121e6319078835ca/vmlinuz-6.8.6-200.fc39.x86_64 has invalid signature. error: ../../grub-core/loader/i386/efi/linux.c:205:you need to load the kernel first. Press any key to continue… ``` 2. What is the Version-Release number of the kernel: 6.8.6-200 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 6.8.6-200, but I am unsure whether there was an update before 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes. Just boot. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Dunno how to do this on Silverblue? Should I just try a rebase to Fedora 40? 6. Are you running any modules that not shipped with directly Fedora's kernel?: no? 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Impossible, kernel is not even booted, so no logs are there. --- More details now here, coped/cross-posted from the Fedora community forum: https://discussion.fedoraproject.org/t/boot-issue-grub-in-fedora-silverblue-after-upgrading-latest-known-working-39-20240325-0-error-ostree-vmlinuz-has-invalid-signature-kernel-6-8-6-200-fc39/113484 This system is old and for years I had multiple system upgrades (probably started with sth. like Fedora 36 or 37 or so). However, some time ago it broke booting. So I just used the cool #silverblue powers to choose the version before and I have to say unfortunately kinda forgot this. Now, I remembered and tried upgrading again, seeing whether it would possible have been fixed in the while and working, but no it does not… unfortunately. Thus, I am writing… So, the current update today did _not_ cause the bug, but unfortunately I AFAIK cannot discover which version `rpm-ostree` has been layering before I upgraded it (obviously I am booting into the old version and upgrading it there, I forgot to take a note of the packages there, sorry). `39.20240325.0` is the latest version I am currently using that is known to be working. ## STR Booting… Grub shows selection, I choose the latest one:  (Don't ask me why it shows it twice, I don't care, it has always worked like this and this has been duplicated a long time ago, likely.) The first, current version is definitively broken (`39.20240417.0`) as it then directly shows this:  ``` error: ../../grub-core/loader/i386/efi/linux.c:307:Chd0.gpt2/ostree/fedora-da43c8b4ba977e49d0cc9c0ae4eff847bf933c397e8bbaa8121e6319078835ca/vmlinuz-6.8.6-200.fc39.x86_64 has invalid signature. error: ../../grub-core/loader/i386/efi/linux.c:205:you need to load the kernel first. Press any key to continue… ``` (I hope this makes search engines find these, I tried my best typing/OCR'ing this from the screenshot.) To do this, I found the file it complains about at: `/ostree/boot.0.1/fedora/da43c8b4ba977e49d0cc9c0ae4eff847bf933c397e8bbaa8121e6319078835ca/0/usr/lib/modules/6.8.6-200.fc39.x86_64/vmlinuz` Striking, there is an HMAC file next to it. I have no idea how exactly it is calculated, but `sha512hmac` actually seems to match it(?): ``` $ pwd /ostree/boot.0.1/fedora/da43c8b4ba977e49d0cc9c0ae4eff847bf933c397e8bbaa8121e6319078835ca/0/usr/lib/modules/6.8.6-200.fc39.x86_64 $ cat .vmlinuz.hmac b1f3eeee1491911730ce604dc478222f64b52dcf796273ef26157cd36c1fd2f01f075d61d00e43d379521d1652a143ab21d54e9585d61698fbf7e53246114dc7 vmlinuz-6.8.6-200.fc39.x86_64 $ sha512hmac vmlinuz b1f3eeee1491911730ce604dc478222f64b52dcf796273ef26157cd36c1fd2f01f075d61d00e43d379521d1652a143ab21d54e9585d61698fbf7e53246114dc7 vmlinuz $ sha512sum vmlinuz 60ca1640030655ff0e28ca65a5d94d8c3741425b603b7f3917cefe7c546754a63ab3103f8a1f56231d9991d7583b3a264eadba25c60c93141b621cb5fb9e1e89 vmlinuz ``` So is the file corrupt? If anyone has the proper Linux kernel [compressed file aka vmlinuz](https://serverfault.com/questions/429652/what-is-vmlinuz-and-why-do-i-care), please let me know/send it to me, I can freely make some diffs or so. But if it is not corrupt, why does grub complain about an invalid signature? ## System ``` $ rpm-ostree status -v State: idle AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot Deployments: fedora:fedora/39/x86_64/silverblue (index: 0) Version: 39.20240417.0 (2024-04-17T00:39:00Z) BaseCommit: 4df34ac077503fe659aac6abbfe2e68699c4cad2f162a136d2f9cfde6c32e71e ├─ repo-0 (2023-11-01T00:12:39Z) ├─ repo-1 (2024-04-17T00:16:31Z) └─ repo-2 (2024-04-17T00:24:03Z) Commit: 9c4e874d6f1cbb74eb0ad4c2745c8573b5bb15dbdccf2486f06c311a451c667b ├─ fedora (2023-11-01T00:12:39Z) ├─ fedora-cisco-openh264 (2023-12-12T17:22:46Z) ├─ rpmfusion-free (2023-11-04T16:49:08Z) ├─ rpmfusion-free-updates (2024-04-15T16:04:13Z) ├─ updates (2024-04-17T02:06:56Z) └─ updates-archive (2024-04-17T02:54:03Z) Staged: no StateRoot: fedora GPGSignature: 1 signature Signature made Mi 17 Apr 2024 02:40:08 CEST using RSA key ID 75CF5AC418B8E74C Good signature from "Fedora <fedora-39-primary>" Upgraded: alsa-sof-firmware 2023.12.1-1.fc39 -> 2024.03-2.fc39 amd-gpu-firmware 20240312-1.fc39 -> 20240410-1.fc39 amd-ucode-firmware 20240312-1.fc39 -> 20240410-1.fc39 at-spi2-atk 2.50.1-1.fc39 -> 2.50.2-1.fc39 at-spi2-core 2.50.1-1.fc39 -> 2.50.2-1.fc39 atheros-firmware 20240312-1.fc39 -> 20240410-1.fc39 atk 2.50.1-1.fc39 -> 2.50.2-1.fc39 atkmm 2.28.3-3.fc39 -> 2.28.4-1.fc39 b43-fwcutter 019-24.fc39 -> 019-36.fc39 bluez 5.72-1.fc39 -> 5.73-3.fc39 bluez-cups 5.72-1.fc39 -> 5.73-3.fc39 bluez-libs 5.72-1.fc39 -> 5.73-3.fc39 bluez-obexd 5.72-1.fc39 -> 5.73-3.fc39 bolt 0.9.6-2.fc39 -> 0.9.7-1.fc39 brcmfmac-firmware 20240312-1.fc39 -> 20240410-1.fc39 breeze-icon-theme 5.113.0-1.fc39 -> 5.115.0-1.fc39 btrfs-progs 6.7.1-1.fc39 -> 6.8-1.fc39 buildah 1.35.0-1.fc39 -> 1.35.3-1.fc39 c-ares 1.25.0-1.fc39 -> 1.28.1-1.fc39 cirrus-audio-firmware 20240312-1.fc39 -> 20240410-1.fc39 emacs-filesystem 1:29.2-2.fc39 -> 1:29.3-1.fc39 epiphany-runtime 1:45.2-1.fc39 -> 1:45.3-1.fc39 expat 2.6.0-1.fc39 -> 2.6.2-1.fc39 firefox 124.0.1-1.fc39 -> 125.0-1.fc39 firefox-langpacks 124.0.1-1.fc39 -> 125.0-1.fc39 firewalld 2.0.3-1.fc39 -> 2.0.4-1.fc39 firewalld-filesystem 2.0.3-1.fc39 -> 2.0.4-1.fc39 fwupd 1.9.15-1.fc39 -> 1.9.16-1.fc39 fwupd-plugin-flashrom 1.9.15-1.fc39 -> 1.9.16-1.fc39 fwupd-plugin-modem-manager 1.9.15-1.fc39 -> 1.9.16-1.fc39 fwupd-plugin-uefi-capsule-data 1.9.15-1.fc39 -> 1.9.16-1.fc39 glib-networking 2.78.0-1.fc39 -> 2.78.1-1.fc39 glibc 2.38-16.fc39 -> 2.38-17.fc39 glibc-all-langpacks 2.38-16.fc39 -> 2.38-17.fc39 glibc-common 2.38-16.fc39 -> 2.38-17.fc39 glibc-gconv-extra 2.38-16.fc39 -> 2.38-17.fc39 gnome-tweaks 45.1-1.fc39 -> 45.2-1.fc39 gnome-user-docs 45.1-1.fc39 -> 45.5-1.fc39 gtkmm3.0 3.24.8-1.fc39 -> 3.24.9-1.fc39 hplip 3.23.12-2.fc39 -> 3.23.12-6.fc39 hplip-common 3.23.12-2.fc39 -> 3.23.12-6.fc39 hplip-libs 3.23.12-2.fc39 -> 3.23.12-6.fc39 hwdata 0.380-1.fc39 -> 0.381-1.fc39 intel-audio-firmware 20240312-1.fc39 -> 20240410-1.fc39 intel-gpu-firmware 20240312-1.fc39 -> 20240410-1.fc39 iwlegacy-firmware 20240312-1.fc39 -> 20240410-1.fc39 iwlwifi-dvm-firmware 20240312-1.fc39 -> 20240410-1.fc39 iwlwifi-mvm-firmware 20240312-1.fc39 -> 20240410-1.fc39 kernel 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-core 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-modules 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-modules-core 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-modules-extra 6.7.9-200.fc39 -> 6.8.6-200.fc39 libblkid 2.39.3-6.fc39 -> 2.39.4-1.fc39 libblockdev 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-btrfs 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-crypto 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-dm 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-fs 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-loop 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-lvm 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-mdraid 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-mpath 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-nvdimm 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-nvme 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-part 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-plugins-all 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-swap 3.1.0-1.fc39 -> 3.1.1-1.fc39 libblockdev-utils 3.1.0-1.fc39 -> 3.1.1-1.fc39 libbsd 0.11.7-5.fc39 -> 0.12.2-1.fc39 libeconf 0.5.2-1.fc39 -> 0.5.2-2.fc39 libertas-firmware 20240312-1.fc39 -> 20240410-1.fc39 libfdisk 2.39.3-6.fc39 -> 2.39.4-1.fc39 libgweather4 4.4.0-1.fc39 -> 4.4.2-1.fc39 libinput 1.25.0-1.fc39 -> 1.25.0-4.fc39 libmount 2.39.3-6.fc39 -> 2.39.4-1.fc39 libnfsidmap 1:2.6.4-0.rc5.fc39 -> 1:2.6.4-0.rc6.fc39 libopenmpt 0.6.12-1.fc39 -> 0.7.6-1.fc39 libphonenumber 8.13.30-1.fc39 -> 8.13.33-1.fc39 librepo 1.17.0-1.fc39 -> 1.17.1-1.fc39 libsane-hpaio 3.23.12-2.fc39 -> 3.23.12-6.fc39 libsmartcols 2.39.3-6.fc39 -> 2.39.4-1.fc39 libsmbclient 2:4.19.5-1.fc39 -> 2:4.19.6-1.fc39 libuuid 2.39.3-6.fc39 -> 2.39.4-1.fc39 libwbclient 2:4.19.5-1.fc39 -> 2:4.19.6-1.fc39 libxmlb 0.3.15-1.fc39 -> 0.3.18-1.fc39 libzstd 1.5.5-4.fc39 -> 1.5.6-1.fc39 linux-firmware 20240312-1.fc39 -> 20240410-1.fc39 linux-firmware-whence 20240312-1.fc39 -> 20240410-1.fc39 mbedtls 2.28.7-1.fc39 -> 2.28.8-1.fc39 mt7xxx-firmware 20240312-1.fc39 -> 20240410-1.fc39 nextcloud-client 3.12.1-1.fc39 -> 3.12.3-1.fc39 nextcloud-client-libs 3.12.1-1.fc39 -> 3.12.3-1.fc39 nextcloud-client-nautilus 3.12.1-1.fc39 -> 3.12.3-1.fc39 nfs-utils 1:2.6.4-0.rc5.fc39 -> 1:2.6.4-0.rc6.fc39 nspr 4.35.0-18.fc39 -> 4.35.0-19.fc39 nss 3.98.0-1.fc39 -> 3.99.0-1.fc39 nss-softokn 3.98.0-1.fc39 -> 3.99.0-1.fc39 nss-softokn-freebl 3.98.0-1.fc39 -> 3.99.0-1.fc39 nss-sysinit 3.98.0-1.fc39 -> 3.99.0-1.fc39 nss-util 3.98.0-1.fc39 -> 3.99.0-1.fc39 nvidia-gpu-firmware 20240312-1.fc39 -> 20240410-1.fc39 nxpwireless-firmware 20240312-1.fc39 -> 20240410-1.fc39 ostree 2024.4-1.fc39 -> 2024.5-1.fc39 ostree-grub2 2024.4-1.fc39 -> 2024.5-1.fc39 ostree-libs 2024.4-1.fc39 -> 2024.5-1.fc39 passt 0^20240220.g1e6f92b-1.fc39 -> 0^20240326.g4988e2b-1.fc39 passt-selinux 0^20240220.g1e6f92b-1.fc39 -> 0^20240326.g4988e2b-1.fc39 podman 5:4.9.3-1.fc39 -> 5:4.9.4-1.fc39 podman-plugins 5:4.9.3-1.fc39 -> 5:4.9.4-1.fc39 power-profiles-daemon 0.20-1.fc39 -> 0.21-2.fc39 python3-blockdev 3.1.0-1.fc39 -> 3.1.1-1.fc39 python3-firewall 2.0.3-1.fc39 -> 2.0.4-1.fc39 python3-pillow 10.2.0-1.fc39 -> 10.3.0-1.fc39 qadwaitadecorations-qt5 0.1.4-2.fc39 -> 0.1.5-1.fc39 realtek-firmware 20240312-1.fc39 -> 20240410-1.fc39 rpm-ostree 2024.3-3.fc39 -> 2024.4-6.fc39 rpm-ostree-libs 2024.3-3.fc39 -> 2024.4-6.fc39 samba-client 2:4.19.5-1.fc39 -> 2:4.19.6-1.fc39 samba-client-libs 2:4.19.5-1.fc39 -> 2:4.19.6-1.fc39 samba-common 2:4.19.5-1.fc39 -> 2:4.19.6-1.fc39 samba-common-libs 2:4.19.5-1.fc39 -> 2:4.19.6-1.fc39 skopeo 1:1.14.2-1.fc39 -> 1:1.15.0-1.fc39 system-config-printer-libs 1.5.18-5.fc39 -> 1.5.18-9.fc39 system-config-printer-udev 1.5.18-5.fc39 -> 1.5.18-9.fc39 tiwilink-firmware 20240312-1.fc39 -> 20240410-1.fc39 util-linux 2.39.3-6.fc39 -> 2.39.4-1.fc39 util-linux-core 2.39.3-6.fc39 -> 2.39.4-1.fc39 vim-data 2:9.1.181-1.fc39 -> 2:9.1.309-1.fc39 vim-minimal 2:9.1.181-1.fc39 -> 2:9.1.309-1.fc39 xorg-x11-server-Xorg 1.20.14-30.fc39 -> 1.20.14-35.fc39 xorg-x11-server-common 1.20.14-30.fc39 -> 1.20.14-35.fc39 Removed: python3-pycurl-7.45.2-5.fc39.x86_64 LayeredPackages: adb blivet-gui brightnessctl btop dconf-editor git git-credential-libsecret git-subtree *** pipewire-codec-aptx podman-compose rpmfusion-free-release *** ● fedora:fedora/39/x86_64/silverblue (index: 1) Version: 39.20240325.0 (2024-03-25T00:37:19Z) BaseCommit: 8b2ab1dc8e53e928d23de9ed3be548c0338c3dec3fcb1c28b1caa0df70b35b7f ├─ repo-0 (2023-11-01T00:12:39Z) ├─ repo-1 (2024-03-25T00:16:24Z) └─ repo-2 (2024-03-25T00:23:02Z) Commit: def9b08e2c7fe12425846c7be578337db17535c8bf605fbf291e18a27cf83cc4 ├─ fedora (2023-11-01T00:12:39Z) ├─ fedora-cisco-openh264 (2023-12-12T17:22:46Z) ├─ rpmfusion-free (2023-11-04T16:49:08Z) ├─ rpmfusion-free-updates (2024-03-24T12:21:36Z) ├─ updates (2024-03-25T01:01:46Z) └─ updates-archive (2024-03-25T01:29:06Z) StateRoot: fedora GPGSignature: 1 signature Signature made Mo 25 Mär 2024 01:38:19 CET using RSA key ID 75CF5AC418B8E74C Good signature from "Fedora <fedora-39-primary>" LayeredPackages: adb blivet-gui brightnessctl btop dconf-editor git git-credential-libsecret git-subtree *** pipewire-codec-aptx podman-compose rpmfusion-free-release *** ``` What is striking are… * the ostree/grub upgrades: ``` ostree 2024.4-1.fc39 -> 2024.5-1.fc39 ostree-grub2 2024.4-1.fc39 -> 2024.5-1.fc39 ostree-libs 2024.4-1.fc39 -> 2024.5-1.fc39 ``` * the kernel upgrades: ``` kernel 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-core 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-modules 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-modules-core 6.7.9-200.fc39 -> 6.8.6-200.fc39 kernel-modules-extra 6.7.9-200.fc39 -> 6.8.6-200.fc39 ``` Booted from the old, working version (as obviously I can only run the commands there): ``` $ uname -a Linux **** 6.7.9-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 6 19:35:04 UTC 2024 x86_64 GNU/Linux ``` BTW I also tried to self-help me [and asked ChatGPT](https://chat.openai.com/share/2682b09c-1208-4efb-b5ff-37b7ba827513), but it mostly just did the parrot and explained me possible causes/ideas to investigate I already know… :thinking: [Fedora Kernel/booting troubleshooting docs also did not help](https://docs.fedoraproject.org/en-US/quick-docs/kernel-troubleshooting/). Cross-posted from the Fedora community forum: https://discussion.fedoraproject.org/t/boot-issue-grub-in-fedora-silverblue-after-upgrading-latest-known-working-39-20240325-0-error-ostree-vmlinuz-has-invalid-signature-kernel-6-8-6-200-fc39/113484
See https://github.com/fedora-silverblue/issue-tracker/issues/543. This is likely due to the fact that Atomic Desktops do not (yet) update the bootloader on update.