1. Please describe the problem: After dnf system-upgrade, f39 kernel does not appear in the grub menu and the f37 kernel starts (by default). 2. What is the Version-Release number of the kernel: kernel-core-6.5.9-100.fc37.x86_64 is OK and boots kernel-core-6.5.9-300.fc39.x86_64 install scriptlet fails 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : See above. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes. After reboot in f39 (with f37 kernel), I tried to reinstall without success: /// # dnf reinstall kernel-core Last metadata expiration check: 1:02:35 ago on Wed 08 Nov 2023 02:09:52 AM CET. Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Reinstalling: kernel-core x86_64 6.5.9-300.fc39 updates 16 M Transaction Summary ================================================================================ Total download size: 16 M Installed size: 65 M Is this ok [y/N]: y Downloading Packages: kernel-core-6.5.9-300.fc39.x86_64.rpm 3.7 MB/s | 16 MB 00:04 -------------------------------------------------------------------------------- Total 2.9 MB/s | 16 MB 00:05 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Reinstalling : kernel-core-6.5.9-300.fc39.x86_64 1/2 Running scriptlet: kernel-core-6.5.9-300.fc39.x86_64 1/2 Running scriptlet: kernel-core-6.5.9-300.fc39.x86_64 2/2 Cleanup : kernel-core-6.5.9-300.fc39.x86_64 2/2 Running scriptlet: kernel-core-6.5.9-300.fc39.x86_64 2/2 /usr/lib/kernel/install.d/51-dracut-rescue.install: line 81: /boot/efi/loader/entries/1604a63a532947d99d18bf17a754e833-0-rescue.conf: No such file or directory /usr/lib/kernel/install.d/51-dracut-rescue.install failed with exit status 1. Verifying : kernel-core-6.5.9-300.fc39.x86_64 1/2 Verifying : kernel-core-6.5.9-300.fc39.x86_64 2/2 Reinstalled: kernel-core-6.5.9-300.fc39.x86_64 Complete! /// 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: N.A. 6. Are you running any modules that not shipped with directly Fedora's kernel?: N.A. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. no boot entry --> no log Additional info: - This is an rpm package scriptlet problem, not a kernel executable bug. - Package gets installed, but is unusable. - System not reinstalled since f31: only upgraded to f33, f35, f37 and now f39 since then. - The loader subdirectory is located in /boot, not /boot/efi. - Might be a dracut problem. If this is the case, thanks to reassign bug. - IMPORTANT: It probably needs be fixed within the next 3 updates (before f37 kernel gets removed). Reproducible: Always
Still occurs with kernel-core-6.5.10-300.fc39.x86_64
The rescue install bit failing isn't a script failure. It would be good to look at exactly what is failing. While none of this is actually in kernel, it could be in systemd (kernel-install), dracut, or grub. Do you see the kernel and similar files in your /boot directory? If so, which? Do you have an initramfs?
Thanks for your reply. > The rescue install bit failing isn't a script failure. It would be good to look at exactly what is failing. I've manually added "echo $@; set -x" at the start of /usr/lib/kernel/install.d/51-dracut-rescue.install and logged the output of "dnf reinstall kernel-core". Please find it in attachment. > While none of this is actually in kernel, it could be in systemd (kernel-install), dracut, or grub. Do you see the kernel and similar files in your /boot directory? Yes, but from f37. > If so, which? vmlinuz-6.5.9-100.fc37.x86_64 Nothing from f39, although "rpm --verify kernel-core-6.5.10-300.fc39.x86_64" succeeds without output. > Do you have an initramfs? Yes: initramfs-6.5.9-100.fc37.x86_64.img
Created attachment 1998135 [details] trace of /usr/lib/kernel/install.d/51-dracut-rescue.install during kernel-core reinstall
I think I found the problem: there's a leftover machine id directory in /boot/efi. It has been created at initial install time (f27) and subsequent upgrades (odd release numbers) did not remove it but moved the "activity" to /boot. The current dracut script(s) check the presence of this directory to determine the /boot or /boot/efi context, causing a big confusion between the two locations... so this seems to be a dracut bug. I don't know how to fix the scripts, but I succeeded working around the problem: 1) boot using the latest installed kernel (in my case: kernel-6.5.10-100.fc37.x86_64) 2) Uninstall the failed kernel packages (kernel-6.5.11-300.fc39.x86_64 et al. for me). 3) cd /boot/efi and locate the machine id directory (i.e.: b4bf0711a.....9614). Rename it by appending .save to its name (for security). 4) dnf update. This will reinstall the f39 kernel, but without failure this time. 5) Reboot: you should now see the f39 kernel in the grub menu --> start with it. 6) Do what you want with the saved directory when you're sure its contents is not needed anymore. I hope it helps!
(In reply to Patrick Monnerat from comment #5) > I think I found the problem: there's a leftover machine id directory in > /boot/efi. It has been created at initial install time (f27) and subsequent > upgrades (odd release numbers) did not remove it but moved the "activity" to > /boot. The current dracut script(s) check the presence of this directory to > determine the /boot or /boot/efi context, causing a big confusion between > the two locations... so this seems to be a dracut bug. > The kernel-install from the systemd project was converted to a C program, and it changed how it decided to created sd_boot updates or grub2 updates. In current version the existence of /boot/efi/$machineid triggers sd_boot configuration, and in the previous Fedora version it did not.
> In current version the existence of /boot/efi/$machineid triggers sd_boot configuration, and in the previous Fedora version it did not. Thanks for the info. IMO there was some old release that used this directory with grub and it was never removed by any (dnf) system-upgrade since then (on an x86_64, i never used sd_boot). This old leftover directory is the culprit of the described problem and probably ought to be removed prior upgrading to f39 if grub is used.
so, since this seems to be a behaviour change in systemd, reassigning to systemd. also tagging for Common Bugs as we should probably flag this up...
Patrick, can you please run `sudo tree /boot` and attach the output here? Thanks.
I just installed F27 in a VM and I can confirm such $machineid was really created at that time: # ls /boot/520f44320ac640c68996d2e055036d31/ 4.13.9-300.fc27.x86_64 But what is really weird is that it *doesn't match* the actual machine id, nor vmlinuz etc filenames: # cat /etc/machine-id fb40b06b4d1a4a7ea0dcd2f608889a4d # ls /boot/vmlinuz-* vmlinuz-0-rescue-fb40b06b4d1a4a7ea0dcd2f608889a4d vmlinuz-4.13.9-300.fc27.x86_64 Patrick, does your /etc/machine-id value match the file that was present in /boot/efi/?
Created attachment 1999811 [details] Boot tree as requested in comment #9 > Patrick, can you please run `sudo tree /boot` and attach the output here? Thanks. Here it is! > Patrick, does your /etc/machine-id value match the file that was present in /boot/efi/? Yes it does. I removed the directory, but I've checked /etc/machine-id matches the rm command arg in /root/.bash_history !
Hm, this is getting interesting. So you have: /boot/e5dbe4fba10e41a7a55a2e23bba1adc3 /boot/efi/b4bf0711a11041b5896b297815999614 <- this one you removed (I assume the value from other loader entries, and also assume this is your /etc/machine-id) /boot/loader/entries/b4bf0711a11041b5896b297815999614-0-rescue.conf And the error message complains: /boot/efi/loader/entries/1604a63a532947d99d18bf17a754e833-0-rescue.conf: No such file or directory Wow, 3 completely different hashes! What on earth is going on here? (The /boot/$hash situation matches my experience from comment 10). Furthermore, while in your case it was enough to remove /boot/efi/$machineid, on non-EFI machines we also need to look whether different location (like /boot/$machineid) is not queried instead. That would make documenting the workaround even more complex.
Sorry, I think my different posts are a bit confusing: - The initial report and script trace came after the problem occurred on my notebook. 1604a63a532947d99d18bf17a754e833 was on this machine. I reinstalled the latter from scratch since then, so I can't give you more than what is in the script trace. Therefore I think you can forget about this hash. - When the problem occurred for my workstation too, I investigated more and the attached boot tree list comes from it. I did not reinstall it but fixed as in comment #5. Its /etc/machine_id is b4bf0711a11041b5896b297815999614. So: > /boot/e5dbe4fba10e41a7a55a2e23bba1adc3 Seems also to have been created by F27 and left over (dated Nov 5 2017). I really don't remember this 6 years old context! > /boot/efi/b4bf0711a11041b5896b297815999614 <- this one you removed (I assume the value from other loader entries, and also assume this is your /etc/machine-id) Right! > /boot/loader/entries/b4bf0711a11041b5896b297815999614-0-rescue.conf This one is the current and working rescue entry on my workstation. > /boot/efi/loader/entries/1604a63a532947d99d18bf17a754e833-0-rescue.conf: No such file or directory As noted above, you can ignore this hash.
> I just installed F27 in a VM and I can confirm such $machineid was really created at that time: > # ls /boot/520f44320ac640c68996d2e055036d31/ > 4.13.9-300.fc27.x86_64 > But what is really weird is that it *doesn't match* the actual machine id, nor vmlinuz etc filenames: Not real questions but hint tentatives: - Is your VM EFI-enabled? - What do you have in /boot/efi?
It is only the existence of the directory $(cat /etc/machine-id) in the ESP, and the /boot does not qualify as a valid ESP. Thus: if /boot/efi/$(cat /etc/machine-id) exists and is a directory you get BLS layout. You can test mkdir /boot/efi/$(cat /etc/machine-id) kernel-install | grep KERNEL_INSTALL_LAYOUT Then rmdir /boot/efi/$(cat /etc/machine-id) kernel-install | grep KERNEL_INSTALL_LAYOUT It is very possible that some people has experimented with "bootctl install" and thereby created that directory. That could happen some releases ago, as it caused no problems in Fedora 38, but now it does.
I don't recall the details, but we've definitely had some "fun" with this machine ID stuff before and it's changed a lot, systemd kept fiddling with it. I recall some issues with e.g. live images where we'd get files for some machine ID at the time the live image was created then there'd be a different one at install time and that would cause trouble...you can follow a trail from https://github.com/systemd/systemd/pull/22463 to https://github.com/systemd/systemd/issues/22376 to https://github.com/systemd/systemd/pull/21757 to see some of the fun, there may have been another one I'm forgetting too. So I suspect there may have been some different weird cases you could get into depending on when and from what image you installed...
(In reply to Patrick Monnerat from comment #14) > > I just installed F27 in a VM and I can confirm such $machineid was really created at that time: > > > # ls /boot/520f44320ac640c68996d2e055036d31/ > > 4.13.9-300.fc27.x86_64 > Hi, This is a minor point that may not add much, but I've had the same problem and used Patrick's fix to solve it. However in my /boot/efi/$machineid directory, I have kernel entries going back to Fedora 26 and nothing after Fedora 26 except for "6.20", whatever that is.: drwx------. 2 root root 4.0K Nov 11 14:03 0-rescue drwx------. 2 root root 4.0K Sep 19 2017 4.12.13-300.fc26.x86_64 drwx------. 2 root root 4.0K Oct 3 2017 4.12.14-300.fc26.x86_64 drwx------. 2 root root 4.0K Nov 6 2017 4.13.10-200.fc26.x86_64 drwx------. 2 root root 4.0K Nov 12 2017 4.13.11-200.fc26.x86_64 drwx------. 2 root root 4.0K Nov 19 2017 4.13.12-200.fc26.x86_64 drwx------. 2 root root 4.0K Nov 25 2017 4.13.13-200.fc26.x86_64 drwx------. 2 root root 4.0K Nov 29 2017 4.13.15-200.fc26.x86_64 drwx------. 2 root root 4.0K Dec 7 2017 4.13.16-202.fc26.x86_64 drwx------. 2 root root 4.0K Oct 10 2017 4.13.4-200.fc26.x86_64 drwx------. 2 root root 4.0K Oct 20 2017 4.13.5-200.fc26.x86_64 drwx------. 2 root root 4.0K Nov 2 2017 4.13.9-200.fc26.x86_64 drwx------. 2 root root 4.0K Jan 6 2018 4.14.11-200.fc26.x86_64 drwx------. 2 root root 4.0K Jan 27 2018 4.14.14-200.fc26.x86_64 drwx------. 2 root root 4.0K Feb 2 2018 4.14.16-200.fc26.x86_64 drwx------. 2 root root 4.0K Feb 19 2018 4.14.18-200.fc26.x86_64 drwx------. 2 root root 4.0K Dec 17 2017 4.14.5-200.fc26.x86_64 drwx------. 2 root root 4.0K Mar 29 2018 4.15.12-201.fc26.x86_64 drwx------. 2 root root 4.0K Apr 8 2018 4.15.14-200.fc26.x86_64 drwx------. 2 root root 4.0K Apr 20 2018 4.15.17-200.fc26.x86_64 drwx------. 2 root root 4.0K Feb 24 2018 4.15.4-200.fc26.x86_64 drwx------. 2 root root 4.0K Mar 4 2018 4.15.6-200.fc26.x86_64 drwx------. 2 root root 4.0K Nov 11 14:04 6.20 They are all empty however except for 0-rescue.
Just ran into this myself. Same problem as the OP. I removed the rescue directories. Note that I have these mounts: /dev/nvme0n1p2 on /boot type ext4 (rw,noatime,seclabel,discard) /dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro) When trying to install a current kernel: Updating / installing... 1:kernel-core-6.6.4-200.fc39 D: skip 100644 1 ( 0, 0) 160 .vmlinuz-6.6.4-200.fc39.x86_64.hmac D: skip 100600 1 ( 0, 0)8780540 System.map-6.6.4-200.fc39.x86_64 D: skip 100644 1 ( 0, 0)266334 config-6.6.4-200.fc39.x86_64 D: skip 100600 1 ( 0, 0)20971520 initramfs-6.6.4-200.fc39.x86_64.img D: skip 100600 1 ( 0, 0)157932 symvers-6.6.4-200.fc39.x86_64.xz D: skip 100755 1 ( 0, 0)14662152 vmlinuz-6.6.4-200.fc39.x86_64 Note that none of the "usual" files get installed into /boot; I can't figure out why they get skipped by RPM. Then later, for /usr/lib/kernel/install.d/90-loaderentry.install: + COMMAND=add + KERNEL_VERSION=6.6.4-200.fc39.x86_64 + ENTRY_DIR_ABS=/boot/efi/814dc1f7273b4888a3417122dfc4fd86/6.6.4-200.fc39.x86_64 + KERNEL_IMAGE=/lib/modules/6.6.4-200.fc39.x86_64/vmlinuz + INITRD_OPTIONS_SHIFT=4 + '[' bls = bls ']' + MACHINE_ID=814dc1f7273b4888a3417122dfc4fd86 + ENTRY_TOKEN=814dc1f7273b4888a3417122dfc4fd86 + BOOT_ROOT=/boot/efi + '[' -n '' ']' ++ stat -c %m /boot/efi + BOOT_MNT=/boot/efi + '[' /boot/efi = / ']' + ENTRY_DIR=/814dc1f7273b4888a3417122dfc4fd86/6.6.4-200.fc39.x86_64 + KERNEL_DEST=/boot/efi/814dc1f7273b4888a3417122dfc4fd86/6.6.4-200.fc39.x86_64/linux + KERNEL_ENTRY=/814dc1f7273b4888a3417122dfc4fd86/6.6.4-200.fc39.x86_64/linux + LOADER_ENTRY=/boot/efi/loader/entries/814dc1f7273b4888a3417122dfc4fd86-6.6.4-200.fc39.x86_64.conf But all my *old* (f37) kernels are in /boot/loader/entries. Maybe that has something to do with grub2-switch-to-blscfg that apparently disagrees with BOOT_MNT: blsdir=`echo "/boot/loader/entries" | sed 's,//*,/,g'` And 20-grub.install also has: BLS_DIR="/boot/loader/entries" It seems grub stuff doesn't agree with systemd-udev on where things are.
Deleting the /boot/efi/<machine-id> directory and running 'dnf reinstall kernel-core' made things work again. It's pretty clear that grub2 doesn't handle $BOOT being /boot/efi.
(In reply to Dan Williams from comment #19) > Deleting the /boot/efi/<machine-id> directory and running 'dnf reinstall > kernel-core' made things work again. > > It's pretty clear that grub2 doesn't handle $BOOT being /boot/efi. The new C language version of kernel-install first locates the ESP file system and if it has the <machine-id> directory in it, kernel-install will install the sd-boot configuration files.
For me, the directory keeps coming back. I don't know where to go from here. Did I miss a step? This system has been updated through the releases for a long, long time (15-20 years). I have similar mounts as comment 18, although that's probably a coincidence since this is a pretty common setup I would imagine: /dev/nvme0n1p2 on / type ext4 /dev/nvme0n1p1 on /boot/efi type vfat root@atlas:~# l /boot/efi/ total 16K drwxr-xr-x 4 root root 4.0K Dec 31 1969 . dr-xr-xr-x 8 root root 4.0K Dec 14 02:22 .. -rwxr-xr-x 1 root root 0 Apr 3 2022 .cka_this_is_dev-nvme0n1p1 drwxr-xr-x 4 root root 4.0K Dec 14 21:18 e8233d826a10831aff31b7d10000000e drwxr-xr-x 4 root root 4.0K Dec 14 14:01 EFI root@atlas:~# rm -Rf /boot/efi/e8233d826a10831aff31b7d10000000e root@atlas:~# l /boot/efi/ total 12K drwxr-xr-x 3 root root 4.0K Dec 31 1969 . dr-xr-xr-x 8 root root 4.0K Dec 14 02:22 .. -rwxr-xr-x 1 root root 0 Apr 3 2022 .cka_this_is_dev-nvme0n1p1 drwxr-xr-x 4 root root 4.0K Dec 14 14:01 EFI root@atlas:~# dnf reinstall kernel-core <SNIP> Running scriptlet: kernel-core-6.6.6-200.fc39.x86_64 2/2 /usr/lib/kernel/install.d/51-dracut-rescue.install: line 81: /boot/efi/loader/entries/e8233d826a10831aff31b7d10000000e-0-rescue.conf: No such file or directory /usr/lib/kernel/install.d/51-dracut-rescue.install failed with exit status 1. warning: %posttrans(kernel-core-6.6.6-200.fc39.x86_64) scriptlet failed, exit status 1 <SNIP> root@atlas:~# l /boot/efi/ total 16K drwxr-xr-x 4 root root 4.0K Dec 31 1969 . dr-xr-xr-x 8 root root 4.0K Dec 14 02:22 .. -rwxr-xr-x 1 root root 0 Apr 3 2022 .cka_this_is_dev-nvme0n1p1 drwxr-xr-x 4 root root 4.0K Dec 14 21:24 e8233d826a10831aff31b7d10000000e drwxr-xr-x 4 root root 4.0K Dec 14 14:01 EFI
For me the trick was realizing this is no longer a grub setup but an uefi setup. The kernel is there, just not in /boot root@it1notebook ~ # l /boot/efi/12a1f611b7024771b9b102a13c88175a/6.8.0-0.rc5.41.fc40.x86_64/ total 111M drwx------ 2 root root 4.0K Feb 22 19:06 . drwx------ 4 root root 4.0K Feb 22 18:21 .. -rwx------ 1 root root 96M Feb 22 19:06 initrd -rwx------ 1 root root 15M Feb 22 18:40 linux So I ran 'bootctl install && reboot' and then choose "linux" as the boot item at the bios/uefi settings and there was the new kernel booting. Worth mentioning is that secure boot doesn't work atm. I do think we are being thrown in the deep here. A heads-up with some instructions would be nice.
I managed to get secure boot working again with sbctl, https://github.com/Foxboron/sbctl for which a copr exists.
> CC: mcatanza > Flags: needinfo?(systemd-maint) I didn't see any actual question. I'll unset needinfo. --- Someone who has the issue: please paste the output from 'kernel-install -v $version $vmlinuz'. E.g. for kernel-core-6.6.3-200.fc39.x86_64: $ rpm -q --scripts kernel-core-6.6.3-200.fc39.x86_64|grep 'kernel-install add' /bin/kernel-install add 6.6.3-200.fc39.x86_64 /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz || exit $? $ /bin/kernel-install add -v 6.6.3-200.fc39.x86_64 /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz ...
Upgraded from f38 to f39 today and found that I was still running f38-kernel. my workaround: $ sudo ls -l /boot/efi/6e3be9d5c1ce4ac791a3f10d618767f9 total 20 drwx------. 2 root root 4096 8 mar 11.21 0-rescue drwx------. 2 root root 4096 16 aug 2018 4.17.14-102.fc27.x86_64 drwx------. 2 root root 4096 24 aug 2018 4.17.17-100.fc27.x86_64 drwx------. 2 root root 4096 7 sep 2018 4.17.19-100.fc27.x86_64 drwx------. 2 root root 4096 8 mar 11.20 6.7.7-200.fc39.x86_64 $ cat /etc/machine-id 6e3be9d5c1ce4ac791a3f10d618767f9 # dnf remove kernel-core-6.7.7-200.fc39.x86_64 # mv /boot/efi/6e3be9d5c1ce4ac791a3f10d618767f9 /boot/efi/6e3be9d5c1ce4ac791a3f10d618767f9.save # dnf update .. reboot # rm -r /boot/efi/6e3be9d5c1ce4ac791a3f10d618767f9.save/ # uname -a Linux uw006 6.7.7-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 1 16:53:59 UTC 2024 x86_64 GNU/Linux
I've been checking this part of kernel-install/kernel-install.c r = is_dir_at(c->rfd, entry_token_path, /* follow = */ false); if (r < 0 && r != -ENOENT) return log_error_errno(r, "Failed to check if '%s' is a directory: %m", entry_token_path); if (r > 0) { /* If the metadata in $BOOT_ROOT doesn't tell us anything, then check if the entry token * directory already exists. If so, let's assume it's the standard boot loader spec, too. */ c->layout = LAYOUT_BLS; log_debug("%s exists, using layout=%s.", entry_token_path, layout_to_string(c->layout)); return 0; } Here you check if the entry token (that is "/boot/efi/$(cat /etc/machine-id)") exists. I would think that the directory "/boot/efi/loader" should also exist before you conclude that c->layout = LAYOUT_BLS. If the ESP is not mounted at "/boot/efi" replace "/boot/efi" with the mount point of the ESP (or XBOOTLDR).
I also noticed this issue, only after several months because the affected systems are mainly used by other, non tech savvy, family members, and they won't notice the kernel version. While the amount of people affected is limited, the impact of the effect is major. No kernel security updates and running outdated kernel forevermore. I removed /boot/efi/<machine id> as a workaround, but indeed the grub config should be checked first before assuming systemd-boot is in use.