Bug 2241948 - kernel 6.5.5 / grub attempts to install big blobs under /boot/efi, leading to ENOSPC
Summary: kernel 6.5.5 / grub attempts to install big blobs under /boot/efi, leading to...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 39
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2248624 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-10-03 14:57 UTC by Frank Ch. Eigler
Modified: 2024-11-27 21:33 UTC (History)
29 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-11-27 21:33:28 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
kernel-install transcript (8.01 KB, text/plain)
2024-01-18 17:38 UTC, Frank Ch. Eigler
no flags Details

Description Frank Ch. Eigler 2023-10-03 14:57:25 UTC
In a fresh F39 update on a machine with long fedora/efi history, the 6.5.5 kernel has started failing to install.  The "yum install" works, actually, but the "kernel-core" subrpm has all sorts of missing files under "/boot".

What appears to be happening is new.  A bunch of initrd content is being written onto /boot/efi/HEXCODE/N-V-R, which overflows this small partition.  Somehow this is hitting the installation of the main vmlinuz etc. files onto /boot, and yet the subrpm shows a successful installation (!?!?!!).  grub with or without bls doesn't find the bootable image, and therefore the new kernel is inaccessible.

This is a machine with nvidia/dkms blobs present, but I have no reason to suspect that a cause.

Reproducible: Always

Comment 1 Justin M. Forbes 2023-10-03 15:28:05 UTC
This is actually a function of kernel-install which is systemd-udev. I am reassigning it.  For history, this same type of issue also showed up in the F38 cycle, and it seems everyone who hit it had installed several releases ago, and done dnf system-upgrade since.  I know it hit on one of my machines, but not on the rest. I do not recall what the resolution was.

Comment 2 Zbigniew Jędrzejewski-Szmek 2024-01-09 15:04:14 UTC
Sorry for the slow reply.

It's a bit hard to say what exactly is going on without knowing which systemd,
grubby, and grub2 versions are installed. There have been a bunch of fixes
over time.

Please show the output of
  /bin/kernel-install -v add 6.5.5-300.fc39.x86_64 /lib/modules/6.5.5-300.fc39.x86_64/vmlinuz

(Or the similar command for newer rpms. It can be found by
  rpm -q --scripts kernel-core-<version> | grep kernel-install
'-v' should be added as the first argument to enable verbose output.)

Comment 3 Vitaly Kuznetsov 2024-01-11 16:31:53 UTC
Do you by any chance have 'kernel-uki-virt' package installed which you don't really need? 'kernel-uki-virt' ships Fedora UKI and these are getting installed to /boot/efi/EFI/Linux by kernel-install. The behavior is expected: UKIs are normally booted directly from the firmware or by shim and both methods can't load them from anywhere but ESP.

Comment 4 Vitaly Kuznetsov 2024-01-11 16:33:14 UTC
Also, there was a bug that grub was installing UKIs to /boot and this should had been fixed with https://src.fedoraproject.org/rpms/grub2/pull-request/28

Comment 5 Frank Ch. Eigler 2024-01-18 17:37:05 UTC
The problem here seems to be the exhaustion of /boot/efi, not /boot.  No sign of kernel-uki-virt.

% du /boot/efi
132	/boot/efi/EFI/fedora/System/Library/CoreServices
132	/boot/efi/EFI/fedora/System/Library
133	/boot/efi/EFI/fedora/System
6289	/boot/efi/EFI/fedora
1313	/boot/efi/EFI/BOOT/fonts
4560	/boot/efi/EFI/BOOT
132	/boot/efi/EFI/grub/System/Library/CoreServices
132	/boot/efi/EFI/grub/System/Library
133	/boot/efi/EFI/grub/System
133	/boot/efi/EFI/grub
10982	/boot/efi/EFI
2241	/boot/efi/System/Library/CoreServices
2241	/boot/efi/System/Library
2242	/boot/efi/System
36609	/boot/efi/9e3d4b6532ff41e1ab40d6b4e5609907/6.6.11-200.fc39.x86_64
14331	/boot/efi/9e3d4b6532ff41e1ab40d6b4e5609907/0-rescue
1	/boot/efi/9e3d4b6532ff41e1ab40d6b4e5609907/6.4.15-200.fc38.x86_64
50942	/boot/efi/9e3d4b6532ff41e1ab40d6b4e5609907
64193	/boot/efi

Comment 6 Frank Ch. Eigler 2024-01-18 17:38:13 UTC
Created attachment 2009157 [details]
kernel-install transcript

Comment 7 Frank Ch. Eigler 2024-02-14 15:54:51 UTC
Found one non-workaround: enlarging /boot/efi so it should all fit.  Despite
that, kernel-install would not create a functional BLS installation for the
new kernels, despite manual and/or dnf-reinstall runs.

Found one workaround: nuking any trace of BLS support in /boot/efi (by
renaming the /boot/efi/$machinehex and /boot/efi/loader directories), and
using non-BLS booting via grub.  Then the vmlinuz/initrd popped up nicely
in /boot.

Comment 8 Chris Murphy 2024-02-14 22:48:48 UTC
tl;dr I think all you need to do is rm -rf /boot/efi/$machinehex

The Boot Loader Specification calls for kernels and initrds going on the EFI system partition mounted at either /boot /efi or /boot/efi, in a directory matching the machineID. And that's what the attached log looks like is happening to me:
Plugin arguments: add 6.4.15-200.fc38.x86_64 /boot/efi/9e3d4b6532ff41e1ab40d6b4e5609907/6.4.15-200.fc38.x86_64 /lib/modules/6.4.15-200.fc38.x86_64/vmlinuz

Fedora never supported this location by default, but I think it would (and apparently still will) install kernels and initrds if this path exists. It's probably true we need to keep supporting this location since systemd-boot uses it.

So what's the bug? Is it the script that tests for a machineID directory on the ESP, and if found installs kernels and initrds there? Probably not. OK so how did the directory with machineID get there? That's probably the bug and more elusive. I've run into this myself and never figured out what created it, it was just too transient (and I was lazy).

Comment 9 David Tardon 2024-08-07 13:39:14 UTC
*** Bug 2248624 has been marked as a duplicate of this bug. ***

Comment 10 Aoife Moloney 2024-11-27 21:33:28 UTC
Fedora Linux 39 entered end-of-life (EOL) status on 2024-11-26.

Fedora Linux 39 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.