Bug 2071034

Summary:	kernel-install misdetects /boot/efi over /boot as install location on systems
Product:	[Fedora] Fedora	Reporter:	tstruk
Component:	systemd	Assignee:	systemd-maint
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	36	CC:	acaringi, adscvr, airlied, alain.vigne.14, alciregi, augenauf, bob.bogo, bskeggs, fedoraproject, filbranden, flepied, gbcox, hcamp, hdegoede, hpa, hugh, jan.public, jarodwilson, jforbes, jglisse, jonathan, josef, kernel-maint, kothandan.sathiyamoorthy, lgoncalv, linville, lnykryn, martineau, masami256, matthias.andree, mchehab, msekleta, ptalbert, ryncsn, ssahani, s, stanley.king, steved, suchandra.spam+fedora, swt, systemd-maint, unixi, yuwatana, zbyszek
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	systemd-250.6-1.fc36	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-27 01:10:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description tstruk 2022-04-01 16:12:46 UTC

1. Please describe the problem:

   Install fc36 kernel rpms by simply running sudo dnf update.
   The rpms install without error, but the files are missing:
$ cat /etc/fedora-release 
Fedora release 36 (Thirty Six)
$ sudo dnf update -y
...
Complete!
$ sudo rpm -qa | grep kernel
kernel-core-5.16.16-200.fc35.x86_64
kernel-modules-5.16.16-200.fc35.x86_64
kernel-5.16.16-200.fc35.x86_64
kernel-modules-extra-5.16.16-200.fc35.x86_64
kernel-core-5.17.1-300.fc36.x86_64
kernel-modules-5.17.1-300.fc36.x86_64
kernel-5.17.1-300.fc36.x86_64
kernel-modules-extra-5.17.1-300.fc36.x86_64
...
$ sudo rpm -ql kernel-core-5.17.1-300.fc36 | grep vmlinuz
/boot/.vmlinuz-5.17.1-300.fc36.x86_64.hmac
/boot/vmlinuz-5.17.1-300.fc36.x86_64
/lib/modules/5.17.1-300.fc36.x86_64/.vmlinuz.hmac
/lib/modules/5.17.1-300.fc36.x86_64/vmlinuz

$ ls -l /boot/vmlinuz-5.17.1-300.fc36.x86_64
ls: cannot access '/boot/vmlinuz-5.17.1-300.fc36.x86_64': No such file or directory
$ ls -l /boot/
total 101415
-rw-r--r--. 1 root root   240727 Mar 19 07:09 config-5.16.16-200.fc35.x86_64
drwx------. 6 root root    16384 Dec 31  1969 efi
-rw-r--r--. 1 root root   151452 Jan 27 03:54 elf-memtest86+-5.31
drwxr-xr-x. 2 root root     5120 Mar 26 09:57 extlinux
drwx------. 6 root root     1024 Apr  1 07:36 grub2
-rw-------. 1 root root 38893540 Nov 26  2014 initramfs-0-rescue-cfb58b8ae4b44565b430295578b2befe.img
-rw-------. 1 root root 37605352 Mar 23 10:28 initramfs-5.16.16-200.fc35.x86_64.img
-rw-r--r--. 1 root root   560566 Jul  2  2016 initrd-plymouth.img
drwxr-xr-x. 3 root root     1024 Jun 17  2018 loader
drwx------. 2 root root    12288 Nov 26  2014 lost+found
-rw-r--r--. 1 root root   149856 Jan 27 03:54 memtest86+-5.31
-rw-r--r--. 1 root root   103646 Feb 27  2019 rdsosreport.txt
lrwxrwxrwx. 1 root root       47 Mar 23 10:27 symvers-5.16.16-200.fc35.x86_64.gz -> /lib/modules/5.16.16-200.fc35.x86_64/symvers.gz
-rw-------. 1 root root  5958673 Mar 19 07:09 System.map-5.16.16-200.fc35.x86_64
-rwxr-xr-x. 1 root root  8753352 Mar 20  2019 vmlinuz-0-rescue-cfb58b8ae4b44565b430295578b2befe
-rwxr-xr-x. 1 root root 11374768 Mar 19 07:09 vmlinuz-5.16.16-200.fc35.x86_64

but modules do get installed properly:
$ ls -l /lib/modules/
...
drwxr-xr-x. 2 root root 4096 Mar 13 21:35 5.16.9-200.fc35.x86_64
drwxr-xr-x. 8 root root 4096 Mar 29 08:44 5.17.1-300.fc36.x86_64

2. What is the Version-Release number of the kernel:

   5.17

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

   Yes, for the 5.16 kernels

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

   Yes, as above

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

   n/a 

6. Are you running any modules that not shipped with directly Fedora's kernel?:
  
   no

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

   Here is relevant info from /var/log/dnf:

2022-03-29T08:41:45-0700 DEBUG --> Starting dependency resolution
2022-03-29T08:41:45-0700 DEBUG ---> Package kernel.x86_64 5.17.1-300.fc36 will be installed
2022-03-29T08:41:45-0700 DEBUG ---> Package kernel-core.x86_64 5.17.1-300.fc36 will be installed
2022-03-29T08:41:45-0700 DEBUG ---> Package kernel-devel.x86_64 5.17.1-300.fc36 will be installed
2022-03-29T08:41:45-0700 DEBUG ---> Package kernel-modules.x86_64 5.17.1-300.fc36 will be installed
2022-03-29T08:41:45-0700 DEBUG ---> Package kernel-modules-extra.x86_64 5.17.1-300.fc36 will be installed
...
Installing:
 ESC[1mESC[32mkernel                                     ESC(BESC[m x86_64                        5.17.1-300.fc36                                 updates-testing                        161 k
 ESC[1mESC[32mkernel-core                                ESC(BESC[m x86_64                        5.17.1-300.fc36                                 updates-testing                         46 M
 ESC[1mESC[32mkernel-devel                               ESC(BESC[m x86_64                        5.17.1-300.fc36                                 updates-testing                         15 M
 ESC[1mESC[32mkernel-modules                             ESC(BESC[m x86_64                        5.17.1-300.fc36                                 updates-testing                         53 M
 ESC[1mESC[32mkernel-modules-extra                       ESC(BESC[m x86_64                        5.17.1-300.fc36                                 updates-testing                        3.4 M

2022-03-29T08:44:38-0700 DEBUG Installed: kernel-5.17.1-300.fc36.x86_64
2022-03-29T08:44:38-0700 DEBUG Installed: kernel-core-5.17.1-300.fc36.x86_64
2022-03-29T08:44:38-0700 DEBUG Installed: kernel-devel-5.17.1-300.fc36.x86_64
2022-03-29T08:44:38-0700 DEBUG Installed: kernel-modules-5.17.1-300.fc36.x86_64
2022-03-29T08:44:38-0700 DEBUG Installed: kernel-modules-extra-5.17.1-300.fc36.x86_64

Comment 1 Justin M. Forbes 2022-04-01 19:12:58 UTC

This has nothing to do with kernel unfortunately, it seems to be systemd with kernel-install, which is placing kernels in /boot/efi/<machine-id>.  The same thing hit me on upgrade to F36 on Tuesday.  A quick fix seems to be an rm -rf of that directory in /boot/efi and then reinstall the kernel or call kernel-install by hand.

For the systemd people, when it happened to me, I first tried to create an /etc/kernel/install.conf which explicitly passed KERNEL_INSTALL_BOOT_ROOT=/boot but running kernel install with -x showed that it was still getting changed to /boot/efi.

Comment 2 tstruk 2022-04-01 19:53:37 UTC

(In reply to Justin M. Forbes from comment #1)
> This has nothing to do with kernel unfortunately,

true, I didn't know exactly what to log it against.

> it seems to be systemd
> with kernel-install, which is placing kernels in /boot/efi/<machine-id>. 

I thought that it might be the case and checked /boot/efi/, but
it doesn't install it there.
# find /boot/efi/ -name "vmlinuz*" finds nothing.

Comment 3 Justin M. Forbes 2022-04-01 21:21:49 UTC

Oh, hmm. Are you doing that as root? for everyone else this has hit, the kernel is installed in /boot/efi/<machine-id> and the entries do not show up in grub on reboot. Perhaps this is a different but, either way, systemd kernel-install is what puts things in /boot.

Comment 4 tstruk 2022-04-01 22:59:36 UTC

(In reply to Justin M. Forbes from comment #3)
> Oh, hmm. Are you doing that as root? for everyone else this has hit, the
> kernel is installed in /boot/efi/<machine-id> and the entries do not show up
> in grub on reboot. Perhaps this is a different but, either way, systemd
> kernel-install is what puts things in /boot.

What I did was just `sudo dnf update`
and it pulled and installed the new kernel. I was surprised that it didn't boot into 5.17 after reboot so I started digging around.
As I said, there is nothing new installed in my /boot/efi/

Comment 5 James Boyle 2022-05-05 16:03:21 UTC

I think this may be related to an issue I have stumbled across when upgrading from F34 to F36 Beta.  

DNF reports that the kernel has been updated, but if you go back to the logs, you will find that there wasn't enough space for dracut to create the relevant boot images, etc.  So, rpm -qa reports that kernel-core is successfully installed, but the relevant files are not there.  Even as I'm writing this, I'm running a Fedora 34 kernel, though I went through the 'dnf system-upgrade' process.  




$ uname -a
Linux diziet.reple.at 5.17.4-100.fc34.x86_64 #1 SMP PREEMPT Wed Apr 20 14:41:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release 
Fedora release 36 (Thirty Six)

### excerpt from /var/log/dnf.rpm.log:

2022-05-02T14:57:09-0400 INFO ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
/usr/lib/kernel/install.d/51-dracut-rescue.install: line 76: /boot/efi/loader/entries/3ce13bc91cea438ba1644997b3fa4412-0-rescue.conf: No such file or directory
kdump: kernel 5.17.3-302.fc36.x86_64 doesn't exist
warning: %posttrans(kernel-core-5.17.3-302.fc36.x86_64) scriptlet failed, exit status 1

2022-05-02T14:57:09-0400 ERROR Error in POSTTRANS scriptlet in rpm package kernel-core
2022-05-02T14:57:33-0400 INFO yes: standard output: Broken pipe

### tail end of most recent dnf -y upgrade output:

Upgraded:                                                                                                                                                                                                   
  libipa_hbac-2.7.0-1.fc36.x86_64                    libsss_autofs-2.7.0-1.fc36.x86_64                  libsss_certmap-2.7.0-1.fc36.x86_64                  libsss_idmap-2.7.0-1.fc36.x86_64                
  libsss_nss_idmap-2.7.0-1.fc36.x86_64               libsss_sudo-2.7.0-1.fc36.x86_64                    lpf-spotify-client-1.1.84.716-1.fc36.x86_64         python3-sssdconfig-2.7.0-1.fc36.noarch          
  rpmfusion-free-release-36-1.noarch                 rpmfusion-free-release-tainted-36-1.noarch         rpmfusion-nonfree-release-36-1.noarch               selinux-policy-36.8-1.fc36.noarch               
  selinux-policy-targeted-36.8-1.fc36.noarch         sssd-2.7.0-1.fc36.x86_64                           sssd-ad-2.7.0-1.fc36.x86_64                         sssd-client-2.7.0-1.fc36.x86_64                 
  sssd-common-2.7.0-1.fc36.x86_64                    sssd-common-pac-2.7.0-1.fc36.x86_64                sssd-ipa-2.7.0-1.fc36.x86_64                        sssd-krb5-2.7.0-1.fc36.x86_64                   
  sssd-krb5-common-2.7.0-1.fc36.x86_64               sssd-ldap-2.7.0-1.fc36.x86_64                      sssd-nfs-idmap-2.7.0-1.fc36.x86_64                  sssd-proxy-2.7.0-1.fc36.x86_64                  
  xdg-desktop-portal-1.12.4-1.fc36.x86_64            xz-5.2.5-9.fc36.x86_64                             xz-devel-5.2.5-9.fc36.x86_64                        xz-libs-5.2.5-9.fc36.i686                       
  xz-libs-5.2.5-9.fc36.x86_64                                                                                                                                                                               
Installed:                                                                                                                                                                                                  
  kernel-5.17.5-300.fc36.x86_64     kernel-core-5.17.5-300.fc36.x86_64     kernel-devel-5.17.5-300.fc36.x86_64     kernel-modules-5.17.5-300.fc36.x86_64     kernel-modules-extra-5.17.5-300.fc36.x86_64    
  libjose-11-5.fc36.x86_64          sssd-idp-2.7.0-1.fc36.x86_64                                                                                                                                            
Removed:                                                                                                                                                                                                    
  kernel-5.16.19-100.fc34.x86_64    kernel-core-5.16.19-100.fc34.x86_64    kernel-devel-5.16.19-100.fc34.x86_64    kernel-modules-5.16.19-100.fc34.x86_64    kernel-modules-extra-5.16.19-100.fc34.x86_64   
                                                                                                                                                                                                            
Complete!     

# rpm -qa |grep kernel-core |sort
kernel-core-5.16.20-100.fc34.x86_64
kernel-core-5.17.3-302.fc36.x86_64
kernel-core-5.17.4-100.fc34.x86_64
kernel-core-5.17.5-300.fc36.x86_64
# ls /boot
config-5.17.4-100.fc34.x86_64  grub2                                                    lost+found                         vmlinuz-0-rescue-3ce13bc91cea438ba1644997b3fa4412
efi                            initramfs-0-rescue-3ce13bc91cea438ba1644997b3fa4412.img  memtest86+-5.31                    vmlinuz-5.17.4-100.fc34.x86_64
elf-memtest86+-5.31            initramfs-5.17.4-100.fc34.x86_64.img                     symvers-5.17.4-100.fc34.x86_64.gz
extlinux                       loader                                                   System.map-5.17.4-100.fc34.x86_64
# 

### Even after deleting as much as possible (only keeping files related to the currently running kernel),
###  removing and reinstalling the new kernel still fails to actually install the new fc36 kernel,
###  but DNF happily reports that it is installed and fine

# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               4.0M     0  4.0M   0% /dev
tmpfs                   16G   12K   16G   1% /dev/shm
tmpfs                  6.3G  2.2M  6.3G   1% /run
/dev/dm-3              489G  446G   18G  97% /
tmpfs                   16G   48K   16G   1% /tmp
/dev/sda2              339M   68M  255M  21% /boot
/dev/sda1              250M  6.2M  244M   3% /boot/efi
/dev/mapper/b510-data  664G  609G   22G  97% /data
/dev/mapper/b510-vms   251G  114G  125G  48% /vms
tmpfs                  3.2G  168K  3.2G   1% /run/user/1000
# rpm -qa |grep kernel |grep '5\.17\..-...\.fc36'
kernel-modules-5.17.3-302.fc36.x86_64
kernel-devel-5.17.3-302.fc36.x86_64
kernel-5.17.3-302.fc36.x86_64
kernel-modules-extra-5.17.3-302.fc36.x86_64
kernel-tools-libs-5.17.0-300.fc36.x86_64
kernel-headers-5.17.0-300.fc36.x86_64
kernel-core-5.17.3-302.fc36.x86_64
kernel-core-5.17.5-300.fc36.x86_64
kernel-modules-5.17.5-300.fc36.x86_64
kernel-5.17.5-300.fc36.x86_64
kernel-modules-extra-5.17.5-300.fc36.x86_64
kernel-devel-5.17.5-300.fc36.x86_64
# ls /boot
config-5.17.4-100.fc34.x86_64  extlinux                              loader           symvers-5.17.4-100.fc34.x86_64.gz                  vmlinuz-5.17.4-100.fc34.x86_64
efi                            grub2                                 lost+found       System.map-5.17.4-100.fc34.x86_64
elf-memtest86+-5.31            initramfs-5.17.4-100.fc34.x86_64.img  memtest86+-5.31  vmlinuz-0-rescue-3ce13bc91cea438ba1644997b3fa4412
# rpm -qa |grep kernel |grep '5\.17\..-...\.fc36' |grep -v header |grep -v tools |xargs -d '\n' rpm -ev
Preparing packages...
kernel-modules-extra-5.17.5-300.fc36.x86_64
kernel-5.17.5-300.fc36.x86_64
kernel-modules-extra-5.17.3-302.fc36.x86_64
kernel-5.17.3-302.fc36.x86_64
kernel-modules-5.17.3-302.fc36.x86_64
kernel-modules-5.17.5-300.fc36.x86_64
kernel-core-5.17.5-300.fc36.x86_64
kernel-core-5.17.3-302.fc36.x86_64
kernel-devel-5.17.5-300.fc36.x86_64
kernel-devel-5.17.3-302.fc36.x86_64
# rpm -qa |grep 'kernel-core.*fc36'
#

============================================================================================================================================================================================================
 Package                                                  Architecture                               Version                                               Repository                                  Size
============================================================================================================================================================================================================
Installing:
 kernel                                                   x86_64                                     5.17.5-300.fc36                                       fedora                                     163 k
 kernel-core                                              x86_64                                     5.17.5-300.fc36                                       fedora                                      46 M
 kernel-devel                                             x86_64                                     5.17.5-300.fc36                                       fedora                                      15 M
 kernel-modules                                           x86_64                                     5.17.5-300.fc36                                       fedora                                      53 M
 kernel-modules-extra                                     x86_64                                     5.17.5-300.fc36                                       fedora                                     3.4 M

Transaction Summary
============================================================================================================================================================================================================
Install  5 Packages

Total download size: 118 M
Installed size: 207 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): kernel-5.17.5-300.fc36.x86_64.rpm                                                                                                                                    141 kB/s | 163 kB     00:01    
(2/5): kernel-devel-5.17.5-300.fc36.x86_64.rpm                                                                                                                              1.7 MB/s |  15 MB     00:08    
(3/5): kernel-core-5.17.5-300.fc36.x86_64.rpm                                                                                                                               3.0 MB/s |  46 MB     00:15    
(4/5): kernel-modules-5.17.5-300.fc36.x86_64.rpm                                                                                                                            3.4 MB/s |  53 MB     00:15    
(5/5): kernel-modules-extra-5.17.5-300.fc36.x86_64.rpm                                                                                                                      441 kB/s | 3.4 MB     00:07    
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                       6.9 MB/s | 118 MB     00:17     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                    1/1 
  Installing       : kernel-core-5.17.5-300.fc36.x86_64                                                                                                                                                 1/5 
  Running scriptlet: kernel-core-5.17.5-300.fc36.x86_64                                                                                                                                                 1/5 
  Installing       : kernel-modules-5.17.5-300.fc36.x86_64                                                                                                                                              2/5 
  Running scriptlet: kernel-modules-5.17.5-300.fc36.x86_64                                                                                                                                              2/5 
  Installing       : kernel-5.17.5-300.fc36.x86_64                                                                                                                                                      3/5 
  Installing       : kernel-modules-extra-5.17.5-300.fc36.x86_64                                                                                                                                        4/5 
  Running scriptlet: kernel-modules-extra-5.17.5-300.fc36.x86_64                                                                                                                                        4/5 
  Installing       : kernel-devel-5.17.5-300.fc36.x86_64                                                                                                                                                5/5 
  Running scriptlet: kernel-devel-5.17.5-300.fc36.x86_64                                                                                                                                                5/5 
  Running scriptlet: kernel-core-5.17.5-300.fc36.x86_64                                                                                                                                                 5/5 
kdump: kernel 5.17.5-300.fc36.x86_64 doesn't exist

  Running scriptlet: kernel-modules-5.17.5-300.fc36.x86_64                                                                                                                                              5/5 
  Running scriptlet: kernel-devel-5.17.5-300.fc36.x86_64                                                                                                                                                5/5 
  Verifying        : kernel-5.17.5-300.fc36.x86_64                                                                                                                                                      1/5 
  Verifying        : kernel-core-5.17.5-300.fc36.x86_64                                                                                                                                                 2/5 
  Verifying        : kernel-devel-5.17.5-300.fc36.x86_64                                                                                                                                                3/5 
  Verifying        : kernel-modules-5.17.5-300.fc36.x86_64                                                                                                                                              4/5 
  Verifying        : kernel-modules-extra-5.17.5-300.fc36.x86_64                                                                                                                                        5/5 
Installed products updated.

Installed:
  kernel-5.17.5-300.fc36.x86_64     kernel-core-5.17.5-300.fc36.x86_64     kernel-devel-5.17.5-300.fc36.x86_64     kernel-modules-5.17.5-300.fc36.x86_64     kernel-modules-extra-5.17.5-300.fc36.x86_64    

Complete!
# rpm -qa |grep 5.17.5-300
kernel-core-5.17.5-300.fc36.x86_64
kernel-modules-5.17.5-300.fc36.x86_64
kernel-5.17.5-300.fc36.x86_64
kernel-modules-extra-5.17.5-300.fc36.x86_64
kernel-devel-5.17.5-300.fc36.x86_64
# ls /boot
config-5.17.4-100.fc34.x86_64  extlinux                              loader           symvers-5.17.4-100.fc34.x86_64.gz                  vmlinuz-5.17.4-100.fc34.x86_64
efi                            grub2                                 lost+found       System.map-5.17.4-100.fc34.x86_64
elf-memtest86+-5.31            initramfs-5.17.4-100.fc34.x86_64.img  memtest86+-5.31  vmlinuz-0-rescue-3ce13bc91cea438ba1644997b3fa4412
# ls /boot/efi/3ce13bc91cea438ba1644997b3fa4412/5.17.5-300.fc36.x86_64/
initrd  linux
# ls -al /boot/efi/3ce13bc91cea438ba1644997b3fa4412/5.17.5-300.fc36.x86_64/
total 50648
drwx------.  2 root root     4096 May  5 11:49 .
drwx------. 14 root root     4096 May  5 11:48 ..
-rwx------.  1 root root 40048796 May  5 11:48 initrd
-rwx------.  1 root root 11802352 May  5 11:49 linux
[root@diziet boot]# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               4.0M     0  4.0M   0% /dev
tmpfs                   16G   12K   16G   1% /dev/shm
tmpfs                  6.3G  2.3M  6.3G   1% /run
/dev/dm-3              489G  446G   18G  97% /
tmpfs                   16G   48K   16G   1% /tmp
/dev/sda2              339M   68M  255M  21% /boot
/dev/sda1              250M  173M   78M  69% /boot/efi
/dev/mapper/b510-data  664G  609G   22G  97% /data
/dev/mapper/b510-vms   251G  114G  125G  48% /vms
tmpfs                  3.2G  168K  3.2G   1% /run/user/1000

### /boot/efi's usage went from about 6MB to 173MB, but clearly things went wrong.

### I'll also note that after the upgrade completed, there was no indication that anything went wrong at all; the system booted and everything seemed like it was working.  Only when I went to check what kernel version was running did I notice that the new fc36 kernel was not running and not properly installed.  My next step was going to be to back up all my files to another disk and repartition the system disk with very large /boot and /boot/efi partitions.  I'm happy to try other suggestions.  

I think there are two issues:
 [a] DNF failed to recognize that the install didn't complete successfully and didn't remove the remnants of the failed installation
 [b] the dnf system-upgrade plugin failed to warn me that I have insufficient space before attempting the upgrade, if that is the case

Comment 6 Justin M. Forbes 2022-05-05 17:04:41 UTC

Well, the issue occurs, seemingly only on systems that were installed quite some time ago originally. The main problem is kernels aren't supposed to be installed in /boot/efi, something about the F36 systemd kernel-install is now putting them there, but only on a small percentage of systems. If you delete the /boot/efi/<machine-id> directory and reinstall the kernel, it will end up in /boot/ like older ones did, but if everyone does that, there will not be a system to debug the issue with.  Of the 6 systems I have upgraded here, only 1 displayed the issue, and I know adamwill had it on exactly 1 system as well.

Comment 7 Zbigniew Jędrzejewski-Szmek 2022-05-05 17:28:03 UTC

*** Bug 2073734 has been marked as a duplicate of this bug. ***

Comment 8 Harold Campbell 2022-05-05 17:54:44 UTC

(In reply to tstruk from comment #2)
> (In reply to Justin M. Forbes from comment #1)
> > This has nothing to do with kernel unfortunately,
> 
> true, I didn't know exactly what to log it against.
> 
> > it seems to be systemd
> > with kernel-install, which is placing kernels in /boot/efi/<machine-id>. 
> 
> I thought that it might be the case and checked /boot/efi/, but
> it doesn't install it there.
> # find /boot/efi/ -name "vmlinuz*" finds nothing.

It shows up as linux, not vmlinuz.

[root@kibble efi]# find /boot/efi/ -name "linux*"
/boot/efi/<blah>/5.17.3-302.fc36.x86_64/linux
/boot/efi/<blah>/0-rescue/linux
/boot/efi/<blah>/5.17.5-300.fc36.x86_64/linux
[root@kibble efi]# file /boot/efi/<blah>/5.17.5-300.fc36.x86_64/linux
/boot/efi/<blah>/5.17.5-300.fc36.x86_64/linux: Linux kernel x86 boot executable bzImage, version 5.17.5-300.fc36.x86_64 (mockbuild.fedoraproject.org) #1 SMP PREEMPT Thu Apr 28 15:51:30 UTC 2022, RO-rootFS, swap_dev 0XB, Normal VGA

Comment 9 James Boyle 2022-05-05 19:07:42 UTC

(In reply to Justin M. Forbes from comment #6)
> Well, the issue occurs, seemingly only on systems that were installed quite
> some time ago originally. The main problem is kernels aren't supposed to be
> installed in /boot/efi, something about the F36 systemd kernel-install is
> now putting them there, but only on a small percentage of systems. If you
> delete the /boot/efi/<machine-id> directory and reinstall the kernel, it
> will end up in /boot/ like older ones did, but if everyone does that, there
> will not be a system to debug the issue with.  Of the 6 systems I have
> upgraded here, only 1 displayed the issue, and I know adamwill had it on
> exactly 1 system as well.

I've got this issue on 2 of 2 systems I upgraded from F34 --> F36.  I started the second upgrade after the first appeared to have been successful, but before I noticed that I was still running the F34 kernel.  You're right in that both system were originally installed quite a while ago, maybe around F27, but I couldn't say for sure.

Let me know if there's information I can gather that would help; otherwise I'll try deleting the /boot/efi/<machine-id> directory and reinstalling so that I don't get too far behind in kernel versions.

Thanks!

Comment 10 James Boyle 2022-05-11 14:52:20 UTC

I removed /boot/efi/<machine-id> on one of the machines; another still has the issue.  If there isn't any reason to leave the un-upgraded machine in it's current state, or no requests for information from that machine, I'll perform the workaround on Monday and move on.

Thank you,
--James

Comment 11 Justin M. Forbes 2022-05-16 16:37:37 UTC

*** Bug 2084362 has been marked as a duplicate of this bug. ***

Comment 12 Stan King 2022-05-16 23:43:24 UTC

If anyone is still debugging this, you may find my comments on bug 2084362 helpful.  It was declared a duplicate about an hour after I left my observations there.

Comment 13 Flo 2022-05-19 06:59:54 UTC

fyi, I also encountered this bug when upgrading from F35 to F36. This is a dual-boot machine, which was initially set up quite a while ago. I have to look up the date (not near that machine right now) - I'd say it was around F25.

Reason for posting here, I am happy to provide debugging.

Comment 14 Alain V. 2022-05-19 07:24:12 UTC

+1
Similar experience than Flo. rm-rf /boot/efi/xxxxxx then, reinstall the kernel is a working work-around for me.
Any idea if this work-around should be reproduced for next kernel update ?

Comment 15 Zbigniew Jędrzejewski-Szmek 2022-05-19 13:36:02 UTC

OK, I think I understand. Those old machines have the pre-BLS grub2 layout.
kernel-install looks for /efi/<machine-id>/, /boot/<machine-id>/, /boot/efi/<machine-id>/,
and in the affected machines, only the last directory exists. This causes the installation
procedure to use /boot/efi/<machine-id>/<kernel-version>/ as the destination.

I'll need to figure out what changed, or rather why it was working before.

Comment 16 Zbigniew Jędrzejewski-Szmek 2022-05-19 21:18:18 UTC

https://github.com/systemd/systemd/pull/23439 should fix the issue.
The last commit is important, the earlier ones are preparation.

Comment 17 Zbigniew Jędrzejewski-Szmek 2022-05-25 07:15:57 UTC

*** Bug 2085288 has been marked as a duplicate of this bug. ***

Comment 18 Gerald Cox 2022-05-25 15:40:38 UTC

I upgraded to F36 weeks ago and today (May 25th) when I did an dnf offline-upgrade and then rebooted, out of the blue this
problem occurred.

Comment 19 Gerald Cox 2022-05-25 16:07:13 UTC

Changed the title and removed reference to F34 because I had upgraded from F35 weeks ago and then just experienced the issue today.

Comment 20 Fedora Update System 2022-05-25 20:10:59 UTC

FEDORA-2022-3ca356cd2e has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3ca356cd2e

Comment 21 Fedora Update System 2022-05-26 02:32:50 UTC

FEDORA-2022-3ca356cd2e has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3ca356cd2e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3ca356cd2e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 22 Harold Campbell 2022-05-26 17:55:01 UTC

1. Install systemd-250.6-1.fc36.x86_64
2. Reinstall kernel-core
3. Profit!

The new systemd has resolved the problem for me.

Comment 23 Fedora Update System 2022-05-27 01:10:05 UTC

FEDORA-2022-3ca356cd2e has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 24 D. Hugh Redelmeier 2022-06-01 14:15:20 UTC

I had this problem when I did a DNF upgrade F35->F36.

No diagnostics appeared on the console.  This is terrible!

After the update, new kernels were on the system but grub could not see them or boot them.

I could not fix it by re-installing kernels.

I finally figured out that the ESP was full and that this mattered.  Not by diagnostic!

I fixed the problem by doing a clean install fo F36 from the .iso.

After that clean install, /boot/efi contains a rescue kernel and initrd and a normal kernel (without initrd).  Updating adds new kernels but only one (the latest??) is in /boot/esp.

This is horrible:

- no diagnostic.  Not during the upgrade process, not during subsequent kernel installations.

- silent change to requirement for space on ESP

- silent change to how booting works

When is the /boot/efi copy of the kernel used?  Why only one and not all the installed kernels?

(I can see an advantage of the kernel being in the ESP: it means that grub might not need to understand the filesystem or even device of /boot.  Or grub could be eliminated.  This seems really good!)