1. Please describe the problem: I updated a Fedora 39 KDE Plasma installation running the 6.5.7-300.fc39.x86_64 using a dnf offline-upgrade with updates-testing enabled. The upgrade contained kernel-6.5.8-300.fc39. The offline upgrade completed normally. On the first boot of 6.5.8, the Plymouth screen didn't appear. All the systemd and kernel messages were shown as when rhgb quiet was removed from the kernel command line. 6.5.8 kernel booted to a dracut emergency shell with the following errors shown. Failed to start initrd-switch-root.service - Switch Root. See 'systemctl status initrd-switch-root.service' for details Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit the shell to continue. The journal showed the further errors with the switch root process. [ 14.221886] localhost.localdomain systemd[1]: Reached target initrd-switch-root.target - Switch Root. [ 14.232684] localhost.localdomain systemd[1]: Starting initrd-switch-root.service - Switch Root... [ 14.244150] localhost.localdomain @ystemctl[532]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing. [ 14.245337] localhost.localdomain systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE [ 14.245580] localhost.localdomain systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'. [ 14.256682] localhost.localdomain systemd[1]: Failed to start initrd-switch-root.service - Switch Root. [ 14.257352] localhost.localdomain systemd[1]: Startup finished in 4.840s (firmware) + 31.314s (loader) + 2.267s (kernel) + 0 (initrd) + 11.989s (userspace) = 50.412s. [ 14.257573] localhost.localdomain systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies. [ 14.271724] localhost.localdomain systemd[1]: Started emergency.service - Emergency Shell. [ 14.272278] localhost.localdomain systemd[1]: Reached target emergency.target - Emergency Mode. The journal showed that the kernel command line only had its first parameter remaining. [ 0.000000] localhost.localdomain kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.5.8-300.fc39.x86_64 My normal kernel command line parameters which worked with 6.5.7 and earlier were BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.5.8-300.fc39.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rhgb quiet rdrand=force When I added the rest of my normal kernel command line parameters to 6.5.8 in grub, the system booted normally. Therefore, 6.5.8 booted to a dracut emergency shell because all but the first of its kernel command line parameters were removed by something likely in the update process. 2. What is the Version-Release number of the kernel: kernel-6.5.8-300.fc39 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Yes kernel-6.5.7-300.fc39.x86_64 worked normally. The problem appeared with kernel-6.5.8-300.fc39 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: 1. boot a Fedora 39 KDE Plasma installation running the 6.5.7-300.fc39.x86_64 kernel 2. log in to Plasma on Wayland 3. Start Konsole 4. Update to the kernel-6.5.8-300.fc39 with a dnf offline-upgrade with updates-testing enabled. sudo dnf offline-upgrade download sudo dnf offline-upgrade reboot 5. After the offline upgrade completes, boot 6.5.8 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: I haven't tried the latest Rawhide kernel. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Reproducible: Always
Created attachment 1994916 [details] Kernel log from /run/initramfs/rdosreport.txt in the failed boot of 6.5.8
The BLS conf file for 6.5.8 showed that the options line was missing the kernel command line arguments that were there in 6.5.7 and earlier. title Fedora Linux (6.5.8-300.fc39.x86_64) 39 (KDE Plasma) version 6.5.8-300.fc39.x86_64 linux /vmlinuz-6.5.8-300.fc39.x86_64 initrd /initramfs-6.5.8-300.fc39.x86_64.img options grub_users $grub_users grub_arg --unrestricted grub_class fedora title Fedora Linux (6.5.7-300.fc39.x86_64) 39 (KDE Plasma) version 6.5.7-300.fc39.x86_64 linux /vmlinuz-6.5.7-300.fc39.x86_64 initrd /initramfs-6.5.7-300.fc39.x86_64.img options root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rhgb quiet rdrand=force grub_users $grub_users grub_arg --unrestricted grub_class fedora I removed the crashkernel parameter that had been mistakenly added to the kernel command line in F39 and F40 with sudo grubby --remove-args="crashkernel" --update=ALL a week or so ago based on Chris Murphy's comment at https://bugzilla.redhat.com/show_bug.cgi?id=2243068#c6 I have kexec-tools-2.0.27-2.fc39 installed. My / and /home are ext4 on LVM and /boot is ext4.
Kernel actually has nothing to do with this. It does seem similar to https://bugzilla.redhat.com/show_bug.cgi?id=2239542 but as that stayed assigned to kernel, I am guessing the grub maintainer never saw it. Reassigning to grub so they can see what is going on.
(In reply to Justin M. Forbes from comment #3) > Kernel actually has nothing to do with this. It does seem similar to > https://bugzilla.redhat.com/show_bug.cgi?id=2239542 but as that stayed > assigned to kernel, I am guessing the grub maintainer never saw it. > Reassigning to grub so they can see what is going on. OK thanks. Could running sudo grubby --remove-args="crashkernel" --update=ALL as I did have changed the 6.5.7 and earlier BLS files in a way such that the 6.5.8 install programs incorrectly created the 6.5.8 BLS file without the kernel command line parameters on the options line? My 6.5.7 BLS file had options root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rhgb quiet rdrand=force. options $kernelopts was in the example at https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault (though that may be outdated). I manually edited my 6.5.8 BLS file to have options root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rhgb quiet rdrand=force and ran sudo grub2-mkconfig -o /boot/grub2/grub.cfg to work around this problem.
Hi Matt, Running that grubby command should not have caused this. Could you please share the versions of grub and grubby that you have installed? (rpm -qa grub\*) Also, when you did the upgrade, did you only upgrade the kernel, or other packages as well? Could you also please show the outputs of: $ cat /etc/kernel/cmdline $ grep CMDLINE /etc/default/grub thanks!
These are the grub and grubby versions I have installed. rpm -qa grub\* grubby-8.40-72.fc39.x86_64 grub2-common-2.06-104.fc39.noarch grub2-tools-minimal-2.06-104.fc39.x86_64 grub2-tools-2.06-104.fc39.x86_64 grub2-pc-modules-2.06-104.fc39.noarch grub2-pc-2.06-104.fc39.x86_64 grub2-efi-ia32-2.06-104.fc39.x86_64 grub2-efi-x64-2.06-104.fc39.x86_64 grub2-tools-extra-2.06-104.fc39.x86_64 grub2-efi-ia32-cdboot-2.06-104.fc39.x86_64 grub2-efi-x64-cdboot-2.06-104.fc39.x86_64 grub2-tools-efi-2.06-104.fc39.x86_64 There were also the following packages in the upgrade: glibc-2.38-9.fc39.x86_64, libqalculate-4.8.1-1.fc39.x86_64, mutter-45.0-10.fc39.x86_64, ostree-2023.7-2.fc39.x86_64, pam-1.5.3-3.fc39.x86_64, plasma-discover-5.27.8-2.fc39.x86_64, scap-security-guide-0.1.70-1.fc39.noarch. They don't look to be packages that would be involved with changing the BLS or kernel command line parameters to me. /etc/kernel/cmdline is an empty file even though I'm running 6.5.8 with the kernel command line BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.5.8-300.fc39.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rhgb quiet. That might be part of the problem. Thanks. $ cat /etc/kernel/cmdline $ grep CMDLINE /etc/default/grub GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rhgb quiet"
Ok, the empty /etc/kernel/cmdline seems to be the culprit then. I don't know why it's empty in your case, but kernel installation scripts look for that file, or in its absence /usr/lib/kernel/cmdline, and take the boot params from there. Invoking grubby --update-kernel ALL updates /etc/kernel/cmdline, but does not regenerate or fix it if it already exists. If you would want to recreate the file using grubby, you can rm it all together and then $ grubby --update-kernel ALL --args " " I guess you have managed to get everything working..? Can we close the bug?
I removed /etc/kernel/cmdline and ran sudo grubby --update-kernel ALL --args " ". /etc/kernel/cmdline was created as an empty file again. If /etc/kernel/cmdline didn't exist when I ran sudo grubby --remove-args="crashkernel" --update=ALL, that might've created an empty /etc/kernel/cmdline. I removed /etc/kernel/cmdline again and ran sudo grubby --remove-args="crashkernel" --update=ALL, and an empty /etc/kernel/cmdline was created. I worked around the problem to restore the kernel command line parameters as I mentioned in comment 4, but the problem of what made /etc/kernel/cmdline and the BLS file options line empty doesn't seem to have been fixed. Is it possible for the cause of the problem to be found and fixed? If not, I suppose the bug could be closed. If others follow the draft commonbugs advice to remove the crashkernel parameter at https://bugzilla.redhat.com/show_bug.cgi?id=2243068#c6, they may have the same problem. Should I put my kernel command line parameters into /etc/kernel/cmdline so that this problem doesn't happen with the next kernel update? Thanks.
Ok, a little more information, and I think we'll figure this out. :) Could you please: $ cat /usr/lib/kernel/cmdline $ grubby --info ALL thanks!
Here's the requested output. cat /usr/lib/kernel/cmdline cat: /usr/lib/kernel/cmdline: No such file or directory sudo grubby --info ALL index=0 kernel="/boot/vmlinuz-6.5.8-300.fc39.x86_64" args="ro rd.lvm.lv=fedora/root rhgb quiet" root="/dev/mapper/fedora-root" initrd="/boot/initramfs-6.5.8-300.fc39.x86_64.img" title="Fedora Linux (6.5.8-300.fc39.x86_64) 39 (KDE Plasma)" id="cf0bf479bcf04633b727cb244f663cd7-6.5.8-300.fc39.x86_64" index=1 kernel="/boot/vmlinuz-6.5.7-300.fc39.x86_64" args="ro rd.lvm.lv=fedora/root rhgb quiet" root="/dev/mapper/fedora-root" initrd="/boot/initramfs-6.5.7-300.fc39.x86_64.img" title="Fedora Linux (6.5.7-300.fc39.x86_64) 39 (KDE Plasma)" id="cf0bf479bcf04633b727cb244f663cd7-6.5.7-300.fc39.x86_64" index=2 kernel="/boot/vmlinuz-6.5.6-300.fc39.x86_64" args="ro rd.lvm.lv=fedora/root rhgb quiet" root="/dev/mapper/fedora-root" initrd="/boot/initramfs-6.5.6-300.fc39.x86_64.img" title="Fedora Linux (6.5.6-300.fc39.x86_64) 39 (KDE Plasma Prerelease)" id="cf0bf479bcf04633b727cb244f663cd7-6.5.6-300.fc39.x86_64" index=3 kernel="/boot/vmlinuz-6.5.5-300.fc39.x86_64" args="ro rd.lvm.lv=fedora/root rhgb quiet" root="/dev/mapper/fedora-root" initrd="/boot/initramfs-6.5.5-300.fc39.x86_64.img" title="Fedora Linux (6.5.5-300.fc39.x86_64) 39 (KDE Plasma Prerelease)" id="cf0bf479bcf04633b727cb244f663cd7-6.5.5-300.fc39.x86_64" index=4 kernel="/boot/vmlinuz-0-rescue-cf0bf479bcf04633b727cb244f663cd7" args="ro rd.lvm.lv=fedora/root rhgb quiet" root="/dev/mapper/fedora-root" initrd="/boot/initramfs-0-rescue-cf0bf479bcf04633b727cb244f663cd7.img" title="Fedora Linux (0-rescue-cf0bf479bcf04633b727cb244f663cd7) 39 (KDE Plasma Prerelease)" id="cf0bf479bcf04633b727cb244f663cd7-0-rescue" index=5 kernel="/boot/memtest86+x64.efi" args="" initrd="/boot" title="Memtest86+ (memtest86+x64.efi-6.20)" id="cf0bf479bcf04633b727cb244f663cd7-0-memtest86+" I think if grubby could be changed not to create an empty /etc/kernel/cmdline if it doesn't exist when running those commands, the problem might be avoided. Thanks.
Ok, here's what happened: you ran the grubby command to remove crashkernel from your cmdline and it generated /etc/kernel/cmdline, which was empty because you have empty args in your last bls entry (index=5). grubby populates this file with args from the last entry; it's usually the rescue kernel. Then you updated your kernel, which tried to get its boot params from /etc/kernel/cmdline, and got nothing because the file is empty. I don't understand why you didn't have /etc/kernel/cmdline in the first place. When grubby gets installed, it creates that file if it doesn't exist, and I imagine that when you installed your system, you didn't have memtest with no args as the last entry from the beginning? Or maybe you did..? Kernel installation scripts try /etc/kernel/cmdline, /usr/lib/kernel/cmdline, and /proc/cmdline (in that order) to get boot params. Problem is, that if one of those exists, but is empty, then boot params don't get set. If you populate your /etc/kernel/cmdline appropriately, you should not have this problem when you do your next kernel update, but I believe the behaviour of grubby will stay as is. :)
Your explanation makes sense. I installed my system in July 2020 from a Fedora Rawhide/F33 live image. I built and installed many mainline kernels while bisecting problems which might've changed or deleted /etc/kernel/cmdline possibly when I was removing them. I'm not sure what /etc/kernel/cmdline was when I ran sudo grubby --remove-args="crashkernel" --update=ALL the first time. memtest86+ was only added to the grub menu in updates in mid 2023. I had a problem with two Memtest86+ grub menu entries after updating to memtest86+-6.20-2.fc38.x86_64 and the 6.3.11 kernel in which the earlier memtest86+ was in the first grub entry and the later one was in the last grub entry which was fixed by memtest86+-6.20-3.fc38 removing the first entry. https://bugzilla.redhat.com/show_bug.cgi?id=2219178 Since memtest86+ was installed by default at least on my system in the last grub entry since then, I don't think that is a good assumption for grubby to choose the kernel command line parameters of the last grub entry to put in /etc/kernel/cmdline if it doesn't exist. The first grub entry would be a better choice as it's the latest kernel. Should this report be reassigned to grubby? Thanks.
This message is a reminder that Fedora Linux 39 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 39 on 2024-11-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '39'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 39 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 39 entered end-of-life (EOL) status on 2024-11-26. Fedora Linux 39 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.