When new kernel is installed with the not-the-latest systemd, it will not generate initramfs and vmlinuz files correctly and also adds broken grub entry. --------------------------------------------------------------------------------------------- # STEPS TO REPRODUCE 1) Install Fedora 30 2) exclude kernel-core and systemd from DNF, update the rest of the system (to verify only those excluded pkgs are problematic) # echo "exclude=systemd* kernel-core" >> /etc/dnf/dnf.conf # dnf update 3) drop the exlude and verify the installed version of the originally excluded packages: # dnf list installed | grep -e kernel -e systemd kernel-core.x86_64 5.0.9-301.fc30 @fedora-custom rpm-plugin-systemd-inhibit.x86_64 4.14.2.1-4.fc30.1 @fedora-custom systemd.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-bootchart.x86_64 233-4.fc30 @fedora-custom systemd-libs.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-pam.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-rpm-macros.noarch 239-10.git3bf819c.fc30 @@commandline systemd-udev.x86_64 239-10.git3bf819c.fc30 @@commandline 4) take a look at /boot/ # ls -1 /boot/ | grep -e initramfs -e vmlimuz initramfs-0-rescue-83c27267f5a745bb8c1f380d84074e4e.img initramfs-5.0.9-301.fc30.x86_64.img vmlinuz-0-rescue-83c27267f5a745bb8c1f380d84074e4e vmlinuz-5.0.9-301.fc30.x86_64 5) exclude systemd package # echo "exclude=systemd*" >> /etc/dnf/dnf.conf 6) update the kernel # dnf update kernel-core x86_64 5.0.17-300.fc30 --------------------------------------------------------------------------------------------- # VERIFICATION OF THE ISSUE 7) take a look at /boot/ again. No new initramfs or vmlinuz added. # ls -1 /boot/ | grep -e initramfs -e vmlimuz initramfs-0-rescue-83c27267f5a745bb8c1f380d84074e4e.img initramfs-5.0.9-301.fc30.x86_64.img vmlinuz-0-rescue-83c27267f5a745bb8c1f380d84074e4e vmlinuz-5.0.9-301.fc30.x86_64 *) but a wild hash appeared! # ls -1 /boot/ config-5.0.9-301.fc30.x86_64 efi grub2 initramfs-0-rescue-83c27267f5a745bb8c1f380d84074e4e.img initramfs-5.0.9-301.fc30.x86_64.img loader lost+found System.map-5.0.9-301.fc30.x86_64 vmlinuz-0-rescue-83c27267f5a745bb8c1f380d84074e4e vmlinuz-5.0.9-301.fc30.x86_64 83c27267f5a745bb8c1f380d84074e4e # tree /boot/83c27267f5a745bb8c1f380d84074e4e/ /boot/83c27267f5a745bb8c1f380d84074e4e/ ├── 0-rescue │ ├── initrd │ └── linux └── 5.0.17-300.fc30.x86_64 ├── initrd └── linux *) And while 5.0.9 kernel has valid grub boot entry, 5.0.17 doesn't. # cat /boot/loader/entries/83c27267f5a745bb8c1f380d84074e4e-5.0.9-301.fc30.x86_64.conf title Fedora (5.0.9-301.fc30.x86_64) 30 (Thirty) version 5.0.9-301.fc30.x86_64 linux /vmlinuz-5.0.9-301.fc30.x86_64 initrd /initramfs-5.0.9-301.fc30.x86_64.img options $kernelopts grub_users $grub_users grub_arg --unrestricted grub_class kernel # cat /boot/loader/entries/83c27267f5a745bb8c1f380d84074e4e-5.0.17-300.fc30.x86_64.conf title Fedora 30 (Thirty) version 5.0.17-300.fc30.x86_64 machine-id 83c27267f5a745bb8c1f380d84074e4e options BOOT_IMAGE=vmlinuz root=live:CDLABEL=Fedora-Cinn-Live-30-1-2 rd.live.image quiet linux /83c27267f5a745bb8c1f380d84074e4e/5.0.17-300.fc30.x86_64/linux initrd /83c27267f5a745bb8c1f380d84074e4e/5.0.17-300.fc30.x86_64/initrd --------------------------------------------------------------------------------------------- # VERIFICATION THAT SYSTEMD PACKAGE IS THE CAUSE 8) Let's uninstall the last kernel update and delete the hash directory under /boot/ # dnf history # dnf history undo <number> # rm -rf /boot/83c27267f5a745bb8c1f380d84074e4e 9) drop all of the excludes from /etc/dnf/dnf.conf *) So now we are back on: # dnf list installed | grep -e kernel -e systemd kernel-core.x86_64 5.0.9-301.fc30 @fedora-custom rpm-plugin-systemd-inhibit.x86_64 4.14.2.1-4.fc30.1 @fedora-custom systemd.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-bootchart.x86_64 233-4.fc30 @fedora-custom systemd-libs.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-pam.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-rpm-macros.noarch 239-10.git3bf819c.fc30 @@commandline systemd-udev.x86_64 239-10.git3bf819c.fc30 @@commandline 10) Update (now both systemd and kernel-core will update) # dnf update # dnf list installed | grep -e kernel -e systemd kernel-core.x86_64 5.0.9-301.fc30 @fedora-custom kernel-core.x86_64 5.0.17-300.fc30 @updates rpm-plugin-systemd-inhibit.x86_64 4.14.2.1-4.fc30.1 @fedora-custom systemd.x86_64 241-8.git9ef65cb.fc30 @updates systemd-bootchart.x86_64 233-4.fc30 @fedora-custom systemd-libs.x86_64 241-8.git9ef65cb.fc30 @updates systemd-pam.x86_64 241-8.git9ef65cb.fc30 @updates systemd-rpm-macros.noarch 241-8.git9ef65cb.fc30 @updates systemd-udev.x86_64 241-8.git9ef65cb.fc30 @updates *) Bot now the same as in step 7) happened !! 11) only after reinstalling of the kernel-core package AFTER the systemd was updated, it will produce correct output: # dnf remove kernel-core-5.0.17-300.fc30.x86_64 # rm -rf /boot/83c27267f5a745bb8c1f380d84074e4e # dnf install kernel-core-5.0.17-300.fc30.x86_64 # ls -1 /boot/ | grep -e initramfs -e vmlinuz initramfs-0-rescue-83c27267f5a745bb8c1f380d84074e4e.img initramfs-5.0.17-300.fc30.x86_64.img initramfs-5.0.9-301.fc30.x86_64.img vmlinuz-0-rescue-83c27267f5a745bb8c1f380d84074e4e vmlinuz-5.0.17-300.fc30.x86_64 vmlinuz-5.0.9-301.fc30.x86_64 # cat /boot/loader/entries/83c27267f5a745bb8c1f380d84074e4e-5.0.17-300.fc30.x86_64.conf title Fedora (5.0.17-300.fc30.x86_64) 30 (Thirty) version 5.0.17-300.fc30.x86_64 linux /vmlinuz-5.0.17-300.fc30.x86_64 initrd /initramfs-5.0.17-300.fc30.x86_64.img options $kernelopts grub_users $grub_users grub_arg --unrestricted grub_class kernel --------------------------------------------------------------------------------------------- Filling against kernel, since it is actually kernel post scriplet that fails to create valid files and grub boot entries. I did this in chroot on a mounted system. No systemd was running, no journal exists. Setting "Severity: urgent", since it blocks #1712935 thus leads to bricked system. Always reproducible for me. A "@fedora-custom" repo is mentioned above. it was generated by: echo "\ [fedora-custom] name=fedora-custom enabled=1 gpgcheck=0 metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$OS&arch=x86_64" \ > /etc/yum.repos.d/fedora-custom.repo So it is actually just a custom entry pointing to the official repo.
I swapped steps 2) & 3). Once you exclude something, even "dnf list installed" won't find it. --------------------------------------------------------------------------------------------- Anyway, reproduces on a living system installed by Anaconda. 1) Update the rest fo the system # echo "exclude=systemd* kernel-core" >> /etc/dnf/dnf.conf # dnf update *) Drop the exclude in dnf.conf, check the installed packages # dnf list installed | grep -e kernel -e systemd abrt-addon-kerneloops.x86_64 2.12.0-2.fc30 @anaconda kernel.x86_64 5.0.9-301.fc30 @anaconda kernel-core.x86_64 5.0.9-301.fc30 @anaconda kernel-headers.x86_64 5.0.16-300.fc30 @updates kernel-modules.x86_64 5.0.9-301.fc30 @anaconda kernel-modules-extra.x86_64 5.0.9-301.fc30 @anaconda libreport-plugin-kerneloops.x86_64 2.10.0-3.fc30 @updates libreport-plugin-systemd-journal.x86_64 2.10.0-3.fc30 @updates python-systemd-doc.x86_64 234-8.fc30 @anaconda python3-systemd.x86_64 234-8.fc30 @anaconda rpm-plugin-systemd-inhibit.x86_64 4.14.2.1-4.fc30.1 @anaconda systemd.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-bootchart.x86_64 233-4.fc30 @anaconda systemd-libs.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-pam.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-rpm-macros.noarch 239-10.git3bf819c.fc30 @@commandline systemd-udev.x86_64 239-10.git3bf819c.fc30 @@commandline *) Check /boot/ # ls -1 /boot/ | grep -e initramfs -e vmlinuz initramfs-0-rescue-11d01583f74b42f5bdfa370ec6d2b135.img initramfs-5.0.9-301.fc30.x86_64.img vmlinuz-0-rescue-11d01583f74b42f5bdfa370ec6d2b135 vmlinuz-5.0.9-301.fc30.x86_64 # cat /boot/loader/entries/11d01583f74b42f5bdfa370ec6d2b135-5.0.9-301.fc30.x86_64.conf title Fedora (5.0.9-301.fc30.x86_64) 30 (Thirty) version 5.0.9-301.fc30.x86_64 linux /vmlinuz-5.0.9-301.fc30.x86_64 initrd /initramfs-5.0.9-301.fc30.x86_64.img options $kernelopts grub_users $grub_users grub_arg --unres 2) exclude systemd* only, update kernel # echo "exclude=systemd*" >> /etc/dnf/dnf.conf # dnf update 3) Quick check of /boot/ ... same as 7) # ls -1 /boot/ | grep -e initramfs -e vmlinuz initramfs-0-rescue-11d01583f74b42f5bdfa370ec6d2b135.img initramfs-5.0.9-301.fc30.x86_64.img vmlinuz-0-rescue-11d01583f74b42f5bdfa370ec6d2b135 vmlinuz-5.0.9-301.fc30.x86_64 # tree /boot/11d01583f74b42f5bdfa370ec6d2b135/ /boot/11d01583f74b42f5bdfa370ec6d2b135/ ├── 0-rescue │ ├── initrd │ └── linux └── 5.0.17-300.fc30.x86_64 ├── initrd └── linux # cat /boot/loader/entries/11d01583f74b42f5bdfa370ec6d2b135-5.0.17-300.fc30.x86_64.conf title Fedora 30 (Thirty) version 5.0.17-300.fc30.x86_64 machine-id 11d01583f74b42f5bdfa370ec6d2b135 options BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.0.9-301.fc30.x86_64 root=/dev/sda3 ro rhgb quiet linux /11d01583f74b42f5bdfa370ec6d2b135/5.0.17-300.fc30.x86_64/linux initrd /11d01583f74b42f5bdfa370ec6d2b135/5.0.17-300.fc30.x86_64/initrd *) Again we see there's some totally random hash directory created in /boot/ . However this time, the bootloader entry is actually bootable. But it still is damaged somehow, not showing kernel version, for example. # dnf list installed | grep -e kernel -e systemd abrt-addon-kerneloops.x86_64 2.12.0-2.fc30 @anaconda kernel.x86_64 5.0.9-301.fc30 @anaconda kernel.x86_64 5.0.17-300.fc30 @updates kernel-core.x86_64 5.0.9-301.fc30 @anaconda kernel-core.x86_64 5.0.17-300.fc30 @updates kernel-headers.x86_64 5.0.16-300.fc30 @updates kernel-modules.x86_64 5.0.9-301.fc30 @anaconda kernel-modules.x86_64 5.0.17-300.fc30 @updates kernel-modules-extra.x86_64 5.0.9-301.fc30 @anaconda kernel-modules-extra.x86_64 5.0.17-300.fc30 @updates libreport-plugin-kerneloops.x86_64 2.10.0-3.fc30 @updates libreport-plugin-systemd-journal.x86_64 2.10.0-3.fc30 @updates python-systemd-doc.x86_64 234-8.fc30 @anaconda python3-systemd.x86_64 234-8.fc30 @anaconda rpm-plugin-systemd-inhibit.x86_64 4.14.2.1-4.fc30.1 @anaconda systemd.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-bootchart.x86_64 233-4.fc30 @anaconda systemd-libs.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-pam.x86_64 239-10.git3bf819c.fc30 @@commandline systemd-rpm-macros.noarch 239-10.git3bf819c.fc30 @@commandline systemd-udev.x86_64 239-10.git3bf819c.fc30 @@commandline --------------------------------------------------------------------------------------------- 4) So let's reboot from that old kernel and undo the dnf update # dnf history undo <number> *) And remove the trash left behind # rm -rf /boot/11d01583f74b42f5bdfa370ec6d2b135/ *) exclude only "kernel-core" this time, update (systemd) # vi /etc/dnf/dnf.conf # dnf update *) Drop all excludes and update one last time (the kerne-core this time) # vi /etc/dnf/dnf.conf # dnf update *) The result is following: # ls -1 /boot/ | grep -e initramfs -e vmlinuz initramfs-0-rescue-11d01583f74b42f5bdfa370ec6d2b135.img initramfs-5.0.17-300.fc30.x86_64.img initramfs-5.0.9-301.fc30.x86_64.img vmlinuz-0-rescue-11d01583f74b42f5bdfa370ec6d2b135 vmlinuz-5.0.17-300.fc30.x86_64 vmlinuz-5.0.9-301.fc30.x86_64 # cat /boot/loader/entries/11d01583f74b42f5bdfa370ec6d2b135-5.0.17-300.fc30.x86_64.conf title Fedora (5.0.17-300.fc30.x86_64) 30 (Thirty) version 5.0.17-300.fc30.x86_64 linux /vmlinuz-5.0.17-300.fc30.x86_64 initrd /initramfs-5.0.17-300.fc30.x86_64.img options $kernelopts grub_users $grub_users grub_arg --unrestricted grub_class kernel # dnf list installed | grep -e kernel -e systemd abrt-addon-kerneloops.x86_64 2.12.0-2.fc30 @anaconda kernel.x86_64 5.0.9-301.fc30 @anaconda kernel.x86_64 5.0.17-300.fc30 @updates kernel-core.x86_64 5.0.9-301.fc30 @anaconda kernel-core.x86_64 5.0.17-300.fc30 @updates kernel-headers.x86_64 5.0.16-300.fc30 @updates kernel-modules.x86_64 5.0.9-301.fc30 @anaconda kernel-modules.x86_64 5.0.17-300.fc30 @updates kernel-modules-extra.x86_64 5.0.9-301.fc30 @anaconda kernel-modules-extra.x86_64 5.0.17-300.fc30 @updates libreport-plugin-kerneloops.x86_64 2.10.0-3.fc30 @updates libreport-plugin-systemd-journal.x86_64 2.10.0-3.fc30 @updates python-systemd-doc.x86_64 234-8.fc30 @anaconda python3-systemd.x86_64 234-8.fc30 @anaconda rpm-plugin-systemd-inhibit.x86_64 4.14.2.1-4.fc30.1 @anaconda systemd.x86_64 241-8.git9ef65cb.fc30 @updates systemd-bootchart.x86_64 233-4.fc30 @anaconda systemd-libs.x86_64 241-8.git9ef65cb.fc30 @updates systemd-pam.x86_64 241-8.git9ef65cb.fc30 @updates systemd-rpm-macros.noarch 241-8.git9ef65cb.fc30 @updates systemd-udev.x86_64 241-8.git9ef65cb.fc30 @updates
Created attachment 1573705 [details] systemd journal from the steps in comment 1 systemd journal from the steps in comment 1
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 30 kernel bugs. Fedora 30 has now been rebased to 5.2.9-200.fc30. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 31, and are still experiencing this issue, please change the version to Fedora 31. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.