Created attachment 1342921 [details] Screenshot Description of problem: Booting the new 4.13.8-300 kernel on F27 beta results in kernel panic on Dell XPS 13 9350. Booting works fine on previous kernel 4.13.5-300 Version-Release number of selected component (if applicable): 4.13.8-300 How reproducible: Always Steps to Reproduce: 1. Boot Actual results: Kernel Panic Expected results: Boot to gdm Additional info: Standard kernel, nothing special
We should probably at least look into this one as a Final blocker. What is your storage configuration? Is it some sort of fairly typical install, or a more unusual one? Thanks.
Just a standard LVM workstation install. [jh@xps ~]$ LANG=c sudo fdisk -l Disk /dev/nvme0n1: 238.5 GiB, 256060514304 bytes, 500118192 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 9FCDD38C-7D3F-48A0-8340-9F0061ACAAD5 Device Start End Sectors Size Type /dev/nvme0n1p1 2048 411647 409600 200M EFI System /dev/nvme0n1p2 411648 1435647 1024000 500M Linux filesystem /dev/nvme0n1p3 1435648 500117503 498681856 237.8G Linux LVM Disk /dev/mapper/fedora-root: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/fedora-swap: 7.8 GiB, 8321499136 bytes, 16252928 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/fedora-home: 180 GiB, 193311277056 bytes, 377561088 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes [jh@xps ~]$ cat /etc/fstab # # /etc/fstab # Created by anaconda on Sun May 15 13:42:41 2016 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/fedora-root / ext4 defaults 1 1 UUID=30f2cdfb-0861-479e-9e9d-acf2a5473d03 /boot ext4 defaults 1 2 UUID=9CF8-D369 /boot/efi vfat umask=0077,shortname=winnt 0 2 /dev/mapper/fedora-home /home ext4 defaults 1 2 /swap swap swap defaults 0 0
I just remembered that the swap file at the bottom of the fstab might be non-standard, as I added this to get more swap space for a memory intensive application. However removing that doesn't fix the problem. Neither does adding back the LVM swap partition, which for some reason got lost. The previous kernel worked fine on all these configurations.
The screenshot doesn't tell anything useful. Can you get details about the messages before that message? Can you also ls -l /boot/initramfs-*
There are no messages before that message. It shows grub and then immediately the error message I photographed. Do I need to enable further debug? If yes, how? I can post the ls output in about 8 hours when I'm home, currently at work.
Yes, please remove 'quiet' from the kernel command line
Created attachment 1343315 [details] Screenshot 2 Tried to take a screenshot, not sure if its helpful. Also the font is really tiny on the hidpi screen. The initramfs seems to be missing: [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img also I noticed while trying to take the screenshot that the grub commandline is missing the bottom initrd line for the problematic kernel.
Created attachment 1343319 [details] generated grub.cfg
initramfs missing would certainly be the problem. If you remove and re-install the kernel (while booted from a *different* kernel, of course), does the same thing happen?
Yes, the same problem happens [jh@xps ~]$ sudo dnf remove kernel-4.13.8-300.fc27.x86_64 Dependencies resolved. ============================================================================================================================================ Package Arch Version Repository Size ============================================================================================================================================ Removing: kernel x86_64 4.13.8-300.fc27 @System 0 Transaction Summary ============================================================================================================================================ Remove 1 Package Freed space: 0 Is this ok [y/N]: y Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : kernel-4.13.8-300.fc27.x86_64 1/1 Running scriptlet: kernel-4.13.8-300.fc27.x86_64 1/1 Verifying : kernel-4.13.8-300.fc27.x86_64 1/1 Removed: kernel.x86_64 4.13.8-300.fc27 Complete! [jh@xps ~]$ sudo dnf install kernel-4.13.8-300.fc27.x86_64 Failed to synchronize cache for repo 'fedora-steam', disabling. Last metadata expiration check: 1:04:37 ago on Wed 25 Oct 2017 18:25:19 CEST. Dependencies resolved. ============================================================================================================================================ Package Arch Version Repository Size ============================================================================================================================================ Installing: kernel x86_64 4.13.8-300.fc27 updates-testing 74 k Transaction Summary ============================================================================================================================================ Install 1 Package Total size: 74 k Installed size: 0 Is this ok [y/N]: y Downloading Packages: [SKIPPED] kernel-4.13.8-300.fc27.x86_64.rpm: Already downloaded Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : kernel-4.13.8-300.fc27.x86_64 1/1 Verifying : kernel-4.13.8-300.fc27.x86_64 1/1 Installed: kernel.x86_64 4.13.8-300.fc27 Complete! [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img After reboot: Same problem with this kernel, initramfs file still missing, still have to boot the previous kernel.
that's...huh. that's rather strange. No errors reported, but clearly the initramfs generation just didn't happen. What happens if you run dracut directly? I think it'd be: dracut -f --kver 4.13.8-300.fc27
That caused the initramfs file to generate: [jh@xps ~]$ sudo dracut -f --kver 4.13.8-300.fc27.x86_64 [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img -rw-------. 1 root root 69574214 Oct 25 20:17 /boot/initramfs-4.13.8-300.fc27.x86_64.img
Hum, okay. How about if you remove the package again, re-install it, then run: /bin/kernel-install add 4.13.8-300.fc27.x86_64 /lib/modules/4.13.8-300.fc27.x86_64/vmlinuz ?
Okay, removing and reinstalling changed nothing, the initramfs file was neither removed nor regenerated. The kernel-install command created a new initramfs file (timestamp and file size change). [jh@xps ~]$ sudo dnf remove kernel-4.13.8-300.fc27.x86_64 [sudo] password for jh: Dependencies resolved. ================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================ Removing: kernel x86_64 4.13.8-300.fc27 @updates-testing 0 Transaction Summary ================================================================================================================================ Remove 1 Package Freed space: 0 Is this ok [y/N]: y Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : kernel-4.13.8-300.fc27.x86_64 1/1 Running scriptlet: kernel-4.13.8-300.fc27.x86_64 1/1 Verifying : kernel-4.13.8-300.fc27.x86_64 1/1 Removed: kernel.x86_64 4.13.8-300.fc27 Complete! [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img -rw-------. 1 root root 69574214 Oct 25 20:17 /boot/initramfs-4.13.8-300.fc27.x86_64.img [jh@xps ~]$ sudo dnf install kernel-4.13.8-300.fc27.x86_64 Failed to synchronize cache for repo 'fedora-steam', disabling. Last metadata expiration check: 2:41:19 ago on Wed 25 Oct 2017 18:25:19 CEST. Dependencies resolved. ================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================ Installing: kernel x86_64 4.13.8-300.fc27 updates-testing 74 k Transaction Summary ================================================================================================================================ Install 1 Package Total download size: 74 k Installed size: 0 Is this ok [y/N]: y Downloading Packages: kernel-4.13.8-300.fc27.x86_64.rpm 496 kB/s | 74 kB 00:00 -------------------------------------------------------------------------------------------------------------------------------- Total 46 kB/s | 74 kB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : kernel-4.13.8-300.fc27.x86_64 1/1 Verifying : kernel-4.13.8-300.fc27.x86_64 1/1 Installed: kernel.x86_64 4.13.8-300.fc27 Complete! [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img -rw-------. 1 root root 69574214 Oct 25 20:17 /boot/initramfs-4.13.8-300.fc27.x86_64.img [jh@xps ~]$ sudo /bin/kernel-install add 4.13.8-300.fc27.x86_64 /lib/modules/4.13.8-300.fc27.x86_64/vmlinuz [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img -rw-------. 1 root root 69574196 Oct 25 21:07 /boot/initramfs-4.13.8-300.fc27.x86_64.img
Also I tried to remove some older kernels because my boot partition is getting full, but that doesn't remove the corresponding initfamfs files: [jh@xps ~]$ sudo dnf remove kernel-4.11.11-300.fc26.x86_64 kernel-4.12.8-300.fc26.x86_64 After that running the ls command shows that both corrsponging initramfs files are still there and no space is freed up. I assume some link between the kernel packages and the initramfs files is broken for me.
Oh, duh - I just realized to an extent we're testing wrong. You need to be removing / reinstalling kernel-core as well as kernel. kernel-core is the package that should produce the initramfs file (during the %posttrans phase). Removing kernel-core for the older kernels should also remove the actual kernel and initramfs; you'll also need to remove all other subpackages to fully remove the kernel (inc. kernel-modules and kernel-modules-extra if it's installed). Can you: 1. From a different kernel, remove the kernel and kernel-core packages of the relevant version 2. Delete the initramfs file if it stays around 3. Re-install kernel and kernel-core 4. See if initramfs file shows up this time 5. If not, run /bin/kernel-install add 4.13.8-300.fc27.x86_64 /lib/modules/4.13.8-300.fc27.x86_64/vmlinuz and see what happens thanks!
Sure. This time a new initramfs file is generated. However there is a weird "cat: write error: Broken pipe" during the installation. [jh@xps ~]$ sudo dnf remove kernel-core-4.13.8-300.fc27.x86_64 Dependencies resolved. ================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================ Removing: kernel-core x86_64 4.13.8-300.fc27 @updates-testing 55 M Removing dependent packages: kernel x86_64 4.13.8-300.fc27 @updates-testing 0 kernel-modules x86_64 4.13.8-300.fc27 @updates-testing 24 M kernel-modules-extra x86_64 4.13.8-300.fc27 @updates-testing 2.0 M Transaction Summary ================================================================================================================================ Remove 4 Packages Freed space: 80 M Is this ok [y/N]: y Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : kernel-4.13.8-300.fc27.x86_64 1/4 Running scriptlet: kernel-4.13.8-300.fc27.x86_64 1/4 Erasing : kernel-modules-extra-4.13.8-300.fc27.x86_64 2/4 Running scriptlet: kernel-modules-extra-4.13.8-300.fc27.x86_64 2/4 Erasing : kernel-modules-4.13.8-300.fc27.x86_64 3/4 Running scriptlet: kernel-modules-4.13.8-300.fc27.x86_64 3/4 Running scriptlet: kernel-core-4.13.8-300.fc27.x86_64 4/4 Erasing : kernel-core-4.13.8-300.fc27.x86_64 4/4 Verifying : kernel-modules-4.13.8-300.fc27.x86_64 1/4 Verifying : kernel-core-4.13.8-300.fc27.x86_64 2/4 Verifying : kernel-modules-extra-4.13.8-300.fc27.x86_64 3/4 Verifying : kernel-4.13.8-300.fc27.x86_64 4/4 Removed: kernel-core.x86_64 4.13.8-300.fc27 kernel.x86_64 4.13.8-300.fc27 kernel-modules.x86_64 4.13.8-300.fc27 kernel-modules-extra.x86_64 4.13.8-300.fc27 Complete! [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img [jh@xps ~]$ sudo dnf install kernel-4.13.8-300.fc27.x86_64 kernel-core-4.13.8-300.fc27.x86_64 kernel-modules-extra-4.13.8-300.fc27.x86_64 Failed to synchronize cache for repo 'fedora-steam', disabling. Last metadata expiration check: 3:06:02 ago on Wed 25 Oct 2017 18:25:19 CEST. Dependencies resolved. ================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================ Installing: kernel x86_64 4.13.8-300.fc27 updates-testing 74 k kernel-core x86_64 4.13.8-300.fc27 updates-testing 21 M kernel-modules-extra x86_64 4.13.8-300.fc27 updates-testing 2.2 M Installing dependencies: kernel-modules x86_64 4.13.8-300.fc27 updates-testing 24 M Transaction Summary ================================================================================================================================ Install 4 Packages Total download size: 47 M Installed size: 80 M Is this ok [y/N]: y Downloading Packages: (1/4): kernel-4.13.8-300.fc27.x86_64.rpm 466 kB/s | 74 kB 00:00 (2/4): kernel-modules-extra-4.13.8-300.fc27.x86_64.rpm 1.0 MB/s | 2.2 MB 00:02 (3/4): kernel-core-4.13.8-300.fc27.x86_64.rpm 1.2 MB/s | 21 MB 00:17 (4/4): kernel-modules-4.13.8-300.fc27.x86_64.rpm 1.1 MB/s | 24 MB 00:21 -------------------------------------------------------------------------------------------------------------------------------- Total 2.1 MB/s | 47 MB 00:22 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : kernel-core-4.13.8-300.fc27.x86_64 1/4 Running scriptlet: kernel-core-4.13.8-300.fc27.x86_64 1/4 Installing : kernel-modules-4.13.8-300.fc27.x86_64 2/4 Running scriptlet: kernel-modules-4.13.8-300.fc27.x86_64 2/4 Installing : kernel-4.13.8-300.fc27.x86_64 3/4 Installing : kernel-modules-extra-4.13.8-300.fc27.x86_64 4/4 Running scriptlet: kernel-modules-extra-4.13.8-300.fc27.x86_64 4/4 Running scriptlet: kernel-core-4.13.8-300.fc27.x86_64 4/4 cat: write error: Broken pipe Verifying : kernel-4.13.8-300.fc27.x86_64 1/4 Verifying : kernel-core-4.13.8-300.fc27.x86_64 2/4 Verifying : kernel-modules-extra-4.13.8-300.fc27.x86_64 3/4 Verifying : kernel-modules-4.13.8-300.fc27.x86_64 4/4 Installed: kernel.x86_64 4.13.8-300.fc27 kernel-core.x86_64 4.13.8-300.fc27 kernel-modules-extra.x86_64 4.13.8-300.fc27 kernel-modules.x86_64 4.13.8-300.fc27 Complete! [jh@xps ~]$ ls -l /boot/initramfs-* -rw-------. 1 root root 52620296 May 15 2016 /boot/initramfs-0-rescue-6db87eaf8c314c6fa971af0b823b551e.img -rw-------. 1 root root 63629512 Jul 25 18:43 /boot/initramfs-4.11.11-300.fc26.x86_64.img -rw-------. 1 root root 63956434 Aug 27 20:13 /boot/initramfs-4.12.8-300.fc26.x86_64.img -rw-------. 1 root root 65817534 Oct 14 08:01 /boot/initramfs-4.13.5-200.fc26.x86_64.img -rw-------. 1 root root 69543251 Oct 23 20:17 /boot/initramfs-4.13.5-300.fc27.x86_64.img -rw-------. 1 root root 69574196 Oct 25 21:32 /boot/initramfs-4.13.8-300.fc27.x86_64.img
The problem is solved now however, I can boot the new kernel, despite the weird "cat" error.
hum, so sounds like this was possibly some kind of weird transient error, possibly a result of you running out of space in /boot ? doesn't seem to be a general problem, in any case, as several folks have given positive reports on -8 and -9, and I don't think the code related to generating initramfs was actually *changed* at all. So I'm withdrawing the blocker nomination at least...
Yes, you're probably right. Thanks for investigating, I will keep an eye open if it happens again the next update. I've freed up some space on the boot partition.
We've seen issues like this in the past and it's been related to grubby. I'll move this over there for tracking for now.
This message is a reminder that Fedora 27 is nearing its end of life. On 2018-Nov-30 Fedora will stop maintaining and issuing updates for Fedora 27. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '27'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 27 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.