Bug 1028477

Summary: Fedora won't boot after kernel update to 3.11.7-200
Product: [Fedora] Fedora Reporter: Paulo Fessel <pfessel>
Component: dracutAssignee: dracut-maint
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 20CC: chaapala, collura, dracut-maint, gansalmon, harald, itamar, jonathan, kernel-maint, madhu.chinakonda
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-29 12:51:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
sosreport.txt generated when the boot process failed none

Description Paulo Fessel 2013-11-08 14:37:36 UTC
Created attachment 821647 [details]
sosreport.txt generated when the boot process failed

Description of problem: Fedora 19 won't boot after kernel update


Version-Release number of selected component (if applicable): 3.11.7-200


How reproducible: update kernel to Fedora's latest version. 


Steps to Reproduce:
1. Update kernel to latest version 3.11.7-200. In my case, I've done a "sudo yum -y update".
2. Reboot machine.
3. Wait until the messages appear

Warning: Could not boot.
Warning: /dev/disk/by-uuid/759e16c6-94d8-480f-bf99-9e3a65a09b38 does not exist
Starting Dracut Emergency Shell...

Actual results: system won't boot and will drop to an emergency shell from where sosreport.txt can be copied.

Expected results: system should boot normally.

Additional info: my machine has three md arrays, and the UUID the message refers to belongs to my /dev/md2 which is my swap filesystem. This mirror is successfully activated when I boot with 3.11.6-201. Couldn't check if MD driver was activated at all during this boot.

Comment 1 Paulo Fessel 2013-11-08 15:06:43 UTC
Just found that removing the options "rd.md=0 rd.lvm=0 rd.dm=0 rd.luks=0" before booting fixes the issue. System is now up and running, but in my oppinion this is just a workaround since the system worked previously and booted correctly even with these boot options.

Comment 2 Josh Boyer 2013-11-08 15:25:38 UTC
Weird.  Those options aren't parsed by the kernel, so sounds like a dracut or mdadm issue in the initramfs.

Comment 3 Harald Hoyer 2013-11-11 09:45:25 UTC
(In reply to Paulo Fessel from comment #1)
> Just found that removing the options "rd.md=0 rd.lvm=0 rd.dm=0 rd.luks=0"
> before booting fixes the issue. System is now up and running, but in my
> oppinion this is just a workaround since the system worked previously and
> booted correctly even with these boot options.

Did you add a swap device on your raid recently?

Comment 4 Paulo Fessel 2013-11-11 11:27:44 UTC
Yes. I've made space on a 1.8TB RAID space which I previously had in order to create a new swap device, as I was moving my boot device (which also contained the swap space) to a SSD. However, the system booted without problems on 3.11.6-200 and 3.11.6-201 - the problem occurred only with 3.11.7-200.

Comment 5 Harald Hoyer 2013-11-11 12:35:02 UTC
(In reply to Paulo Fessel from comment #4)
> Yes. I've made space on a 1.8TB RAID space which I previously had in order
> to create a new swap device, as I was moving my boot device (which also
> contained the swap space) to a SSD. However, the system booted without
> problems on 3.11.6-200 and 3.11.6-201 - the problem occurred only with
> 3.11.7-200.

Just remove "rd.md=0" from the kernel command line.

The problem is, that for resume to work, we have to wait for all swap devices to be assembled.

To handle that situation more gracefully is still on my TODO list. Sorry.

Comment 6 Clay Haapala 2013-11-14 20:39:04 UTC
I just installed a fresh Fed 19 from the live disk. I have a BIOS raid mirror and a md raid mirror, booting from the BIOS raid, which has the swap.

The install's 3.9.5-301 kernel boots fine, but when I installed the 1133 update packages and picked up 3.11.7-200 (and the mdadm, etc. updates), I can no longer boot. The system hangs displaying flashing graphic garbage on my old nVidia GeForce (nvidia drivers not installed). Luckily, the 3.9.5 kernel remains.

Prior to the fresh install, the system had been running since Fedora 16 and all that remained on it were two or three 3.11.x kernels, since old kernels are pruned off at update time. This meant that no bootable kernel remained after receive the mdadm-or-whatever update that triggers this behavior.

I consider that situation pretty dire.

Comment 7 Clay Haapala 2013-11-19 17:46:16 UTC
So, we have a laptop resume problem conflicting with a desktops with RAID failing-to-boot problem?

Removing "rd.md=0" from the kernel command line and remaking grub config is not doing it for me. Is it because initramfs is not being built correctly?

The current error message is that the kernel in the path /vmlinuz-blah cannot be found. That's for the two 3.11.{7,8} kernels I have (re)installed.
The 3.9.5 kernel is still bootable from the same grub menu.

Comment 8 Paulo Fessel 2013-11-19 19:50:06 UTC
You can try making it as it follows:

* Boot your system with 3.9.5 which is booting fine per your accounts.
* Become root and remove rd.md=0 from /etc/default/grub as advised
* grub2-mkconfig -o /boot/grub2/grub.cfg
* dracut -f -v --kver 3.11.8-200.fc19.x86_64
* Reboot and you should be fine

Ever since I removed rd.md=0 and regenerated the ramdisk, I've got no more problems on my installation.

Comment 9 Clay Haapala 2013-11-20 02:18:41 UTC
I hadn't done the dracut step, previously, but the end result is the same: grub complains "You must load the kernel first" or some such, for 3.11.8, and sends me back to the menu, where I am still able to select 3.9.5 and boot.

Might this be because my boot drive is a nVidia BIOS RAID mirror?

Comment 10 Paulo Fessel 2013-11-20 02:48:39 UTC
Try removing rd.dm=0 then and redoing the process. AFAIK your setup is a fakeraid one and thus it needs detecting of the DM array, something that rd.dm=0 disables (see https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html#_speeding_up_the_boot_process). 

IMO if these parameters are meant just to speed up the boot process, it's better to remove them - unless we're talking about servers with dozens of LUNs, which I hardly doubt it's the case here.

Comment 11 Clay Haapala 2013-11-20 03:49:31 UTC
The "rd.md=0" is now removed. I didn't have an "rd.dm=0" field. Here is the original line as got configured by install:

GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora_sampo/root rd.md=0 vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.dm.uuid=nvidia_aaafaffb rd.luks=0 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora_sampo/swap rhgb quiet"

Comment 12 Clay Haapala 2013-11-20 03:56:43 UTC
Here's /etc/default/grub and the output of grub2-mkconfg and dracut. Is the skipping of "device mapper rules" of concern?

[root@sampo ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
#GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora_sampo/root rd.md=0 vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.dm.uuid=nvidia_aaafaffb rd.luks=0 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora_sampo/swap rhgb quiet"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora_sampo/root vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.dm.uuid=nvidia_aaafaffb rd.luks=0 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora_sampo/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
[root@sampo ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.11.8-200.fc19.x86_64
Found initrd image: /boot/initramfs-3.11.8-200.fc19.x86_64.img
Found linux image: /boot/vmlinuz-3.11.7-200.fc19.x86_64
Found initrd image: /boot/initramfs-3.11.7-200.fc19.x86_64.img
Found linux image: /boot/vmlinuz-3.9.5-301.fc19.x86_64
Found initrd image: /boot/initramfs-3.9.5-301.fc19.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-7725dfc225d14958a625ddaaaea5962b
Found initrd image: /boot/initramfs-0-rescue-7725dfc225d14958a625ddaaaea5962b.img
done
[root@sampo ~]# dracut -f -v --kver 3.11.8-200.fc19.x86_64
I: *** Including module: i18n ***
I: *** Including module: drm ***
I: *** Including module: plymouth ***
I: *** Including module: dm ***
I: Skipping udev rule: 64-device-mapper.rules
I: *** Including module: dmraid ***
I: *** Including module: kernel-modules ***
I: *** Including module: lvm ***
I: Skipping udev rule: 64-device-mapper.rules
I: *** Including module: mdraid ***
I: *** Including module: resume ***
I: *** Including module: rootfs-block ***
I: *** Including module: terminfo ***
I: *** Including module: udev-rules ***
I: *** Including module: biosdevname ***
I: *** Including module: systemd ***
I: *** Including module: usrmount ***
I: *** Including module: base ***
I: *** Including module: fs-lib ***
I: *** Including module: shutdown ***
I: *** Including modules done ***
I: *** Installing kernel module dependencies and firmware ***
I: *** Installing kernel module dependencies and firmware done ***
I: *** Resolving executable dependencies ***
I: *** Resolving executable dependencies done***
I: *** Pre-linking files ***
I: *** Pre-linking files done ***
I: *** Hardlinking files ***
I: *** Hardlinking files done ***
I: *** Stripping files ***
I: *** Stripping files done ***
I: *** Creating image file ***
I: *** Creating image file done ***
I: Wrote /boot/initramfs-3.11.8-200.fc19.x86_64.img:
I: -rw-------. 1 root root 10424286 Nov 19 21:54 /boot/initramfs-3.11.8-200.fc19.x86_64.img
[root@sampo ~]#

Comment 13 Fedora End Of Life 2015-01-09 20:31:47 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 14 Fedora End Of Life 2015-05-29 09:43:19 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 15 Fedora End Of Life 2015-06-29 12:51:35 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.