Bug 487358

Summary: rawhide upgrade of F10 fails to boot with "mount: could not find filesystem '/dev/root'"
Product: [Fedora] Fedora Reporter: Clark Williams <williams>
Component: mkinitrdAssignee: Peter Jones <pjones>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: medium    
Version: rawhideCC: aron.griffis, cuviper, hdegoede, katzj, pjones, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-03-16 18:48:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
serial console log of boot failure on Lenovo desktop
none
modified 'init' from failing initrd
none
grub.conf from failing system
none
PATCH fixing booting with older kernels none

Description Clark Williams 2009-02-25 16:34:12 UTC
Created attachment 333172 [details]
serial console log of boot failure on Lenovo desktop

Description of problem:

A system updated from a basic F10 install to rawhide fails to boot.

Did a clean install of F10 from DVD to a Lenovo 3000 J Series desktop box (64-bit kernel). Used all anaconda defaults and system booted fine after the install. Enabled rawhide repository and did a yum update to rawhide. Following successful update of 900-some packages, rebooted and the boot failed with:

    mount: could not find filesystem '/dev/root'



Version-Release number of selected component (if applicable):

mkinitrd-6.0.78-1.fc11.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install F10
2. Update to rawhide
3. reboot the system
  
Actual results:

Boot fails for rawhide kernel; succeeds for F10 kernel


Additional info:

I'll attach a serial console log of the failed boot. This was with a slightly hacked 'init' script from the initrd (commented out setquiet and plymouth references).

I'll attach the 'init' script as well as the grub.conf from the failing system.

Comment 1 Clark Williams 2009-02-25 16:35:21 UTC
Created attachment 333173 [details]
modified 'init' from failing initrd

This is the init script from the failing initrd; I've commented out the 'setquiet' and all references to plymouth, otherwise no modifications

Comment 2 Clark Williams 2009-02-25 16:36:22 UTC
Created attachment 333174 [details]
grub.conf from failing system

This is the grub.conf from the failing system; only modification was to change the timeout parameter from 0 to 10.

Comment 3 Hans de Goede 2009-02-25 18:29:35 UTC
Ok, I think I know whats going on, I'm pretty sure actually. The new mkinitrd depends on some sysfs layout changes only present in the rawhide kernel, so when you do a yum update from F-10, it won't work as then its running 2.6.27.x .

As a workaround (and to prove my theory) can you please boot the working F-10 kernel and then edit /sbin/mkinitrd and changing line 326 from:
    if [ -f "$sysfs/partition" ]; then
Into:
    if [ -f "$sysfs/start" ]; then

And then regenerate your initrd using:

mkinitrd -f /boot/initrd-2.6.29-0.145.rc6.fc11.x86_64.img \
  2.6.29-0.145.rc6.fc11.x86_64

And then reboot into the new kernel? If that works I'll make this change to mkinitrd in rawhide,

Thank you!

Comment 4 Clark Williams 2009-02-25 19:07:03 UTC
That did it for the desktop box! Thanks!

Unfortunately the same edit on my laptop doesn't work. I think that's because I'm running a 2.6.26.8 based kernel there (only one that still works).  When rebuilding an initrd on my laptop, I get the message:

WARNING: /sys/block/dm-0 is a not a block sysfs path, skipping
WARNING: /sys/block/dm-1 is a not a block sysfs path, skipping

and the booted kernel fails with the same 'mount: could not...' error. I noticed that the /sys layout on my bootable kernel (2.6.26.8-rt16) doesn't have /sys/devices/virtual, which is there on 2.6.29 kernels. 

Any ideas for a hack to get my laptop booting fedora kernels again? Wonder if I could boot from an F11 DVD and run mkinitrd there?

Comment 5 Hans de Goede 2009-03-09 08:36:42 UTC
Created attachment 334475 [details]
PATCH fixing booting with older kernels

Sorry for the long silence, I got hit by a wall of (other) work.

Can you give this patch (can be applied directly to /sbin/mkinitrd) a try on the laptop please?

Comment 6 Hans de Goede 2009-03-09 09:37:08 UTC
*** Bug 479602 has been marked as a duplicate of this bug. ***

Comment 7 Clark Williams 2009-03-09 14:30:25 UTC
unfortunately, I reinstalled my laptop (back to F10) so I'm not seeing this problem anymore. I'll probably take it back to rawhide in another week, so I'll keep an eye out for this issue.

Comment 8 Clark Williams 2009-03-10 16:19:27 UTC
My desktop box got into the situation where it was unable to find /dev/root when booting the current rawhide kernel (kernel-2.6.29-0.215.rc7.fc11.x86_64) and the kernel-rt-2.6.24.7-107.el5rt.x86_64 package, so I booted into an earlier kernel, and applied your patch from comment #5 to /sbin/mkinitrd and that seemed to fix the problem. 

At least, I have a working initrd for rawhide. I'm continuing to test on the other installed kernels.

Comment 9 Hans de Goede 2009-03-10 17:26:43 UTC
(In reply to comment #8)
> My desktop box got into the situation where it was unable to find /dev/root
> when booting the current rawhide kernel (kernel-2.6.29-0.215.rc7.fc11.x86_64)
> and the kernel-rt-2.6.24.7-107.el5rt.x86_64 package, so I booted into an
> earlier kernel, and applied your patch from comment #5 to /sbin/mkinitrd and
> that seemed to fix the problem. 
> 
> At least, I have a working initrd for rawhide. I'm continuing to test on the
> other installed kernels.  

Thanks for testing this!

Comment 10 Hans de Goede 2009-03-16 18:48:03 UTC
This is fixed in mkinitrd-6.0.81, which should be in tomorrows rawhide.