Bug 729626

Summary: 3.1.0-0.rc1.git1.1 unable to mount root fs on unknown-block(0,0)
Product: [Fedora] Fedora Reporter: Mark Wielaard <mjw>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: aquini, gansalmon, itamar, jonathan, jwboyer, kernel-maint, madhu.chinakonda
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-10 19:32:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
failing boot with 3.1.0-0.rc1.git1.1.fc17.x86_64
none
successful boot with 3.1.0-0.rc0.git21.1.fc17.x86_64 none

Description Mark Wielaard 2011-08-10 12:15:46 UTC
Description of problem:

Unable to boot.

Version-Release number of selected component (if applicable):

kernel-3.1.0-0.rc1.git1.1.fc17.x86_64

How reproducible:

Always.

Steps to Reproduce:
1. Boot into 3.1.0-0.rc1.git1.1 kernel.
  
Actual results:

[    1.785995] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    1.787580] Pid: 1, comm: swapper Not tainted 3.1.0-0.rc1.git1.1.fc17.x86_64 #1
[    1.788991] Call Trace:
[    1.790088]  [<ffffffff814eb11e>] panic+0xa0/0x1b9
[    1.791623]  [<ffffffff81d4f014>] mount_block_root+0x250/0x27b
[    1.792788]  [<ffffffff81d4f092>] mount_root+0x53/0x57
[    1.793730]  [<ffffffff81d4f1d3>] prepare_namespace+0x13d/0x176
[    1.795087]  [<ffffffff81d4ed00>] kernel_init+0x154/0x159
[    1.796225]  [<ffffffff814feee4>] kernel_thread_helper+0x4/0x10
[    1.797350]  [<ffffffff814f6334>] ? retint_restore_args+0x13/0x13
[    1.798496]  [<ffffffff81d4ebac>] ? start_kernel+0x3ea/0x3ea
[    1.799660]  [<ffffffff814feee0>] ? gs_change+0x13/0x13

Expected results:

A booting system.

Additional info:

Selecting a previous kernel in grub (3.1.0-0.rc0.git21.1.fc17.x86_64) works fine.

Comment 1 Josh Boyer 2011-08-10 12:33:41 UTC
Odd.  Can you attached a boot log from the good and bad kernels?  What kind of root device is it?

Comment 2 Mark Wielaard 2011-08-10 13:32:54 UTC
Created attachment 517601 [details]
failing boot with 3.1.0-0.rc1.git1.1.fc17.x86_64

Comment 3 Mark Wielaard 2011-08-10 13:34:02 UTC
Created attachment 517602 [details]
successful boot with 3.1.0-0.rc0.git21.1.fc17.x86_64

Comment 4 Josh Boyer 2011-08-10 13:47:50 UTC
Did you happen to update dracut, udev, or device-mapper when you updated the kernel?

The kernel found the disk fine:

[    1.858218] sd 0:0:0:0: [sda] Attached SCSI disk

but then it sems like dracut is ignoring the rd_NO_MD variable on the command line (or something).

yum.log should have the list of things updated.

Comment 5 Mark Wielaard 2011-08-10 14:05:57 UTC
The update log doesn't show dracut, udev or device-mapper in the last yum upgrade set:

Aug 10 12:25:23 Updated: krb5-libs-1.9.1-9.fc17.x86_64
Aug 10 12:25:31 Updated: rpm-4.9.1.1-2.fc17.x86_64
Aug 10 12:25:35 Updated: rpm-libs-4.9.1.1-2.fc17.x86_64
Aug 10 12:25:40 Updated: rpm-build-libs-4.9.1.1-2.fc17.x86_64
Aug 10 12:26:20 Updated: openssh-5.8p2-19.fc17.x86_64
Aug 10 12:29:17 Updated: kernel-debuginfo-common-x86_64-3.1.0-0.rc1.git1.1.fc17.x86_64
Aug 10 12:29:22 Updated: 1:NetworkManager-glib-0.8.9997-7.git20110721.fc16.x86_64
Aug 10 12:30:10 Updated: 1:NetworkManager-0.8.9997-7.git20110721.fc16.x86_64
Aug 10 12:30:11 Updated: 1:emacs-filesystem-23.3-8.fc17.x86_64
Aug 10 12:31:31 Updated: 1:emacs-common-23.3-8.fc17.x86_64
Aug 10 12:32:13 Updated: binutils-2.21.53.0.2-1.fc17.x86_64
Aug 10 12:32:19 Updated: rpm-build-4.9.1.1-2.fc17.x86_64
Aug 10 12:32:45 Updated: 1:emacs-23.3-8.fc17.x86_64
Aug 10 12:33:11 Updated: 1:NetworkManager-gnome-0.8.9997-7.git20110721.fc16.x86_64
Aug 10 12:33:13 Updated: perf-debuginfo-3.1.0-0.rc1.git1.1.fc17.x86_64
Aug 10 12:38:11 Updated: kernel-debuginfo-3.1.0-0.rc1.git1.1.fc17.x86_64
Aug 10 12:38:24 Updated: openssh-clients-5.8p2-19.fc17.x86_64
Aug 10 12:38:51 Updated: openssh-server-5.8p2-19.fc17.x86_64
Aug 10 12:38:54 Updated: rpm-python-4.9.1.1-2.fc17.x86_64
Aug 10 12:39:05 Updated: rpm-devel-4.9.1.1-2.fc17.x86_64
Aug 10 12:39:08 Updated: redhat-rpm-config-9.1.0-15.fc17.noarch
Aug 10 12:39:17 Updated: krb5-devel-1.9.1-9.fc17.x86_64
Aug 10 12:39:25 Updated: 1:cups-libs-1.5.0-3.fc17.x86_64
Aug 10 12:39:31 Updated: krb5-workstation-1.9.1-9.fc17.x86_64
Aug 10 12:39:57 Updated: libmx-1.3.0-2.fc17.x86_64
Aug 10 12:41:34 Installed: kernel-devel-3.1.0-0.rc1.git1.1.fc17.x86_64
Aug 10 12:41:59 Updated: cdrdao-1.2.3-10.fc17.x86_64
Aug 10 12:42:05 Updated: seed-3.1.1-2.fc17.x86_64
Aug 10 12:42:41 Updated: kernel-headers-3.1.0-0.rc1.git1.1.fc17.x86_64
Aug 10 12:42:43 Updated: libtalloc-2.0.6-1.fc17.x86_64
Aug 10 12:42:46 Updated: binutils-devel-2.21.53.0.2-1.fc17.x86_64
Aug 10 12:42:49 Updated: ibus-m17n-1.3.2-10.fc17.x86_64
Aug 10 12:42:52 Updated: usb_modeswitch-1.1.9-2.fc17.x86_64
Aug 10 12:42:54 Updated: logrotate-3.8.0-5.fc17.x86_64
Aug 10 12:43:29 Updated: perf-3.1.0-0.rc1.git1.1.fc17.x86_64
Aug 10 12:44:25 Installed: kernel-3.1.0-0.rc1.git1.1.fc17.x86_64

The last updates of udev, dracut and device-mapper were:
Aug 04 12:14:03 Updated: udev-173-1.fc17.x86_64
Aug 04 12:16:49 Updated: dracut-011-15.git20110720.noarch
Aug 04 15:27:21 Updated: device-mapper-event-1.02.65-5.fc17.x86_64

The previous (working) kernel was installed after those were last upgraded:
Aug 06 20:36:19 Installed: kernel-3.1.0-0.rc0.git21.1.fc17.x86_64

Comment 6 Josh Boyer 2011-08-10 14:32:09 UTC
OK, I looked harder at the logs.  Sorry for missing this.

The good load shows the kernel unpacking the initramfs:

[    1.150817] PCI: CLS 0 bytes, default 64
[    1.154315] Unpacking initramfs...
[    1.978895] Freeing initrd memory: 15476k freed

The "bad" kernel never seems to even see an initrafms to unpack.  There's nothing in the logs at all about it.  That would explain why the kernel started searching for MD devices, since it didn't do the switch to the initramfs.

Does the grub entry for kernel-3.1.0-0.rc1.git1.1.fc17.x86_64 have an initrd line?  If so, is the initramfs file present in /boot, and does it's size look comparable to the good kernel?

Comment 7 Mark Wielaard 2011-08-10 15:25:12 UTC
You are right, there is neither an initramfs file present in /boot for this kernel version, nor is there an initrd line in grub.conf. I have no idea how that happened.

How does one "regenerate" a missing initrmfs file?

Comment 8 Josh Boyer 2011-08-10 18:59:47 UTC
Either yum remove/install the kernel again, or run this as root:

/sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 3.1.0-0.rc1.git1.1.fc17.x86_64

In either case, make sure the grub.conf is updated as well.

Comment 9 Mark Wielaard 2011-08-10 19:32:07 UTC
I did a yum remove kernel-3.1.0-0.rc1.git1.1.fc17.x86_64 && yum install kernel-3.1.0-0.rc1.git1.1.fc17.x86_64 which created the initramfs under /boot and the correct initrd line in grub.conf. Booting into 3.1.0-0.rc1.git1.1 went fine after that.

Lets assume I missed some weird install error the last time and mark this as invalid. Sorry about the noise, and thanks for the help.