Red Hat Bugzilla – Bug 826689
All kernel-3.5.0-0* on F18 fails to boot
Last modified: 2012-06-07 04:30:04 EDT
Description of problem: All kernels fail to boot on F18 Unable to mount root fs on unkown-block Fails to load initial ramfs
Is this on i686 or x86_64? Do you have an initramfs line in the grub2.cfg file for the kernels?
Created attachment 587824 [details] 3.5.0.0.rc0.git8.1
Created attachment 587825 [details] 3.5.0-0.rc0.git7.1
Same problem on x86_64 running on Thinkpad X61s (see attachments)
Somebody please remove 'rhgb quiet' from the command line and capture as much of the kernel boot as possible. Also, please verify you have an initramfs line in the grub2.cfg file.
Created attachment 587853 [details] Here is my grub2.cfg
x86_64 on thinkpad x220
(In reply to comment #6) > Created attachment 587853 [details] > Here is my grub2.cfg You have no initrd line for the 3.5 kernels at all. Do the initramfs files that correspond with the 3.5 kernels exist in /boot?
# grep initramfs -R /boot/* Binary file /boot/extlinux/lua.c32 matches /boot/grub2/grub.cfg: initrd /initramfs-3.4.0-2.fc18.x86_64.img /boot/System.map-3.4.0-2.fc18.x86_64:ffffffff82038058 T __initramfs_start /boot/System.map-3.4.0-2.fc18.x86_64:ffffffff82038258 T __initramfs_size /boot/System.map-3.5.0-0.rc0.git7.1.fc18.x86_64:ffffffff8203b040 T __initramfs_start /boot/System.map-3.5.0-0.rc0.git7.1.fc18.x86_64:ffffffff8203b240 T __initramfs_size /boot/System.map-3.5.0-0.rc0.git8.1.fc18.x86_64:ffffffff8203b060 T __initramfs_start /boot/System.map-3.5.0-0.rc0.git8.1.fc18.x86_64:ffffffff8203b260 T __initramfs_size # grep initramfs /etc/grub2.cfg initrd /initramfs-3.4.0-2.fc18.x86_64.img
Autorun DONE VFS: Cannot open root device "mapper/vg_dhcp189250-lv_root" on unknown-block(0.0): error -6 Please append a correct "root=" boot options: here are the available partitions: 0800 156290904 sda driver: sd 08001 512000 sda1 0000... 08002 144777024 sda2 0000... Kernel Panic ... Pid: 1, comm: swapper/0 Not tainted 3.5.0-0 ... Call Trace .... mount_block_root_0x1d9/0x28a
Total size: 27 M Installed size: 119 M Downloading Packages: Running Transaction Check Running Transaction Test Transaction Test Succeeded Running Transaction Installing : kernel-3.5.0-0.rc0.git8.1.fc18.x86_64 1/1 Non-fatal POSTTRANS scriptlet failure in rpm package kernel-3.5.0-0.rc0.git8.1.fc18.x86_64 Verifying : kernel-3.5.0-0.rc0.git8.1.fc18.x86_64 1/1 Installed: kernel.x86_64 0:3.5.0-0.rc0.git8.1.fc18 Might be related to this.
Warning: RPMDB altered outside of yum. Updating : perf-3.5.0-0.rc0.git9.1.fc18.x86_64 1/4 Installing : kernel-devel-3.5.0-0.rc0.git9.1.fc18.x86_64 2/4 Installing : kernel-3.5.0-0.rc0.git9.1.fc18.x86_64 3/4 Cleanup : perf-3.5.0-0.rc0.git8.1.fc18.x86_64 4/4 Non-fatal POSTTRANS scriptlet failure in rpm package kernel-3.5.0-0.rc0.git9.1.fc18.x86_64 Verifying : kernel-devel-3.5.0-0.rc0.git9.1.fc18.x86_64 1/4 Verifying : kernel-3.5.0-0.rc0.git9.1.fc18.x86_64 2/4 Verifying : perf-3.5.0-0.rc0.git9.1.fc18.x86_64 3/4 Verifying : perf-3.5.0-0.rc0.git8.1.fc18.x86_64 4/4 Installed: kernel.x86_64 0:3.5.0-0.rc0.git9.1.fc18 kernel-devel.x86_64 0:3.5.0-0.rc0.git9.1.fc18 Updated: perf.x86_64 0:3.5.0-0.rc0.git9.1.fc18 Looks familiar. grep initrd /etc/grub2.cfg initrd /initramfs-3.4.0-2.fc18.x86_64.img initrd /initramfs-3.4.0-0.rc7.git0.1.fc18.x86_64.debug.img
(In reply to comment #9) > # grep initramfs -R /boot/* That isn't really what I was asking. 'ls /boot/initramfs*' would have been more appropriate. > # grep initramfs /etc/grub2.cfg > initrd /initramfs-3.4.0-2.fc18.x86_64.img That's helpful though. You're also lacking the initrd line for any of the 3.5 kernel.
(In reply to comment #12) > Warning: RPMDB altered outside of yum. > Updating : perf-3.5.0-0.rc0.git9.1.fc18.x86_64 > 1/4 > Installing : kernel-devel-3.5.0-0.rc0.git9.1.fc18.x86_64 > 2/4 > Installing : kernel-3.5.0-0.rc0.git9.1.fc18.x86_64 > 3/4 > Cleanup : perf-3.5.0-0.rc0.git8.1.fc18.x86_64 > 4/4 > Non-fatal POSTTRANS scriptlet failure in rpm package > kernel-3.5.0-0.rc0.git9.1.fc18.x86_64 OK, yeah. So the kernel's posttrans script is basically calling new-kernel-pkg, which is what creates the initramfs for that kernel and then updates the grub config file. This is either something in dracut dying, or in grubby. I'm going to guess dracut for now. Can you run: 'dracut /boot/initramfs-3.5.0-0.rc0.git9.1.fc18.x86_64.img \ 3.5.0-0.rc0.git9.1.fc18.x86_64' and see if it works.
I am seeing this with the 3.5.0-0.rc0.git9.2.fc18.i686.PAE kernel. The initramfs file is missing. I'll report on whether running dracut manually worked shortly.
dracut failed: [root@bruno bruno]# dracut /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.i686.PAE.img 3.5.0-0.rc0.git9.2.fc18.i686.PAE F: installkernel failed in module multipath
Removed dm-mpath-pkgs to get dracut working but boot fails [root@localhost ~]# dracut /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.img 3.5.0-0.rc0.git9.2.fc18.x86_64 F: installkernel failed in module multipath [root@localhost ~]# rpm -qa | grep multipath device-mapper-multipath-0.4.9-22.fc18.x86_64 device-mapper-multipath-libs-0.4.9-22.fc18.x86_64 [root@localhost ~]# yum erase device-mapper-multipath* <omitted for brevity> [root@localhost ~]# dracut /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.img 3.5.0-0.rc0.git9.2.fc18.x86_64 [root@localhost ~]# ls /boot/initramfs* /boot/initramfs-3.4.0-2.fc18.x86_64.img /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.img
I notice that all the other kernel-3.5.0-0* show this in /var/log/dracut.log-* [root@localhost ~]# tail -5 /var/log/dracut.log-20120526 D: Installing /lib/udev/rules.d/40-multipath.rules I: Skipping program $env{MPATH_SBIN_PATH}/multipath using in udev rule 40-multipath.rules as it cannot be found D: Installing /lib/modules/3.5.0-0.rc0.git7.1.fc18.x86_64/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko D: Installing /lib/modules/3.5.0-0.rc0.git7.1.fc18.x86_64/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko F: installkernel failed in module multipath
OK. I'm going to move this over to device-mapper then and put Harald on CC for dracut. Seems there some incompatible thing between them.
+1 on the dracut/multipath problem. I see it too. HOWEVER, on another note, since uninstalling the multipatch rpm's and reinstalling the latest 3.5.0 rpm from rawhide, my boot fails (machine hangs) when the nouveau driver get's loaded. I see the console output happening as the boot is taking place and then, as soon as nouveau get's loaded, BAM, machine is hung. Screen goes black (I assume it's trying to switch over to the default fonts for the nouveau driver), no keyboard response, just a blinking light on the caps-lock key. Have to power down and up. JUst thought I'd let you know.
This fails even with kernels < 3.5. I give it a try with 3.4 from rawhide. [0] rawhide/~ # uname -a Linux rawhide.localdomain 3.4.0-0.rc7.git6.1.fc18.x86_64 #1 SMP Sun May 20 12:59:46 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux [0] rawhide/~ # rpm -q kernel kernel-3.4.0-0.rc7.git6.1.fc18.x86_64 [0] rawhide/~ # dracut test F: installkernel failed in module multipath [0] rawhide/~ # echo $? 1 ---- And now with the "--debug" option: [0] rawhide/~ # dracut --debug test <debug stuff printed> [0] rawhide/~ # echo $? 0 And finished successfuly, creating the initramfs. Looking at the /usr/lib/dracut/modules.d/90multipath/module-setup.sh and the "installkernel" fn: installkernel() { ... ... ... [[ $debug ]] && set -x } And that's all that is at the end of the fn! So it seems like there's improper handling of the return value. (The return value of the whole fn is the return value of the last statement which in this case ends up with a failure if debug option is not used.) Reassigning to dracut...
Hmm, that's interesting. All of my 3.4 kernels from rawhide had initramfs files built just fine and were able to be booted with no issues. Only 3.5 was having a problem.
(In reply to comment #22) > Hmm, that's interesting. All of my 3.4 kernels from rawhide had initramfs > files built just fine and were able to be booted with no issues. Only 3.5 > was having a problem. If you updated dracut before or concurrently with the 3.5 kernel, then that isn't really all that interesting.
Dracut tries to add "multipath" binary, as specified in /lib/udev/rules.d/40-multipath.rules. The problem is: dracut don't expand variables from this udev rule. This is from strace of dracut: faccessat(AT_FDCWD, "/lib/udev/$env{MPATH_SBIN_PATH}/multipath", X_OK) = -1 ENOENT (No such file or directory) Also, MPATH_SBIN_PATH is absolute path, not relative to /lib/udev. With this, dracut's multipath module fails and initramfs is not created.
(In reply to comment #24) > Dracut tries to add "multipath" binary, as specified in > /lib/udev/rules.d/40-multipath.rules. The problem is: dracut don't expand > variables from this udev rule. This is from strace of dracut: > > faccessat(AT_FDCWD, "/lib/udev/$env{MPATH_SBIN_PATH}/multipath", X_OK) = -1 > ENOENT (No such file or directory) > > Also, MPATH_SBIN_PATH is absolute path, not relative to /lib/udev. > With this, dracut's multipath module fails and initramfs is not created. I will fix dracut, but MPATH_SBIN_PATH is crazy stuff... we don't need that!
(In reply to comment #24) > Dracut tries to add "multipath" binary, as specified in > /lib/udev/rules.d/40-multipath.rules. The problem is: dracut don't expand > variables from this udev rule. This is from strace of dracut: > > faccessat(AT_FDCWD, "/lib/udev/$env{MPATH_SBIN_PATH}/multipath", X_OK) = -1 > ENOENT (No such file or directory) > > Also, MPATH_SBIN_PATH is absolute path, not relative to /lib/udev. > With this, dracut's multipath module fails and initramfs is not created. I: Skipping program $env{MPATH_SBIN_PATH}/multipath using in udev rule 40-multipath.rules as it cannot be found multipath and multipathd are installed previously explicitly
*** Bug 827734 has been marked as a duplicate of this bug. ***
(In reply to comment #25) > (In reply to comment #24) > I will fix dracut, but MPATH_SBIN_PATH is crazy stuff... we don't need that! Actually, AFAIK, we still do need that for anaconda to use these rules. LVM does the same thing out of necessity. And isn't it perfectly valid to use an absolute pathname in a udev RUN command?
(In reply to comment #28) > (In reply to comment #25) > > (In reply to comment #24) > > I will fix dracut, but MPATH_SBIN_PATH is crazy stuff... we don't need that! > > Actually, AFAIK, we still do need that for anaconda to use these rules. LVM > does the same thing out of necessity. And isn't it perfectly valid to use > an absolute pathname in a udev RUN command? I don't think anaconda puts these binaries in different places nowadays. Absolute pathnames are fine and actually required in the RUN command.
Removing device-mapper-multipath from my system and then running the kernel update script worked to get an initramfs built.
*** Bug 824981 has been marked as a duplicate of this bug. ***
Is this a regression? So it looks to me like device-mapper-multipath gets pulled in from live media desktop edition installs because we leave anaconda installed (which is a bug, but...). Suggesting that people remove it can't be the fix. The last change to d-m-m is: commit 5fe519f23fd8007c3fac68b50ea4bb764dac4ba9 Author: Benjamin Marzinski <bmarzins@redhat.com> Date: Fri Feb 10 09:34:20 2012 -0600 Add back missing patches So something else in the interim changed. What was it?
can you try again with dracut-019 in rawhide?
(In reply to comment #33) > can you try again with dracut-019 in rawhide? Ok, so: * Using yum to install dracut-019 from koji * Rebuilding the initramfs manually for the new kernel * Running grub2-mkconfig > /boot/grub2/grub.cfg does appear to fix it.
After today's rawhide update, running the kernel update script produced a new initramfs file (even with device-mapper-multipath reinstalld), so it looks like this issue is fixed.