This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 826689 - All kernel-3.5.0-0* on F18 fails to boot
All kernel-3.5.0-0* on F18 fails to boot
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
:
: 824981 827734 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-30 14:57 EDT by Daniel Walsh
Modified: 2012-06-07 04:30 EDT (History)
25 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-07 04:30:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
3.5.0.0.rc0.git8.1 (1.06 MB, image/jpeg)
2012-05-30 15:22 EDT, Simon Sekidde
no flags Details
3.5.0-0.rc0.git7.1 (1.05 MB, image/jpeg)
2012-05-30 15:22 EDT, Simon Sekidde
no flags Details
Here is my grub2.cfg (4.55 KB, text/plain)
2012-05-30 16:50 EDT, Daniel Walsh
no flags Details

  None (edit)
Description Daniel Walsh 2012-05-30 14:57:22 EDT
Description of problem:

All kernels fail to boot on F18

Unable to mount root fs on unkown-block
Fails to load initial ramfs
Comment 1 Josh Boyer 2012-05-30 15:03:39 EDT
Is this on i686 or x86_64?  Do you have an initramfs line in the grub2.cfg file for the kernels?
Comment 2 Simon Sekidde 2012-05-30 15:22:05 EDT
Created attachment 587824 [details]
3.5.0.0.rc0.git8.1
Comment 3 Simon Sekidde 2012-05-30 15:22:44 EDT
Created attachment 587825 [details]
3.5.0-0.rc0.git7.1
Comment 4 Simon Sekidde 2012-05-30 15:23:29 EDT
Same problem on x86_64 running on Thinkpad X61s (see attachments)
Comment 5 Josh Boyer 2012-05-30 15:46:08 EDT
Somebody please remove 'rhgb quiet' from the command line and capture as much of the kernel boot as possible.  Also, please verify you have an initramfs line in the grub2.cfg file.
Comment 6 Daniel Walsh 2012-05-30 16:50:17 EDT
Created attachment 587853 [details]
Here is my grub2.cfg
Comment 7 Daniel Walsh 2012-05-30 16:51:31 EDT
x86_64 on thinkpad x220
Comment 8 Josh Boyer 2012-05-30 16:52:35 EDT
(In reply to comment #6)
> Created attachment 587853 [details]
> Here is my grub2.cfg

You have no initrd line for the 3.5 kernels at all.  Do the initramfs files that correspond with the 3.5 kernels exist in /boot?
Comment 9 Simon Sekidde 2012-05-30 16:55:07 EDT
# grep initramfs -R /boot/*
Binary file /boot/extlinux/lua.c32 matches
/boot/grub2/grub.cfg:	initrd /initramfs-3.4.0-2.fc18.x86_64.img
/boot/System.map-3.4.0-2.fc18.x86_64:ffffffff82038058 T __initramfs_start
/boot/System.map-3.4.0-2.fc18.x86_64:ffffffff82038258 T __initramfs_size
/boot/System.map-3.5.0-0.rc0.git7.1.fc18.x86_64:ffffffff8203b040 T __initramfs_start
/boot/System.map-3.5.0-0.rc0.git7.1.fc18.x86_64:ffffffff8203b240 T __initramfs_size
/boot/System.map-3.5.0-0.rc0.git8.1.fc18.x86_64:ffffffff8203b060 T __initramfs_start
/boot/System.map-3.5.0-0.rc0.git8.1.fc18.x86_64:ffffffff8203b260 T __initramfs_size

# grep initramfs /etc/grub2.cfg 
	initrd /initramfs-3.4.0-2.fc18.x86_64.img
Comment 10 Daniel Walsh 2012-05-30 17:00:01 EDT
Autorun DONE
VFS: Cannot open root device "mapper/vg_dhcp189250-lv_root" on unknown-block(0.0): error -6
Please append a correct "root=" boot options: here are the available partitions:
0800 156290904 sda driver: sd
08001 512000 sda1 0000...
08002 144777024 sda2 0000...

Kernel Panic ...
Pid: 1, comm: swapper/0 Not tainted 3.5.0-0 ...

Call Trace

....

mount_block_root_0x1d9/0x28a
Comment 11 Daniel Walsh 2012-05-30 17:15:14 EDT
Total size: 27 M
Installed size: 119 M
Downloading Packages:
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : kernel-3.5.0-0.rc0.git8.1.fc18.x86_64                        1/1 
Non-fatal POSTTRANS scriptlet failure in rpm package kernel-3.5.0-0.rc0.git8.1.fc18.x86_64
  Verifying  : kernel-3.5.0-0.rc0.git8.1.fc18.x86_64                        1/1 

Installed:
  kernel.x86_64 0:3.5.0-0.rc0.git8.1.fc18    

Might be related to this.
Comment 12 Daniel Walsh 2012-05-30 17:22:42 EDT
Warning: RPMDB altered outside of yum.
  Updating   : perf-3.5.0-0.rc0.git9.1.fc18.x86_64                          1/4 
  Installing : kernel-devel-3.5.0-0.rc0.git9.1.fc18.x86_64                  2/4 
  Installing : kernel-3.5.0-0.rc0.git9.1.fc18.x86_64                        3/4 
  Cleanup    : perf-3.5.0-0.rc0.git8.1.fc18.x86_64                          4/4 
Non-fatal POSTTRANS scriptlet failure in rpm package kernel-3.5.0-0.rc0.git9.1.fc18.x86_64
  Verifying  : kernel-devel-3.5.0-0.rc0.git9.1.fc18.x86_64                  1/4 
  Verifying  : kernel-3.5.0-0.rc0.git9.1.fc18.x86_64                        2/4 
  Verifying  : perf-3.5.0-0.rc0.git9.1.fc18.x86_64                          3/4 
  Verifying  : perf-3.5.0-0.rc0.git8.1.fc18.x86_64                          4/4 

Installed:
  kernel.x86_64 0:3.5.0-0.rc0.git9.1.fc18                                       
  kernel-devel.x86_64 0:3.5.0-0.rc0.git9.1.fc18                                 

Updated:
  perf.x86_64 0:3.5.0-0.rc0.git9.1.fc18  


Looks familiar.
 grep initrd /etc/grub2.cfg 
	initrd /initramfs-3.4.0-2.fc18.x86_64.img
	initrd /initramfs-3.4.0-0.rc7.git0.1.fc18.x86_64.debug.img
Comment 13 Josh Boyer 2012-05-30 17:45:56 EDT
(In reply to comment #9)
> # grep initramfs -R /boot/*

That isn't really what I was asking.  'ls /boot/initramfs*' would have been more appropriate.

> # grep initramfs /etc/grub2.cfg 
> 	initrd /initramfs-3.4.0-2.fc18.x86_64.img

That's helpful though.  You're also lacking the initrd line for any of the 3.5 kernel.
Comment 14 Josh Boyer 2012-05-30 17:49:46 EDT
(In reply to comment #12)
> Warning: RPMDB altered outside of yum.
>   Updating   : perf-3.5.0-0.rc0.git9.1.fc18.x86_64                         
> 1/4 
>   Installing : kernel-devel-3.5.0-0.rc0.git9.1.fc18.x86_64                 
> 2/4 
>   Installing : kernel-3.5.0-0.rc0.git9.1.fc18.x86_64                       
> 3/4 
>   Cleanup    : perf-3.5.0-0.rc0.git8.1.fc18.x86_64                         
> 4/4 
> Non-fatal POSTTRANS scriptlet failure in rpm package
> kernel-3.5.0-0.rc0.git9.1.fc18.x86_64

OK, yeah.  So the kernel's posttrans script is basically calling new-kernel-pkg, which is what creates the initramfs for that kernel and then updates the grub config file.

This is either something in dracut dying, or in grubby.  I'm going to guess dracut for now.

Can you run:

'dracut /boot/initramfs-3.5.0-0.rc0.git9.1.fc18.x86_64.img \ 
3.5.0-0.rc0.git9.1.fc18.x86_64'

and see if it works.
Comment 15 Bruno Wolff III 2012-05-31 01:54:57 EDT
I am seeing this with the 3.5.0-0.rc0.git9.2.fc18.i686.PAE kernel. The initramfs file is missing. I'll report on whether running dracut manually worked shortly.
Comment 16 Bruno Wolff III 2012-05-31 01:56:30 EDT
dracut failed:
[root@bruno bruno]# dracut /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.i686.PAE.img 3.5.0-0.rc0.git9.2.fc18.i686.PAE
F: installkernel failed in module multipath
Comment 17 Simon Sekidde 2012-05-31 07:17:54 EDT
Removed dm-mpath-pkgs to get dracut working but boot fails

[root@localhost ~]# dracut /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.img 3.5.0-0.rc0.git9.2.fc18.x86_64
F: installkernel failed in module multipath
[root@localhost ~]# rpm -qa | grep multipath
device-mapper-multipath-0.4.9-22.fc18.x86_64
device-mapper-multipath-libs-0.4.9-22.fc18.x86_64

[root@localhost ~]# yum erase device-mapper-multipath*
<omitted for brevity>

[root@localhost ~]# dracut /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.img 3.5.0-0.rc0.git9.2.fc18.x86_64

[root@localhost ~]# ls /boot/initramfs*
/boot/initramfs-3.4.0-2.fc18.x86_64.img  /boot/initramfs-3.5.0-0.rc0.git9.2.fc18.img
Comment 18 Simon Sekidde 2012-05-31 07:31:31 EDT
I notice that all the other kernel-3.5.0-0* show this in /var/log/dracut.log-*

[root@localhost ~]# tail -5 /var/log/dracut.log-20120526
D: Installing /lib/udev/rules.d/40-multipath.rules
I: Skipping program $env{MPATH_SBIN_PATH}/multipath using in udev rule
40-multipath.rules as it cannot be found
D: Installing /lib/modules/3.5.0-0.rc0.git7.1.fc18.x86_64/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko
D: Installing /lib/modules/3.5.0-0.rc0.git7.1.fc18.x86_64/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko
F: installkernel failed in module multipath
Comment 19 Josh Boyer 2012-05-31 09:55:14 EDT
OK.  I'm going to move this over to device-mapper then and put Harald on CC for dracut.  Seems there some incompatible thing between them.
Comment 20 kevin martin 2012-05-31 16:32:54 EDT
+1 on the dracut/multipath problem.  I see it too.  HOWEVER, on another note, since uninstalling the multipatch rpm's and reinstalling the latest 3.5.0 rpm from rawhide, my boot fails (machine hangs) when the nouveau driver get's loaded.  I see the console output happening as the boot is taking place and then, as soon as nouveau get's loaded, BAM, machine is hung.  Screen goes black (I assume it's trying to switch over to the default fonts for the nouveau driver), no keyboard response, just a blinking light on the caps-lock key.  Have to power down and up.  JUst thought I'd let you know.
Comment 21 Peter Rajnoha 2012-06-01 10:04:05 EDT
This fails even with kernels < 3.5. I give it a try with 3.4 from rawhide.

[0] rawhide/~ # uname -a
Linux rawhide.localdomain 3.4.0-0.rc7.git6.1.fc18.x86_64 #1 SMP Sun May 20 12:59:46 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[0] rawhide/~ # rpm -q kernel
kernel-3.4.0-0.rc7.git6.1.fc18.x86_64

[0] rawhide/~ # dracut test
F: installkernel failed in module multipath

[0] rawhide/~ # echo $?            
1

----

And now with the "--debug" option:

[0] rawhide/~ # dracut --debug test
<debug stuff printed>

[0] rawhide/~ # echo $?
0

And finished successfuly, creating the initramfs. Looking at the /usr/lib/dracut/modules.d/90multipath/module-setup.sh and the "installkernel" fn:

installkernel() {
 ...
 ...
 ...
 [[ $debug ]] && set -x
}

And that's all that is at the end of the fn! So it seems like there's improper handling of the return value. (The return value of the whole fn is the return value of the last statement which in this case ends up with a failure if debug option is not used.)

Reassigning to dracut...
Comment 22 kevin martin 2012-06-01 11:12:29 EDT
Hmm, that's interesting.  All of my 3.4 kernels from rawhide had initramfs files built just fine and were able to be booted with no issues.  Only 3.5 was having a problem.
Comment 23 Josh Boyer 2012-06-03 16:26:19 EDT
(In reply to comment #22)
> Hmm, that's interesting.  All of my 3.4 kernels from rawhide had initramfs
> files built just fine and were able to be booted with no issues.  Only 3.5
> was having a problem.

If you updated dracut before or concurrently with the 3.5 kernel, then that isn't really all that interesting.
Comment 24 Tomasz Torcz 2012-06-04 05:59:41 EDT
Dracut tries to add "multipath" binary, as specified in /lib/udev/rules.d/40-multipath.rules. The problem is: dracut don't expand variables from this udev rule. This is from strace of dracut:

faccessat(AT_FDCWD, "/lib/udev/$env{MPATH_SBIN_PATH}/multipath", X_OK) = -1 ENOENT (No such file or directory)

Also, MPATH_SBIN_PATH is absolute path, not relative to /lib/udev.
With this, dracut's multipath module fails and initramfs is not created.
Comment 25 Harald Hoyer 2012-06-04 06:38:28 EDT
(In reply to comment #24)
> Dracut tries to add "multipath" binary, as specified in
> /lib/udev/rules.d/40-multipath.rules. The problem is: dracut don't expand
> variables from this udev rule. This is from strace of dracut:
> 
> faccessat(AT_FDCWD, "/lib/udev/$env{MPATH_SBIN_PATH}/multipath", X_OK) = -1
> ENOENT (No such file or directory)
> 
> Also, MPATH_SBIN_PATH is absolute path, not relative to /lib/udev.
> With this, dracut's multipath module fails and initramfs is not created.

I will fix dracut, but MPATH_SBIN_PATH is crazy stuff... we don't need that!
Comment 26 Harald Hoyer 2012-06-04 06:50:21 EDT
(In reply to comment #24)
> Dracut tries to add "multipath" binary, as specified in
> /lib/udev/rules.d/40-multipath.rules. The problem is: dracut don't expand
> variables from this udev rule. This is from strace of dracut:
> 
> faccessat(AT_FDCWD, "/lib/udev/$env{MPATH_SBIN_PATH}/multipath", X_OK) = -1
> ENOENT (No such file or directory)
> 
> Also, MPATH_SBIN_PATH is absolute path, not relative to /lib/udev.
> With this, dracut's multipath module fails and initramfs is not created.

I: Skipping program $env{MPATH_SBIN_PATH}/multipath using in udev rule 40-multipath.rules as it cannot be found


multipath and multipathd are installed previously explicitly
Comment 27 Paulo Andrade 2012-06-04 14:06:36 EDT
*** Bug 827734 has been marked as a duplicate of this bug. ***
Comment 28 Ben Marzinski 2012-06-04 14:13:44 EDT
(In reply to comment #25)
> (In reply to comment #24)
> I will fix dracut, but MPATH_SBIN_PATH is crazy stuff... we don't need that!

Actually, AFAIK, we still do need that for anaconda to use these rules. LVM does the same thing out of necessity.  And isn't it perfectly valid to use an absolute pathname in a udev RUN command?
Comment 29 Harald Hoyer 2012-06-05 05:18:22 EDT
(In reply to comment #28)
> (In reply to comment #25)
> > (In reply to comment #24)
> > I will fix dracut, but MPATH_SBIN_PATH is crazy stuff... we don't need that!
> 
> Actually, AFAIK, we still do need that for anaconda to use these rules. LVM
> does the same thing out of necessity.  And isn't it perfectly valid to use
> an absolute pathname in a udev RUN command?

I don't think anaconda puts these binaries in different places nowadays. Absolute pathnames are fine and actually required in the RUN command.
Comment 30 Bruno Wolff III 2012-06-05 11:22:14 EDT
Removing device-mapper-multipath from my system and then running the kernel update script worked to get an initramfs built.
Comment 31 Frank Murphy 2012-06-05 12:15:03 EDT
*** Bug 824981 has been marked as a duplicate of this bug. ***
Comment 32 Colin Walters 2012-06-05 13:41:06 EDT
Is this a regression?  

So it looks to me like device-mapper-multipath gets pulled in from live media desktop edition installs because we leave anaconda installed (which is a bug, but...).

Suggesting that people remove it can't be the fix.  

The last change to d-m-m is:

commit 5fe519f23fd8007c3fac68b50ea4bb764dac4ba9
Author: Benjamin Marzinski <bmarzins@redhat.com>
Date:   Fri Feb 10 09:34:20 2012 -0600

    Add back missing patches

So something else in the interim changed.  What was it?
Comment 33 Harald Hoyer 2012-06-05 14:07:21 EDT
can you try again with dracut-019 in rawhide?
Comment 34 Colin Walters 2012-06-05 14:50:52 EDT
(In reply to comment #33)
> can you try again with dracut-019 in rawhide?

Ok, so:

* Using yum to install dracut-019 from koji
* Rebuilding the initramfs manually for the new kernel
* Running grub2-mkconfig > /boot/grub2/grub.cfg

does appear to fix it.
Comment 35 Bruno Wolff III 2012-06-06 14:18:12 EDT
After today's rawhide update, running the kernel update script produced a new initramfs file (even with device-mapper-multipath reinstalld), so it looks like this issue is fixed.

Note You need to log in before you can comment on or make changes to this bug.