Bug 747632 - using iSCSI target as root filesystem results in kernel panic at boot
Summary: using iSCSI target as root filesystem results in kernel panic at boot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 16
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: dracut-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F16Blocker, F16FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2011-10-20 14:49 UTC by Tim Flink
Modified: 2011-10-25 16:10 UTC (History)
7 users (show)

Fixed In Version: dracut-013-16.fc16
Clone Of:
Environment:
Last Closed: 2011-10-25 03:44:24 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
console log from failed boot (23.33 KB, text/plain)
2011-10-20 14:49 UTC, Tim Flink
no flags Details
fstab from system (502 bytes, text/plain)
2011-10-20 15:05 UTC, Tim Flink
no flags Details
grub configuration (2.71 KB, text/plain)
2011-10-20 15:19 UTC, Tim Flink
no flags Details
anaconda log (21.98 KB, text/plain)
2011-10-20 15:20 UTC, Tim Flink
no flags Details
anaconda program.log (94.58 KB, text/plain)
2011-10-20 15:20 UTC, Tim Flink
no flags Details
anaconda storage.log (215.43 KB, text/plain)
2011-10-20 15:21 UTC, Tim Flink
no flags Details
anaconda install.log (10.52 KB, text/plain)
2011-10-20 15:59 UTC, Tim Flink
no flags Details
dracut.log after running failed new-kernel-pkg cmd (85.73 KB, text/plain)
2011-10-20 17:12 UTC, Tim Flink
no flags Details

Description Tim Flink 2011-10-20 14:49:07 UTC
Description of problem:

I did a minimal i386 install with the following partition layout:
Local IDE:
 - 500M /boot
 - 2048 swap

iSCSI Target
 - 20G /

When the system reboots, I get a kernel panic when dracut tries to mount the root filesystem.

Note that this is a regular NIC and does not support iBFT -> /boot is on the local disk.

I have been unable to get into the dracut shell, even when I pass in rd.shell or rd.break as kernel options.

Boot Args:
BOOT_IMAGE=/vmlinuz-3.1.0-0.rc10.git0.1.fc16.i686.PAE root=/dev/sdb1 ro ifname=em1:00:07:e9:5c:e0:30 iscsi_initiator=iqn.1994-05.com.domain:01.0e9b92 rd.md=0 rd.lvm=0 rd.dm=0 ip=em1:dhcp KEYTABLE=us SYSFONT=latarcyrheb-sun16 netroot=iscsi:@192.168.0.5::3260::iqn.2009-02.net.tirfa:for.all rd.luks=0 LANG=en_US.UTF-8 console=ttyS0,38400n8 rd.debug rd.udev.debug


Version-Release number of selected component (if applicable):
dracut-013-15.fc16

How reproducible:
Every time, using LVM or not.

Steps to Reproduce:
1. do minimal install with / on iSCSI, /boot and swap on local disk
2. reboot
  
Actual results:
kernel panic on init

Expected results:
boot into working system

Additional info:

Comment 1 Tim Flink 2011-10-20 14:49:40 UTC
Created attachment 529303 [details]
console log from failed boot

Comment 2 Tim Flink 2011-10-20 15:05:11 UTC
Created attachment 529318 [details]
fstab from system

Comment 3 Tim Flink 2011-10-20 15:07:14 UTC
Proposing as a blocker for Fedora 16 final as it violates the following final release criterion [1]:

The installer must be able to complete an installation using any network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel)

[1] http://fedoraproject.org/wiki/Fedora_16_Final_Release_Criteria

Comment 4 Harald Hoyer 2011-10-20 15:10:09 UTC
[    3.748491] VFS: Cannot open root device "sdb1" or unknown-block(0,0)

This indicates, that no initramfs has been loaded and dracut was not even executed... 

Check your grub configuration!

Comment 5 Tim Flink 2011-10-20 15:19:16 UTC
Created attachment 529322 [details]
grub configuration

This is the grub configuration on that system. There is neither an initramfs on /boot nor a initrd directive in the grub configuration

Comment 6 Tim Flink 2011-10-20 15:20:13 UTC
Created attachment 529324 [details]
anaconda log

Comment 7 Tim Flink 2011-10-20 15:20:43 UTC
Created attachment 529325 [details]
anaconda program.log

Comment 8 Tim Flink 2011-10-20 15:21:12 UTC
Created attachment 529326 [details]
anaconda storage.log

Comment 9 David Lehman 2011-10-20 15:31:01 UTC
Looks to me like everything anaconda was supposed to put in GRUB_CMDLINE_LINUX is there for the iscsi root. Apparently either there is no initrd or else grub2-mkconfig failed to find/recognize it. Tim, can you tell us if there's an initramfs to match your kernel in /boot?

Comment 10 Tim Flink 2011-10-20 15:46:56 UTC
(In reply to comment #9)
> Looks to me like everything anaconda was supposed to put in GRUB_CMDLINE_LINUX
> is there for the iscsi root. Apparently either there is no initrd or else
> grub2-mkconfig failed to find/recognize it. Tim, can you tell us if there's an
> initramfs to match your kernel in /boot?

There are no initramfs files in /boot

Comment 11 Tim Flink 2011-10-20 15:59:18 UTC
Created attachment 529337 [details]
anaconda install.log

Comment 12 David Lehman 2011-10-20 16:02:25 UTC
From install.log:

mkinitrd failed
warning: %posttrans(kernel-PAE-3.1.0-0.rc10.git0.1.fc16.i686) scriptlet failed, exit status 1


Tim, can you reproduce and then run whatever's in the kernel's %posttrans from the command line so we can see what fails?

Comment 13 Tim Flink 2011-10-20 17:02:40 UTC
(In reply to comment #12)
> Tim, can you reproduce and then run whatever's in the kernel's %posttrans from
> the command line so we can see what fails?

The two commands in %posttrans for kernel-PAE are:
/sbin/new-kernel-pkg --package kernel-PAE --mkinitrd --dracut --depmod --update 3.1.0-0.rc10.git0.1.fc16.i686.PAE || exit $?

/sbin/new-kernel-pkg --package kernel-PAE --rpmposttrans 3.1.0-0.rc10.git0.1.fc16.i686.PAE || exit $?


The failure is in the first command:

# /sbin/new-kernel-pkg --package kernel-PAE --mkinitrd --dracut --depmod --update 3.1.0-0.rc10.git0.1.fc16.i686.PAE

F: Failed to install wget
mkinitrd failed

Comment 14 Tim Flink 2011-10-20 17:12:19 UTC
Created attachment 529343 [details]
dracut.log after running failed new-kernel-pkg cmd

Comment 15 Tim Flink 2011-10-20 17:47:43 UTC
I re-ran the kernel %posttrans commands after adding the following to dracut.conf:
omit_dracutmodules+="livenet"

I got an initramfs and the system now boots with the iSCSI target as /.

Re-assigning to dracut.

Comment 16 Tim Flink 2011-10-20 18:13:22 UTC
I tried the patch that wwoods posted:

http://article.gmane.org/gmane.linux.kernel.initramfs/2183

After reverting dracut.conf and verified that the kernel posttrans commands failed again. I applied the patch to /usr/share/dracut/modules.d/90livenet/module-setup.sh, re-ran the kernel posttrans commands and they completed successfully.

Comment 17 Fedora Update System 2011-10-21 10:59:48 UTC
dracut-013-16.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-16.fc16

Comment 18 Adam Williamson 2011-10-21 17:28:40 UTC
Discussed at 2011-10-21 blocker review meeting. Accepted as a blocker per criterion "The installer must be able to complete an installation using any network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel)" - technically the bug only affects minimal installations, but iSCSI is a case where you're probably quite likely to do a minimal install, and it seems a serious enough bug to take as a blocker.

Comment 19 Fedora Update System 2011-10-22 00:10:43 UTC
Package dracut-013-16.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-013-16.fc16'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14704
then log in and leave karma (feedback).

Comment 20 Fedora Update System 2011-10-25 03:44:24 UTC
dracut-013-16.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 21 Tim Flink 2011-10-25 16:10:13 UTC
Verified with an i686 minimal install of Fedora 16 using iSCSI target as /


Note You need to log in before you can comment on or make changes to this bug.