Bug 747632 - using iSCSI target as root filesystem results in kernel panic at boot
using iSCSI target as root filesystem results in kernel panic at boot
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
AcceptedBlocker
:
Depends On:
Blocks: F16Blocker/F16FinalBlocker
  Show dependency treegraph
 
Reported: 2011-10-20 10:49 EDT by Tim Flink
Modified: 2011-10-25 12:10 EDT (History)
7 users (show)

See Also:
Fixed In Version: dracut-013-16.fc16
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-10-24 23:44:24 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
console log from failed boot (23.33 KB, text/plain)
2011-10-20 10:49 EDT, Tim Flink
no flags Details
fstab from system (502 bytes, text/plain)
2011-10-20 11:05 EDT, Tim Flink
no flags Details
grub configuration (2.71 KB, text/plain)
2011-10-20 11:19 EDT, Tim Flink
no flags Details
anaconda log (21.98 KB, text/plain)
2011-10-20 11:20 EDT, Tim Flink
no flags Details
anaconda program.log (94.58 KB, text/plain)
2011-10-20 11:20 EDT, Tim Flink
no flags Details
anaconda storage.log (215.43 KB, text/plain)
2011-10-20 11:21 EDT, Tim Flink
no flags Details
anaconda install.log (10.52 KB, text/plain)
2011-10-20 11:59 EDT, Tim Flink
no flags Details
dracut.log after running failed new-kernel-pkg cmd (85.73 KB, text/plain)
2011-10-20 13:12 EDT, Tim Flink
no flags Details

  None (edit)
Description Tim Flink 2011-10-20 10:49:07 EDT
Description of problem:

I did a minimal i386 install with the following partition layout:
Local IDE:
 - 500M /boot
 - 2048 swap

iSCSI Target
 - 20G /

When the system reboots, I get a kernel panic when dracut tries to mount the root filesystem.

Note that this is a regular NIC and does not support iBFT -> /boot is on the local disk.

I have been unable to get into the dracut shell, even when I pass in rd.shell or rd.break as kernel options.

Boot Args:
BOOT_IMAGE=/vmlinuz-3.1.0-0.rc10.git0.1.fc16.i686.PAE root=/dev/sdb1 ro ifname=em1:00:07:e9:5c:e0:30 iscsi_initiator=iqn.1994-05.com.domain:01.0e9b92 rd.md=0 rd.lvm=0 rd.dm=0 ip=em1:dhcp KEYTABLE=us SYSFONT=latarcyrheb-sun16 netroot=iscsi:@192.168.0.5::3260::iqn.2009-02.net.tirfa:for.all rd.luks=0 LANG=en_US.UTF-8 console=ttyS0,38400n8 rd.debug rd.udev.debug


Version-Release number of selected component (if applicable):
dracut-013-15.fc16

How reproducible:
Every time, using LVM or not.

Steps to Reproduce:
1. do minimal install with / on iSCSI, /boot and swap on local disk
2. reboot
  
Actual results:
kernel panic on init

Expected results:
boot into working system

Additional info:
Comment 1 Tim Flink 2011-10-20 10:49:40 EDT
Created attachment 529303 [details]
console log from failed boot
Comment 2 Tim Flink 2011-10-20 11:05:11 EDT
Created attachment 529318 [details]
fstab from system
Comment 3 Tim Flink 2011-10-20 11:07:14 EDT
Proposing as a blocker for Fedora 16 final as it violates the following final release criterion [1]:

The installer must be able to complete an installation using any network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel)

[1] http://fedoraproject.org/wiki/Fedora_16_Final_Release_Criteria
Comment 4 Harald Hoyer 2011-10-20 11:10:09 EDT
[    3.748491] VFS: Cannot open root device "sdb1" or unknown-block(0,0)

This indicates, that no initramfs has been loaded and dracut was not even executed... 

Check your grub configuration!
Comment 5 Tim Flink 2011-10-20 11:19:16 EDT
Created attachment 529322 [details]
grub configuration

This is the grub configuration on that system. There is neither an initramfs on /boot nor a initrd directive in the grub configuration
Comment 6 Tim Flink 2011-10-20 11:20:13 EDT
Created attachment 529324 [details]
anaconda log
Comment 7 Tim Flink 2011-10-20 11:20:43 EDT
Created attachment 529325 [details]
anaconda program.log
Comment 8 Tim Flink 2011-10-20 11:21:12 EDT
Created attachment 529326 [details]
anaconda storage.log
Comment 9 David Lehman 2011-10-20 11:31:01 EDT
Looks to me like everything anaconda was supposed to put in GRUB_CMDLINE_LINUX is there for the iscsi root. Apparently either there is no initrd or else grub2-mkconfig failed to find/recognize it. Tim, can you tell us if there's an initramfs to match your kernel in /boot?
Comment 10 Tim Flink 2011-10-20 11:46:56 EDT
(In reply to comment #9)
> Looks to me like everything anaconda was supposed to put in GRUB_CMDLINE_LINUX
> is there for the iscsi root. Apparently either there is no initrd or else
> grub2-mkconfig failed to find/recognize it. Tim, can you tell us if there's an
> initramfs to match your kernel in /boot?

There are no initramfs files in /boot
Comment 11 Tim Flink 2011-10-20 11:59:18 EDT
Created attachment 529337 [details]
anaconda install.log
Comment 12 David Lehman 2011-10-20 12:02:25 EDT
From install.log:

mkinitrd failed
warning: %posttrans(kernel-PAE-3.1.0-0.rc10.git0.1.fc16.i686) scriptlet failed, exit status 1


Tim, can you reproduce and then run whatever's in the kernel's %posttrans from the command line so we can see what fails?
Comment 13 Tim Flink 2011-10-20 13:02:40 EDT
(In reply to comment #12)
> Tim, can you reproduce and then run whatever's in the kernel's %posttrans from
> the command line so we can see what fails?

The two commands in %posttrans for kernel-PAE are:
/sbin/new-kernel-pkg --package kernel-PAE --mkinitrd --dracut --depmod --update 3.1.0-0.rc10.git0.1.fc16.i686.PAE || exit $?

/sbin/new-kernel-pkg --package kernel-PAE --rpmposttrans 3.1.0-0.rc10.git0.1.fc16.i686.PAE || exit $?


The failure is in the first command:

# /sbin/new-kernel-pkg --package kernel-PAE --mkinitrd --dracut --depmod --update 3.1.0-0.rc10.git0.1.fc16.i686.PAE

F: Failed to install wget
mkinitrd failed
Comment 14 Tim Flink 2011-10-20 13:12:19 EDT
Created attachment 529343 [details]
dracut.log after running failed new-kernel-pkg cmd
Comment 15 Tim Flink 2011-10-20 13:47:43 EDT
I re-ran the kernel %posttrans commands after adding the following to dracut.conf:
omit_dracutmodules+="livenet"

I got an initramfs and the system now boots with the iSCSI target as /.

Re-assigning to dracut.
Comment 16 Tim Flink 2011-10-20 14:13:22 EDT
I tried the patch that wwoods posted:

http://article.gmane.org/gmane.linux.kernel.initramfs/2183

After reverting dracut.conf and verified that the kernel posttrans commands failed again. I applied the patch to /usr/share/dracut/modules.d/90livenet/module-setup.sh, re-ran the kernel posttrans commands and they completed successfully.
Comment 17 Fedora Update System 2011-10-21 06:59:48 EDT
dracut-013-16.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-16.fc16
Comment 18 Adam Williamson 2011-10-21 13:28:40 EDT
Discussed at 2011-10-21 blocker review meeting. Accepted as a blocker per criterion "The installer must be able to complete an installation using any network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel)" - technically the bug only affects minimal installations, but iSCSI is a case where you're probably quite likely to do a minimal install, and it seems a serious enough bug to take as a blocker.
Comment 19 Fedora Update System 2011-10-21 20:10:43 EDT
Package dracut-013-16.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-013-16.fc16'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14704
then log in and leave karma (feedback).
Comment 20 Fedora Update System 2011-10-24 23:44:24 EDT
dracut-013-16.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 21 Tim Flink 2011-10-25 12:10:13 EDT
Verified with an i686 minimal install of Fedora 16 using iSCSI target as /

Note You need to log in before you can comment on or make changes to this bug.