Bug 747632

Summary: using iSCSI target as root filesystem results in kernel panic at boot
Product: [Fedora] Fedora Reporter: Tim Flink <tflink>
Component: dracutAssignee: dracut-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16CC: anaconda-maint-list, awilliam, dracut-maint, harald, jonathan, kay, vanmeeuwen+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedBlocker
Fixed In Version: dracut-013-16.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-24 23:44:24 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 713568    
Attachments:
Description Flags
console log from failed boot
none
fstab from system
none
grub configuration
none
anaconda log
none
anaconda program.log
none
anaconda storage.log
none
anaconda install.log
none
dracut.log after running failed new-kernel-pkg cmd none

Description Tim Flink 2011-10-20 10:49:07 EDT
Description of problem:

I did a minimal i386 install with the following partition layout:
Local IDE:
 - 500M /boot
 - 2048 swap

iSCSI Target
 - 20G /

When the system reboots, I get a kernel panic when dracut tries to mount the root filesystem.

Note that this is a regular NIC and does not support iBFT -> /boot is on the local disk.

I have been unable to get into the dracut shell, even when I pass in rd.shell or rd.break as kernel options.

Boot Args:
BOOT_IMAGE=/vmlinuz-3.1.0-0.rc10.git0.1.fc16.i686.PAE root=/dev/sdb1 ro ifname=em1:00:07:e9:5c:e0:30 iscsi_initiator=iqn.1994-05.com.domain:01.0e9b92 rd.md=0 rd.lvm=0 rd.dm=0 ip=em1:dhcp KEYTABLE=us SYSFONT=latarcyrheb-sun16 netroot=iscsi:@192.168.0.5::3260::iqn.2009-02.net.tirfa:for.all rd.luks=0 LANG=en_US.UTF-8 console=ttyS0,38400n8 rd.debug rd.udev.debug


Version-Release number of selected component (if applicable):
dracut-013-15.fc16

How reproducible:
Every time, using LVM or not.

Steps to Reproduce:
1. do minimal install with / on iSCSI, /boot and swap on local disk
2. reboot
  
Actual results:
kernel panic on init

Expected results:
boot into working system

Additional info:
Comment 1 Tim Flink 2011-10-20 10:49:40 EDT
Created attachment 529303 [details]
console log from failed boot
Comment 2 Tim Flink 2011-10-20 11:05:11 EDT
Created attachment 529318 [details]
fstab from system
Comment 3 Tim Flink 2011-10-20 11:07:14 EDT
Proposing as a blocker for Fedora 16 final as it violates the following final release criterion [1]:

The installer must be able to complete an installation using any network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel)

[1] http://fedoraproject.org/wiki/Fedora_16_Final_Release_Criteria
Comment 4 Harald Hoyer 2011-10-20 11:10:09 EDT
[    3.748491] VFS: Cannot open root device "sdb1" or unknown-block(0,0)

This indicates, that no initramfs has been loaded and dracut was not even executed... 

Check your grub configuration!
Comment 5 Tim Flink 2011-10-20 11:19:16 EDT
Created attachment 529322 [details]
grub configuration

This is the grub configuration on that system. There is neither an initramfs on /boot nor a initrd directive in the grub configuration
Comment 6 Tim Flink 2011-10-20 11:20:13 EDT
Created attachment 529324 [details]
anaconda log
Comment 7 Tim Flink 2011-10-20 11:20:43 EDT
Created attachment 529325 [details]
anaconda program.log
Comment 8 Tim Flink 2011-10-20 11:21:12 EDT
Created attachment 529326 [details]
anaconda storage.log
Comment 9 David Lehman 2011-10-20 11:31:01 EDT
Looks to me like everything anaconda was supposed to put in GRUB_CMDLINE_LINUX is there for the iscsi root. Apparently either there is no initrd or else grub2-mkconfig failed to find/recognize it. Tim, can you tell us if there's an initramfs to match your kernel in /boot?
Comment 10 Tim Flink 2011-10-20 11:46:56 EDT
(In reply to comment #9)
> Looks to me like everything anaconda was supposed to put in GRUB_CMDLINE_LINUX
> is there for the iscsi root. Apparently either there is no initrd or else
> grub2-mkconfig failed to find/recognize it. Tim, can you tell us if there's an
> initramfs to match your kernel in /boot?

There are no initramfs files in /boot
Comment 11 Tim Flink 2011-10-20 11:59:18 EDT
Created attachment 529337 [details]
anaconda install.log
Comment 12 David Lehman 2011-10-20 12:02:25 EDT
From install.log:

mkinitrd failed
warning: %posttrans(kernel-PAE-3.1.0-0.rc10.git0.1.fc16.i686) scriptlet failed, exit status 1


Tim, can you reproduce and then run whatever's in the kernel's %posttrans from the command line so we can see what fails?
Comment 13 Tim Flink 2011-10-20 13:02:40 EDT
(In reply to comment #12)
> Tim, can you reproduce and then run whatever's in the kernel's %posttrans from
> the command line so we can see what fails?

The two commands in %posttrans for kernel-PAE are:
/sbin/new-kernel-pkg --package kernel-PAE --mkinitrd --dracut --depmod --update 3.1.0-0.rc10.git0.1.fc16.i686.PAE || exit $?

/sbin/new-kernel-pkg --package kernel-PAE --rpmposttrans 3.1.0-0.rc10.git0.1.fc16.i686.PAE || exit $?


The failure is in the first command:

# /sbin/new-kernel-pkg --package kernel-PAE --mkinitrd --dracut --depmod --update 3.1.0-0.rc10.git0.1.fc16.i686.PAE

F: Failed to install wget
mkinitrd failed
Comment 14 Tim Flink 2011-10-20 13:12:19 EDT
Created attachment 529343 [details]
dracut.log after running failed new-kernel-pkg cmd
Comment 15 Tim Flink 2011-10-20 13:47:43 EDT
I re-ran the kernel %posttrans commands after adding the following to dracut.conf:
omit_dracutmodules+="livenet"

I got an initramfs and the system now boots with the iSCSI target as /.

Re-assigning to dracut.
Comment 16 Tim Flink 2011-10-20 14:13:22 EDT
I tried the patch that wwoods posted:

http://article.gmane.org/gmane.linux.kernel.initramfs/2183

After reverting dracut.conf and verified that the kernel posttrans commands failed again. I applied the patch to /usr/share/dracut/modules.d/90livenet/module-setup.sh, re-ran the kernel posttrans commands and they completed successfully.
Comment 17 Fedora Update System 2011-10-21 06:59:48 EDT
dracut-013-16.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-16.fc16
Comment 18 Adam Williamson 2011-10-21 13:28:40 EDT
Discussed at 2011-10-21 blocker review meeting. Accepted as a blocker per criterion "The installer must be able to complete an installation using any network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel)" - technically the bug only affects minimal installations, but iSCSI is a case where you're probably quite likely to do a minimal install, and it seems a serious enough bug to take as a blocker.
Comment 19 Fedora Update System 2011-10-21 20:10:43 EDT
Package dracut-013-16.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-013-16.fc16'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14704
then log in and leave karma (feedback).
Comment 20 Fedora Update System 2011-10-24 23:44:24 EDT
dracut-013-16.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 21 Tim Flink 2011-10-25 12:10:13 EDT
Verified with an i686 minimal install of Fedora 16 using iSCSI target as /