Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 1100191

Summary: Fails to include required cciss driver
Product: Red Hat Enterprise Linux 5 Reporter: tc
Component: mkinitrdAssignee: Brian Lane <bcl>
Status: CLOSED DUPLICATE QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.10CC: ddumas, jherrman, tc
Target Milestone: rc   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, mkinitrd searched an incorrect path for the cciss block driver device and the findstoragedriver() function could therefore not find cciss in the /sys/block directory. Consequently, the kernel failed to boot after it was updated. This update amends the cciss path and findstoragedriver() is thus able to detect cciss devices correctly. As a result, the kernel now boots as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-05-22 15:07:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
patch for ccis driver
none
bash -x output from unpatched mkinitrd
none
bash -x output from npatched mkinitrd none

Description tc 2014-05-22 08:22:42 UTC
Description of problem:
Fails to include cciss driver even though listed in modprobe.conf, leading to kernel panic on reboot

How reproducible:
Every time

Steps to Reproduce:
mkinitrd -v -f /boot/initrd-$(uname -r).new.img $(uname -r)
# note this says ' Looking for deps of module cciss: scsi_mod ' but doesn't appear in 'Adding' section

gzip -dc /boot/initrd-2.6.18-371.6.1.el5.new.img | cpio -id
# check in init and lib folder, compared to a working initrd I made by hand

#diff initrd-2.6.18-371.6.1.el5/init initrd-2.6.18-371.6.1.el5.new/init
59,60d58
< echo "Loading cciss.ko module"
< insmod /lib/cciss.ko
#ls */lib
initrd-2.6.18-371.6.1.el5/lib:
cciss.ko dm-mem-cache.ko dm-mirror.ko dm-raid45.ko dm-snapshot.ko ehci-hcd.ko firmware ohci-hcd.ko sd_mod.ko
dm-log.ko dm-message.ko dm-mod.ko dm-region_hash.ko dm-zero.ko ext3.ko jbd.ko scsi_mod.ko uhci-hcd.ko

initrd-2.6.18-371.6.1.el5.new/lib:
dm-log.ko dm-message.ko dm-mod.ko dm-region_hash.ko dm-zero.ko ext3.ko jbd.ko scsi_mod.ko uhci-hcd.ko
dm-mem-cache.ko dm-mirror.ko dm-raid45.ko dm-snapshot.ko ehci-hcd.ko firmware ohci-hcd.ko sd_mod.ko

Additional info:
# cat /etc/modprobe.conf
alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter cciss
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd

# mkinitrd -v -f /boot/initrd-$(uname -r).new.img $(uname -r)
Creating initramfs
Modulefile is /etc/modprobe.conf
Looking for deps of module ehci-hcd
Looking for deps of module ohci-hcd
Looking for deps of module uhci-hcd
Looking for deps of module ext3: jbd
Looking for deps of module jbd
Looking for driver for device cciss/c0d0p2
Looking for deps of module cciss: scsi_mod
Looking for deps of module scsi_mod
Looking for deps of module sd_mod: scsi_mod
Looking for deps of module ide-disk
Looking for deps of module dm-mod
Looking for deps of module dm-mirror: dm-mod dm-log
Looking for deps of module dm-log: dm-mod
Looking for deps of module dm-zero: dm-mod
Looking for deps of module dm-snapshot: dm-mod
Looking for deps of module dm-mem-cache
Looking for deps of module dm-region_hash: dm-mod dm-log
Looking for deps of module dm-message
Looking for deps of module dm-raid45: dm-message dm-mod dm-mem-cache dm-log dm-region_hash
Using modules: /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/usb/host/ehci-hcd.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/usb/host/ohci-hcd.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/usb/host/uhci-hcd.ko /lib/modules/2.6.18-371.6.1.el5/kernel/fs/jbd/jbd.ko /lib/modules/2.6.18-371.6.1.el5/kernel/fs/ext3/ext3.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/scsi/scsi_mod.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/scsi/sd_mod.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-mod.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-log.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-mirror.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-zero.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-snapshot.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-mem-cache.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-region_hash.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-message.ko /lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-raid45.ko
/sbin/nash -> /tmp/initrd.i21633/bin/nash
/sbin/insmod.static -> /tmp/initrd.i21633/bin/insmod
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/usb/host/ehci-hcd.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/ehci-hcd.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/usb/host/ohci-hcd.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/ohci-hcd.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/usb/host/uhci-hcd.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/uhci-hcd.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/fs/jbd/jbd.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/jbd.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/fs/ext3/ext3.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/ext3.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/scsi/scsi_mod.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/scsi_mod.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/scsi/sd_mod.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/sd_mod.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-mod.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-mod.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-log.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-log.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-mirror.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-mirror.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-zero.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-zero.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-snapshot.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-snapshot.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-mem-cache.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-mem-cache.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-region_hash.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-region_hash.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-message.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-message.ko' [elf32-i386]
copy from `/lib/modules/2.6.18-371.6.1.el5/kernel/drivers/md/dm-raid45.ko' [elf32-i386] to `/tmp/initrd.i21633/lib/dm-raid45.ko' [elf32-i386]
/sbin/lvm.static -> /tmp/initrd.i21633/bin/lvm
/sbin/dmraid.static -> /tmp/initrd.i21633/bin/dmraid
/sbin/kpartx.static -> /tmp/initrd.i21633/bin/kpartx
Adding module ehci-hcd
Adding module ohci-hcd
Adding module uhci-hcd
Adding module jbd
Adding module ext3
Adding module scsi_mod
Adding module sd_mod
Adding module dm-mod
Adding module dm-log
Adding module dm-mirror
Adding module dm-zero
Adding module dm-snapshot
Adding module dm-mem-cache
Adding module dm-region_hash
Adding module dm-message
Adding module dm-raid45

Comment 1 Brian Lane 2014-05-22 15:07:11 UTC

*** This bug has been marked as a duplicate of bug 988020 ***

Comment 2 tc 2014-05-22 15:08:56 UTC
Thanks BCL.

" You are not authorized to access bug #988020. "

Can you elaborate why this is a duplicate ?

Comment 3 Brian Lane 2014-05-22 20:42:26 UTC
Created attachment 898491 [details]
patch for ccis driver

Sorry about that. It is fixed in the current release -- here's the patch.

Comment 4 tc 2014-05-22 20:50:58 UTC
Ohh, thanks, that looks like the culprit.

By 'current release' you mean a mkinitrd later than that that ships with EL5.10 ? Are there plans to back port this to the mkinitrd v5.1 in EL5.10 then ? It's a nasty bug...

Comment 5 Brian Lane 2014-05-22 21:27:05 UTC
It will be fixed in 5.11

Comment 6 tc 2014-05-23 08:00:48 UTC
No back port even though it kills systems on Kernel updates ?

Comment 7 tc 2014-05-23 11:58:16 UTC
The patch doesn't work after applying to mkinitrd here, the output from a patched mkinitrd is better :

...
Looking for deps of module jbd
Looking for driver for device cciss/c0d0p2
Looking for driver for cciss device cciss/c0d0p2
Looking for deps of module pci:v00000E11d00000046sv00000E11sd00004091bc01sc04i00: scsi_mod 
Looking for deps of module scsi_mod
Looking for deps of module sd_mod: scsi_mod 
...

but the initrd still doesn't contain the cciss module in lib/ or an entry in the init script to load it. There's no reference to it in the various 'copy from' lines.

If it's any help I have a /dev/cciss/c0d0p1 and a /sys/block/cciss!c0d0/cciss!c0d0p1

Comment 8 Brian Lane 2014-05-27 20:49:29 UTC
I don't know why, but cciss/c0d0p2 != cciss/c0d0p1

You could try running with: 
bash -x mkinitrd -v -f initrd-`uname -r`.img `uname -r`

and attach the output here as a text/plain attachment.

Comment 9 tc 2014-05-28 08:14:58 UTC
Sorry, I should have been clearer with my devices - that was an example, here are my actual /boot mount points:

# mount|grep boot
/dev/cciss/c0d0p1 on /boot type ext3 (rw)
# ls -lah /sys/block/cciss\!c0d0/cciss\!c0d0p1
total 0
drwxr-xr-x 3 root root    0 May 23 11:47 .
drwxr-xr-x 7 root root    0 May 23 11:47 ..
-r--r--r-- 1 root root 4.0K May 28 08:05 dev
drwxr-xr-x 2 root root    0 May 17 17:56 holders
-r--r--r-- 1 root root 4.0K May 28 08:05 size
-r--r--r-- 1 root root 4.0K May 28 08:05 start
-r--r--r-- 1 root root 4.0K May 28 08:05 stat
lrwxrwxrwx 1 root root    0 May 28 08:05 subsystem -> ../../../block
--w------- 1 root root 4.0K May 28 08:05 uevent
 
My root FS uses LVM on /dev/cciss/c0d0p2 :
# mount|grep LogVol00
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
# pvdisplay 
  --- Physical volume ---
  PV Name               /dev/cciss/c0d0p2
  VG Name               VolGroup00
  PV Size               33.81 GB / not usable 479.00 KB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              1082
  Free PE               23
  Allocated PE          1059
  PV UUID               Rb75NE-yGGn-QN4u-4UYi-obXM-Eu8I-TkE0qy
# ls -lah /sys/block/cciss\!c0d0/cciss\!c0d0p2
total 0
drwxr-xr-x 3 root root    0 May 28 08:08 .
drwxr-xr-x 7 root root    0 May 28 08:08 ..
-r--r--r-- 1 root root 4.0K May 28 08:08 dev
drwxr-xr-x 2 root root    0 May 23 11:47 holders
-r--r--r-- 1 root root 4.0K May 28 08:12 size
-r--r--r-- 1 root root 4.0K May 28 08:12 start
-r--r--r-- 1 root root 4.0K May 28 08:12 stat
lrwxrwxrwx 1 root root    0 May 28 08:12 subsystem -> ../../../block
--w------- 1 root root 4.0K May 28 08:12 uevent

I will attach patched and unpatched logs in a sec.

Comment 10 tc 2014-05-28 08:18:03 UTC
Created attachment 899872 [details]
bash -x output from unpatched mkinitrd

Comment 11 tc 2014-05-28 08:18:22 UTC
Created attachment 899873 [details]
bash -x output from npatched mkinitrd

Comment 12 tc 2014-06-12 14:59:47 UTC
Our other 2.6.18-371.6.1.el5debug w/ mkinitrd-5.1.19.6-81.el5_10 box just picked up a new kernel, and included the driver OK.

What else does mkinitrd depend on ?

I checked, and the paths look the same on both boxes.

Comment 13 tc 2014-06-27 08:03:43 UTC
Are there any additional debugging steps I can take ?

Comment 14 tc 2014-10-01 08:35:11 UTC
FYI this was resolved when https://rhn.redhat.com/errata/RHBA-2014-1224.html was rolled out this week, as part of http://lists.centos.org/pipermail/centos-announce/2014-September/020632.html