Bug 476368

Created attachment 327155 [details]
patch for critical disks problem

This patch will fix your critical disk issue.

As for the scsi support, its not handled by the code you're looking at.  Its handled in findstoragedriverinsys.  I expect that the dependency tree for your hard drive controller may not contain the needed dependencies.  Could you please do this for me:

Where <X> is the name of a drive that you want kdump to dump to:

cd /sys/block/<X>/device
/sbin/modprobe --show-depends `cat modalias`

That should show you the modules we will pull in to support this module.  If its empty, thats our problem.  Thanks!

There is no /sys/block/md1/device.

There's a /sys/block/md1/dev, a plain file, containing one line, "9:1". There's also a symlink:

lrwxrwxrwx  1 root root    0 2008-12-16 18:17 bdi -> ../../bdi/9:1


/dev/md1 is a RAID-1 volume with physical partitions /dev/sda2 and /dev/sdb2.


lrwxrwxrwx 1 root root 0 2008-12-13 17:57 /sys/block/sda/device -> ../../../0:0:0:0
lrwxrwxrwx 1 root root 0 2008-12-13 17:57 /sys/block/sdb/device -> ../../../1:0:0:0


[root@octopus device]# cat /sys/block/sd[ab]/device/modalias
scsi:t-0x00
scsi:t-0x00
[root@octopus device]# /sbin/modprobe --show-depends scsi:t-0x00
insmod /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/scsi_mod.ko 
insmod /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/sd_mod.ko 

That's all that modprobe reports, but it omits my chipset, sata_nv.ko. The relevant snippets of my lsmod output are:

sd_mod                 33200  10 
scsi_mod              151896  3 sg,libata,sd_mod
libata                155584  4 pata_amd,ata_generic,sata_nv,pata_acpi
sata_nv                31628  8 
sg                     40528  0 
ata_generic            14084  0 
pata_amd               21124  0 
pata_acpi              13184  0 


Your patch added the missing modules to the dumprd img:

...

Adding module ehci-hcd
Adding module ohci-hcd
Adding module uhci-hcd
Adding module ext3
Adding module raid1
Adding module raid456
Adding module scsi_mod
Adding module sd_mod
Adding module libata
Adding module pata_acpi
Adding module sata_nv
Adding module ata_generic

...

However the resulting kdump initrd still won't come up, because etc/critical disks still contains references to md:

# cat etc/critical_disks
md
md
md


I get three copies of md instead of two, now, in critical_disks.

I stand corrected. I ran mkinitrd when I meant mkdumprd. The patched mkdumprd still does not put the right modules into the initrd image. I got all excited prematurely.

The patch is applied:

...

findstoragedriver () {
    for device in $@ ; do
        basedev=`echo $device | sed -e's/\/dev\///' -e's/[0-9]\+//'`
        if [ -d /sys/block/$basedev ]
        then
            #only add devices which have a presence in sysfs
            echo $basedev >> $TMPDISKLIST
        fi
        case " $handleddevices " in
            *" $device "*)
                continue ;;
            *) handleddevices="$handleddevices $device" ;;
        esac

...

Here's what happens when I run mkdumprd:

[root@octopus sbin]# rm -rf foo.img ; mkdumprd -d -v -f foo.img 2.6.27.7-53.fc9.x86_64
Creating initramfs
Looking for deps of module ide-disk
Looking for deps of module ext3: jbd mbcache
Looking for deps of module jbd
Looking for deps of module mbcache
Looking for driver for device md1
Looking for driver for device md2
Using modules: /lib/modules/2.6.27.7-53.fc9.x86_64/kernel/fs/jbd/jbd.ko /lib/modules/2.6.27.7-53.fc9.x86_64/kernel/fs/mbcache.ko /lib/modules/2.6.27.7-53.fc9.x86_64/kernel/fs/ext3/ext3.ko
Looking for deps of module raid1
copy from `/lib/modules/2.6.27.7-53.fc9.x86_64/kernel/fs/jbd/jbd.ko' [elf64-x86-64] to `/tmp/initrd.6eaKY4/lib/jbd.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.27.7-53.fc9.x86_64/kernel/fs/mbcache.ko' [elf64-x86-64] to `/tmp/initrd.6eaKY4/lib/mbcache.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.27.7-53.fc9.x86_64/kernel/fs/ext3/ext3.ko' [elf64-x86-64] to `/tmp/initrd.6eaKY4/lib/ext3.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.27.7-53.fc9.x86_64/kernel/drivers/md/raid1.ko' [elf64-x86-64] to `/tmp/initrd.6eaKY4/lib/raid1.ko' [elf64-x86-64]
Adding module jbd
Adding module mbcache
Adding module ext3
Adding module raid1


As my previous message indicates, which really reports the output from mkinitrd, its output includes sata_nv, and a whole bunch of other modules that are needed to build up the scsi stack.

Created attachment 327235 [details]
new patch

Ok, I thought the sata_nv thing had been a transient error, since we're not that different from mkinitrd in this code.  Anywho, this new patch should pull in all the extra modules you need for your scsi stack (be aware, it may not pull in everything that mkinitrd does, nor does it necessecarily need to, depending on your kdump configuration).  I've not tested it yet, but it should work fine.

The second patch still does not work. I see two reasons for that:

1) The patch introduces special processing when it sees the libata module. The problem is that mkdumprd doesn't even find the libata module in the first place. I already posted the output of mkdumprd -v before, here it is again. I see no difference in the output of mkdumprd. The only think that mkdumprd finds are the /dev/md[012] devices. All my partitions are RAID-1 partitions. That's the only block devices that mkdumprd finds, and it is unable to see beyond the /dev/md[012] layer and identify the drivers for the underlying block devices. Only when it does it, will it see that the underlying block devices use libata, and process all the corresponding dependencies.

2) Once the patch finds libata, it uses lsmod to find its modules. Which means that it will now look at the modules loaded by the running kernel, so mkdumprd cannot be used to build a dumprd image for another installed kernel version.

It may very well be that libata requires special processing. But mkdumprd will not even get to that point unless it sees libata in the dependency list, and it won't get to that point until it figures out what the md block devices drive underneath. What probably needs to happen is for findstoragedriver() to check if the device is /dev/md*, if so, grab what's in /proc/mdstat:

[root@octopus tmp]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4] 
md0 : active raid1 sda1[0] sdb1[1]
      104320 blocks [2/2] [UU]
      
md3 : active raid1 sda5[0] sdb5[1]
      216829184 blocks [2/2] [UU]
      
md2 : active raid1 sda3[0] sdb3[1]
      6289344 blocks [2/2] [UU]
      
md1 : active raid1 sda2[0] sdb2[1]
      20972736 blocks [2/2] [UU]
      
unused devices: <none>

So, mdstat will then need to proceed and extract all the device names. In my case, sdaX and sdbX, then proceed to do dependency analysis for /dev/sdaX and /dev/sdbX:

# cat /sys/block/sda/device/modalias
scsi:t-0x00

[root@octopus device]# /sbin/modprobe --show-depends scsi:t-0x00
insmod /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/scsi_mod.ko 
insmod /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/sd_mod.ko 

[root@octopus device]# lsmod | grep scsi
scsi_mod              151896  3 sg,libata,sd_mod

Only when you get to this point, will you see libata in the dependency list.

======================================================================


[root@octopus tmp]# sum /sbin/mkinitrd
42796    50    (patched mkinitrd)
[root@octopus tmp]# rm -f foo.img; mkdumprd -d -v -f foo.img 2.6.26.6-79.fc9.x86_64
Creating initramfs
WARNING: using /tmp for temporary files
Looking for deps of module ide-disk
Looking for deps of module ext3: jbd mbcache
Looking for deps of module jbd
Looking for deps of module mbcache
Looking for driver for device md1
Looking for driver for device md2
Using modules: /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko
Looking for deps of module raid1
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko' [elf64-x86-64] to `/tmp/initrd.kNO3GU/lib/jbd.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko' [elf64-x86-64] to `/tmp/initrd.kNO3GU/lib/mbcache.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko' [elf64-x86-64] to `/tmp/initrd.kNO3GU/lib/ext3.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko' [elf64-x86-64] to `/tmp/initrd.kNO3GU/lib/raid1.ko' [elf64-x86-64]
Adding module jbd
Adding module mbcache
Adding module ext3
Adding module raid1

One other comment:

[mrsam@octopus ~]$ ls /sys/block
md0  md2  ram0  ram10  ram12  ram14  ram2  ram4  ram6  ram8  sda
md1  md3  ram1  ram11  ram13  ram15  ram3  ram5  ram7  ram9  sdb


The code that creates etc/critical_disks probably needs special logic: if the device is mdX, do not strip off the numerical suffix.

sda2 => sda, but md0 => md0

Created attachment 327332 [details]
RHEL 5 mkdumprd

you know, something just occured to me.  I've been fixing all these things in RAWhid and RHEL, but I've let F-9 slip.  I've got several md fixes in the former two trees, and you can probably just pick up the mkdumprd script from there.  Here's the RHEL5 mkdumprd, and it should solve this.  Please give it a shot.

I agree -- it's better to sync up with RHEL here. That's really is the preferred process flow -- flesh out the new stuff in Fedora, and move it to RHEL when it's stable.


RHEL 5's mkdumprd produces the following output, before aborting:

mkdumprd -d -v -f foo.img 2.6.26.6-79.fc9.x86_64
Creating initramfs
WARNING: using /tmp for temporary files
Looking for deps of module ide-disk
Looking for deps of module ext3: jbd mbcache
Looking for deps of module jbd
Looking for deps of module mbcache
Found RAID component md1
Looking for deps of module raid1
Looking for driver for device sda2
Found RAID component md2
Looking for driver for device sda3
Using modules: /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko' [elf64-x86-64] to `/tmp/initrd.TsBXXI/lib/jbd.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko' [elf64-x86-64] to `/tmp/initrd.TsBXXI/lib/mbcache.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko' [elf64-x86-64] to `/tmp/initrd.TsBXXI/lib/ext3.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko' [elf64-x86-64] to `/tmp/initrd.TsBXXI/lib/raid1.ko' [elf64-x86-64]
Adding module jbd
Adding module mbcache
Adding module ext3
Adding module raid1
cp: cannot stat `/sbin/dmsetup.static': No such file or directory


I see that mkdumprd is now seeing the sda device, which is progress -- it didn't before. But then it fails with this error message. This is probably some RHEL-specific stuff.

It's still not picking up sata_nv, the underlying driver for sda -- perhaps once this error message is fixed it'll find it.

Created attachment 327453 [details]
fixed RHEL5 mkdumprd

You're right, my bad.  RHEL5 still has statically linked items, but F-9 doesn't.  Easy fix.  Here you go.

The results from this mkdumprd:

rm -f foo.img; mkdumprd -d -v -f foo.img 2.6.26.6-79.fc9.x86_64
Creating initramfs
WARNING: using /tmp for temporary files
Looking for deps of module ide-disk
Looking for deps of module ext3: jbd mbcache
Looking for deps of module jbd
Looking for deps of module mbcache
Found RAID component md1
Looking for deps of module raid1
Looking for driver for device sda2
Found RAID component md2
Looking for driver for device sda3
Using modules: /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko' [elf64-x86-64] to `/tmp/initrd.vHgJ0a/lib/jbd.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko' [elf64-x86-64] to `/tmp/initrd.vHgJ0a/lib/mbcache.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko' [elf64-x86-64] to `/tmp/initrd.vHgJ0a/lib/ext3.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko' [elf64-x86-64] to `/tmp/initrd.vHgJ0a/lib/raid1.ko' [elf64-x86-64]
Adding module jbd
Adding module mbcache
Adding module ext3
Adding module raid1


The results:

1) mkdumprd completes succesfully, but scsi_mod, sk_mod, and sata_nv are still not being loaded. mkdumprd sees sda, but is not adding the scsi modules, libata, sata_nv, etc...

2)

# cat etc/critical_disks
sda
sda

Just to be sure, I crashed manually, and, as expected, the resulting dumprd image looped forever waiting for /sys/block/sda to show up, when it couldn't since all the low level SCSI and chipset drivers were not in the dumprd image.


The RAID partitions are RAID-1 across sda and sdb:

md0 : active raid1 sda1[0] sdb1[1]
      104320 blocks [2/2] [UU]
      
md3 : active raid1 sda5[0] sdb5[1]
      216829184 blocks [2/2] [UU]
      
md2 : active raid1 sda3[0] sdb3[1]
      6289344 blocks [2/2] [UU]
      
md1 : active raid1 sda2[0] sdb2[1]
      20972736 blocks [2/2] [UU]
      
mkdumprd sees sda, but not sdb, for some reason. This is not critical, but something is still not working 100%.

Ok, something else must be going on on your system.  I just tried to recreate this on a local system here with F-9 installed and sata drives on board, and it worked fine.  Can you add a set -x to the top of your mkdumprd and send me the output please.  Its like the dependencies on your systems aren't the same as mine.

Created attachment 327726 [details]
mkdumprd output with -x

It's not just SATA, I think, it's /dev/md on top of SATA for all partitions, including swap. All block devices, ext3 and swap, are /dev/md devices.

Here you go.

Thanks for your help.

Created attachment 327764 [details]
new version of mkdumprd

Thank you.  I think I see at least part of the problem.  I'm not sure quite how this happens, but when we handle raid devices we  call handleraid, which then in turns firuges out the slave devices of the raid array, and calls findstoragedriver on them.  When that happens we search for the component devices based on their partition name rather than their base device name (sda2 vs. sda).  Of course that fails. Not sure this is the best solution, but its non-invasive, which I think is good at the moment.  Please give this version of mkdumprd a shot.  It should at least get you farther in the process, if not completely fix the problem.  Thanks!

Created attachment 327785 [details]
New new version of mkdumprd

There was a typo in the script in one of the sed command, and sed was complaining. The typo was obvious to me, so I fixed it. Attached is the fixed script, however it still does not load the sata_nv module. Same as before.

This is a corner case here. I think you're seeing this happening because all partitions, including swap, are hidden behind softraid /dev/md. This is not common. Almost every time there'll be a partition -- usually swap -- that does not use softraid, and mkdumprd picks up the underlying chipset modules through the non-raid partition.


Creating initramfs
WARNING: using /tmp for temporary files
Looking for deps of module ide-disk
Looking for deps of module ext3: jbd mbcache
Looking for deps of module jbd.
Looking for deps of module mbcache
Found RAID component md1
Looking for deps of module raid1
Looking for driver for device sda2
Found RAID component md2
Looking for driver for device sda3
Using modules: /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko' [elf64-x86-64] to `/tmp/initrd.05LqDA/lib/jbd.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko' [elf64-x86-64] to `/tmp/initrd.05LqDA/lib/mbcache.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko' [elf64-x86-64] to `/tmp/initrd.05LqDA/lib/ext3.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko' [elf64-x86-64] to `/tmp/initrd.05LqDA/lib/raid1.ko' [elf64-x86-64]
Adding module jbd
Adding module mbcache
Adding module ext3
Adding module raid1



The problem seems to be here:


Looking for driver for device sda3
++ echo sda3
++ sed '-es/\//\!/g' '-es/[0-9]\+//'
+ device=sda
++ findone -type d /sys/block -name sda
++ echo nash-find -type d /sys/block -name sda
++ /sbin/nash --force --quiet
++ /bin/awk '{ print $1; exit; }'
+ sysfs=


This corresponds to the following snippet of mkdumprd:

        vecho "Looking for driver for device $device"
        device=`echo $device | sed -e's/\//\!/g' -e's/[0-9]\+//'`
        sysfs=$(findone -type d /sys/block -name $device)
        [ -z "$sysfs" ] && return

sysfs is set to empty string, so this function terminates at this point, without finding the driver for sda.

From what I can tell, the above is equivalent to:

find /sys/block -type d -name sda

This, indeed, produces no output:

[root@octopus ~]# find /sys/block -type d -name sda
[root@octopus ~]# 

This is because:

lrwxrwxrwx 1 root root 0 2008-12-20 08:44 /sys/block/sda -> ../devices/pci0000:00/0000:00:05.0/host0/target0:0:0/0:0:0:0/block/sda


My /sys/block/sda is a symlink, not a subdirectory, so it fails the -type d test.

Not sure what needs to be done here.

Created attachment 327786 [details]
Output of modified new version of mkdumprd (without the -type d)

nash's find does not implement -type l. If I drop -type d completely from here:

        vecho "Looking for driver for device $device"
        device=`echo $device | sed -e's/\//\!/g' -e's/[0-9]\+//'`
        sysfs=$(findone /sys/block -name $device)
        [ -z "$sysfs" ] && return


Then, mkdumprd makes more progress:

# rm -f foo.img; mkdumprd -d -v -f foo.img 2.6.26.6-79.fc9.x86_64
Creating initramfs
WARNING: using /tmp for temporary files
Looking for deps of module ide-disk
Looking for deps of module ext3: jbd mbcache
Looking for deps of module jbd
Looking for deps of module mbcache
Found RAID component md1
Looking for deps of module raid1
Looking for driver for device sda2
Looking for deps of module scsi_mod
Looking for deps of module sd_mod: scsi_mod
Looking for driver for device sdb2
Found RAID component md2
Looking for driver for device sda3
Looking for driver for device sdb3
Using modules: /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/scsi_mod.ko /lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/sd_mod.ko
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/jbd/jbd.ko' [elf64-x86-64] to `/tmp/initrd.o6XngB/lib/jbd.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/mbcache.ko' [elf64-x86-64] to `/tmp/initrd.o6XngB/lib/mbcache.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/fs/ext3/ext3.ko' [elf64-x86-64] to `/tmp/initrd.o6XngB/lib/ext3.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/md/raid1.ko' [elf64-x86-64] to `/tmp/initrd.o6XngB/lib/raid1.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/scsi_mod.ko' [elf64-x86-64] to `/tmp/initrd.o6XngB/lib/scsi_mod.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.26.6-79.fc9.x86_64/kernel/drivers/scsi/sd_mod.ko' [elf64-x86-64] to `/tmp/initrd.o6XngB/lib/sd_mod.ko' [elf64-x86-64]
Adding module jbd
Adding module mbcache
Adding module ext3
Adding module raid1
Adding module scsi_mod
Adding module sd_mod


* mkdump now sees both sda and sdb (both drives in the RAID1 array, instead of just sda)

* mkdump now finds scsi_mod and sd_mod, but still doesn't find libata and scsi_nv.

Here's the output of set -x.

... I mean sata_nv. grepping through the output of set -x, I find no reference to libata, nor sata_nv.

I have similar problems on 2 of my systems, one is raid on top of lvm with no mounted partitions and another one is pretty straight forward rootfs on /dev/sda (but with sata_nv as a module).
I am using Fedora 10.
Fedora-10 supplied mkdumprd does not work at all on both systems, on a complex one it fails because it cannot find  module raid5, on a simple one it does not add any modules into the initrd, not even sata_nv, so it cannot find root.
After I taught mkdumprd about raid456 module, I can generate the initrd on a more complex system, but it now does not add actual disk controller driver (and firmware) to the initrd generated.

I decided to give a try to the latest mkdumprd posted here.
On a simple setup it still does not add sata_nv (or any other) modules to initrd and so crashdumping does not work.

On a more complex setup on the other hand it seems like some sed commands are executed on stdin, namely one in line 1570:     options2=`sed -n -e "s/^options[    ][      ]*$module[      ][      ]*//p" $KDUMP_CONFIG_FILE 2>/dev/null`

it turns out I need to run the script with -d option to make KDUMP_CONFIG_FILE defined and overcome this problem, yet no physical device modules are present still in the generated image.

ping

pong.  I just got back from holiday break, and don't have anything new to report.

hey, just out of curiosity, if you specify sata_nv and libata in the extra_modules option in /kdump.conf, does this all work for you?

(In reply to comment #20)
> hey, just out of curiosity, if you specify sata_nv and libata in the
> extra_modules option in /kdump.conf, does this all work for you?  

Yes. It works for me (I though I had reported this back with the original bug, but apparently I did not), but on a simple setup only. Need to run mkdumprd -d for it to take effect if I remember correctly.
In fact only listing sata_nv is enough in my case, it works out dependencies on its own.

Created attachment 336079 [details]
patch to add stat_nv if its found on the running system

ok, you know, I'm in the process of writing a new configuration model anyway, And if using extra_modules fixes it, I'm not above adding a hack to automate that, since it seems theres no clear depedency link for stata_nv.  Please confirm that this fixes your issue, and I'll check it in.  Thanks!

Adding just sata_nv to extra_modules in kdump.conf (there is no libata module in the current kernel, 2.6.27.19) seems to allow initrc to find all the required devices, initialize all devices and continue booting. I'm 

Unfortunately, the kdump boot still ends up getting stuck after switching to the new root, after printing "Starting udev". Starting udevd, in the new kernel environment, apparently hangs, with no further progress being made.

So, it looks like my original bug can be fixed by explicitly listing sata_nv in extra_modules, however that won't fix all of my problems.

Why are you not configuring kdump to capture your vmcore from the initramfs (by specifying a dump target in /etc/kdump.conf)?

I believe I am. Currently, I have:

path /var/crash
extra_modules sata_nv

/var/crash is there and waiting to receive any crash dumps I send its way. However, alt-sysrq-c produces nothing.

Close, but you're not. You need to specify a dump target, which is a configuration line beginning with one of the following directives:
ext3
ext2
minix
net
raw

This tells kdump where to place the vmcore, the path configuarion variable is relative to the above directives.

Please see the kdump docs or the config file notes for examples of how to fully specify the above dump target directives.

Some partial success.

My server has the following in /etc/fstab:

LABEL=/                 /                       ext3    defaults        1 1
LABEL=/home             /home                   ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2

This results in the following mounts:

/dev/md1 on / type ext3 (rw)
/dev/md3 on /home type ext3 (rw)
/dev/md0 on /boot type ext3 (rw)

That is, all my partitions are softraid partitions, mounted using their labels. Grub boots the kernel with "root=LABEL=/".

With the above in mind:

Putting "ext3 /dev/md1" in kdump.conf works. "ext3 LABEL=/" does not.

I'm going to attach some crappy screen captures from my cell phone. First snap will be "ext3 /dev/md1", which seems to produce a good dump in /var/crash, however there's still an error message coming up on the console "fsck.ext3: not found". That does not seem to be fatal, the crash dump proceeds to success.

The second screen dump is what happens with "ext3 LABEL=/". The resulting dumprd image does not work.

Finally, I removed "extra_modules sata_nv", reran mkdumprd, and compared the resulting mkdump initrd image with the known good mkdump initrd that's working. I confirmed that the only differences are:

1) The init script loads sata_nv.ko:

[root@octopus initrd.devmd1]# diff -U 3 ../initrd.nosatanv/init init
--- ../initrd.nosatanv/init	2009-03-23 17:31:51.000000000 -0400
+++ init	2009-03-23 17:31:10.000000000 -0400
@@ -41,6 +41,8 @@
 mknod /dev/ttyS2 c 4 66
 mknod /dev/ttyS3 c 4 67
 mknod /dev/mem c 1 1
+echo "Loading sata_nv.ko module"
+insmod /lib/sata_nv.ko 
 echo "Loading raid1.ko module"
 insmod /lib/raid1.ko 
 echo "Waiting for required block device discovery"

2) The initrd image contains lib/sata_nv.ko, which is to be expected.

So, to summarize, here's what my problems are with kexec-tools-2.0.0-2.fc10.x86_64:

1) When all mounted partitions (including swap), are softraid partitions, mkdumprd does not find the underlying driver, sata_nv.ko, and does not insert it into the initrd image it builds

2) "ext3 LABEL=/[label]" does not work. I must use "ext3 /dev/mdXX"

Created attachment 336409 [details]
Successful crash dump

Bad cell phone pic. Error message: fsck.ext3: not found, however the crash dump still succeeds.

Created attachment 336412 [details]
Results of "ext3 LABEL=/" in kdump.conf

Using "ext3 LABEL=[label]" does not seem to work. I must use "ext3 LABEL=/dev/mdXX".

Don't suppose that you have a serial port capture of that entire dump, do you?

No, I don't, and I can't find my null modem plug. However, I went over my notes, and have more info.

The reason that "ext3 LABEL=/" originally id not work for me is that I still had "extra_modules sata_nv" commented out in that configuration. My error. However, when I put it back in, it still didn't work, and here's why.

This is the relevant part of the init script:

echo Saving to the local filesystem LABEL=/
DUMPDEV=LABEL=/
IS_LABEL=`echo $DUMPDEV | grep LABEL`
IS_UUID=`echo $DUMPDEV | grep UUID`
if [ -n "$IS_LABEL" -o -n "$IS_UUID" ] 
then
  DUMPDEV=`findfs $DUMPDEV`
fi
fsck.ext3 $DUMPDEV
mount -t ext3 $DUMPDEV /mnt
if [ $? == 0 ]
then
  mkdir -p /mnt//var/crash/127.0.0.1-$DATE
  VMCORE=/mnt//var/crash/127.0.0.1-$DATE/vmcore
  export VMCORE
  monitor_cp_progress $VMCORE-incomplete &
  cp /proc/vmcore $VMCORE-incomplete >/dev/null
  exitcode=$?
  if [ $exitcode == 0 ]
  then
      mv $VMCORE-incomplete $VMCORE
      echo -e "\\033[0JSaving core complete"
  fi
fi
umount /mnt

That's the part that corresponds to the screen cap.

1) By inserting additional statements, it appears that "findfs LABEL=/" prints /dev/md0, which is wrong. /dev/md0 is my /boot, so the dumprd image attempts to dump the kernel into the 99mb /boot partition, which doesn't go far. After I reboot, I see the wreckage in /boot/var/crash :-).

The correct partition is, of course, /dev/md1, and, as I said, when I put "ext3 /dev/md1" in kdump.conf, I get a good dump.

I compared again the initrd images with "ext3 LABEL=/" and "ext3 /dev/md1". The only difference in the init script is the DUMPDEV setting.

2) There is no fsck.ext3 in the initrd image. That's what generates the "fsck.ext3 not found" error message in my screen cap:

[root@octopus initrd.nolabel]# ls -al */*fsck*
lrwxrwxrwx 1 root root 7 2009-03-23 17:33 bin/fsck -> busybox
lrwxrwxrwx 1 root root 7 2009-03-23 17:33 bin/fsck.minix -> busybox
lrwxrwxrwx 1 root root 7 2009-03-23 17:33 sbin/fsck -> busybox
lrwxrwxrwx 1 root root 7 2009-03-23 17:33 sbin/fsck.minix -> busybox

There is no fsck.ext3 symlink to busybox. busybox understands only 'fsck' and 'fsck.minix' commands, that's why. For ext3 and ext2 dump targets, mkdumprd should generate a plain fsck command. In mkdumprd:

                    emit "fsck.$config_opt \$DUMPDEV"

That needs to be changed. If config_opt is ext2 or ext3, only a plain fsck command should be emited. So that's one problem, that should be easy to solve.

But the bigger problem is with busybox's findfs command. The rest of my dumprd environment should be fine. I diffed the init script that comes out with "ext3 /dev/md1" and "ext3 LABEL=/", and except for the DUMPDEV setting, it's the same, so it should boot with the same environment.

Something goes wrong with busybox's findfs command, it emits /dev/md0 instead of /dev/md1, in the dumprd environment, and init attempts to write out the crash dump in the wrong partition. After booting my normal kernel, with the system being up normally, busybox gives correct results:

[root@octopus sbin]# busybox findfs LABEL=/
/dev/md1


But, in the dumprd image, busybox prints /dev/md0 instead. That's as far as I can figure things out.

Yeah, not having fsck.ext3 on board is a problem, although the answer should be to turn it on for busybox (it was in RHEL5 when I developed the code).  I would recomment opening a separate bug to request that the busybox maintainer enable it again, along with ext2.

As for the labeling thing, I think thats also likely a separate issue, most likely one involving lavbels or quoting.  Try modifying this line:
emit "  DUMPDEV=\`findfs \$DUMPDEV\`"
to look like this:
emit "  DUMPDEV=\`findfs \"\$DUMPDEV\"\`"
in the mkdumprd script.

Bug 492040 opened for busybox.

This change made no difference. I added 'set -v' and 'set -x', then unpacked the resulting initrd image and verified that the relevant part of its initrd script read:

echo Saving to the local filesystem LABEL=/
DUMPDEV=LABEL=/
IS_LABEL=`echo $DUMPDEV | grep LABEL`
IS_UUID=`echo $DUMPDEV | grep UUID`
set -v
set -x
if [ -n "$IS_LABEL" -o -n "$IS_UUID" ] 
then
  DUMPDEV=`findfs "$DUMPDEV"`
fi
fsck.ext3 $DUMPDEV
mount -t ext3 $DUMPDEV /mnt
sleep 30


The sleep command allowed me to take another fuzzy screen cap.

The screen cap shows that 'findfs LABEL=/' still emits "/dev/md0" when it should be /dev/md1. See comment #27 and comment #31.

So:

* when the regular kernel boots, findfs LABEL=/ correctly emits /dev/md1

* inside the initrd kernel boot, findfs LABEL=/ incorrectly emits /dev/md0

* inside the initrd kernel boot, the /dev/mdXX partition appear to match what they are when the regular kernel boots. The screen caps cuts it off at the top, but mdadm succesfully assembled md0 through md3, same as when the regular kernel boots. init subsequently tries to save the kernel crash dump onto {/dev/md0}/var/crash. After rebooting into the regular kernel, I find a partial dump in /boot/var/crash.

====

Perhaps this is also something inside busybox that goes wrong inside the initrd environment, or something's in the initrd kernel is putting busybox on the wrong track, I have no idea. I'll try to comb through busybox's source in search of some enlightenment, when I get a chance.

Created attachment 336577 [details]
Results of "ext3 LABEL=/" in kdump.conf, take 2.

easy way to check that is to add /sbin/findfs to the extra_bins section of kdump.conf.  That will use the e2fsprogs version of the command rather than the busybox version when searching during a kdump.  Don't forget to restart the kdump service so your initrd gets rebuilt

Some progress to report.

The native findfs identified the correct volume. With the native findfs in the kdump initrd, 'findfs LABEL=/' produced /dev/md1, which resulted in a good crash dump.

Additional notes:

* Adding /sbin/findfs to extra_bins didn't actually work. This is because, when building the kdump initrd, mkdumprd precreates all the busybox symlinks first. It creates "sbin/findfs -> busybox" and "bin/findfs -> busybox" symlinks first, before copying extra bins into the initrd. The results are rather predictable.

Suggested fix to mkdumprd:

@@ -2075,6 +2078,7 @@
     #copy the binaries and their shared libraries to the archive
     for n in $bin $kdump_libs $k_extras; do
         mkdir -p $MNTIMAGE/`dirname $n`
+	rm -f $MNTIMAGE/$n
         cp $n $MNTIMAGE/$n
     

That still won't be enough, because there's both sbin/findfs and bin/findfs links. So, in order to make sure that the standalone findfs got invoked, I added both /sbin/findfs and /bin/findfs to extra_bins (after installing a temporary hardlink in my real /bin, of course). Some creating scripting is probably possible to fix this properly

* So, it looks like that the native findfs works, and busybox's findfs built-in gets it wrong. Still, it might be premature to write this off as a busybox bug. There could still be something in the kdump initrd environment that leads busybox astray. Observe that I demonstrated, in a previous comment, that after a normal boot with a regular kernel, busybox's findfs builtin returns correct results. It's still worth looking at busybox's code to find out how it implements findfs.

Well, it looks like the busybox code is lifted almost verbatim from the e2fsprogs code, so I'm not sure what would be changing here, although I agreee, whatever it is, its likely environmental.  Looking at the code my next guess is that this would have to do with ordering.  I wonder if perhaps you have a /etc/blkid/blkid.tab file in your root filesystem that attributes the label your looking for to md0 rather than md1.  Its possible that you have the same label on two devices, and thats confusing busybox, which I does have some differences in how the cache file is created.  Please try to issue this command:
blkid LABEL=/
from your normal kernel
that should return all devices which have that label on them, and should confirm for us if /dev/md1 actually does hold that label.

I am not familiar with blkid.tab. I never fiddled with it.

This is the only OS on this server, there are no other partitions.

blkid LABEL=/ does not produce any output, but perusing blkid's man page, it looks like the correct command is really "blkid -t LABEL=/", which gives the correct output for me:

[root@octopus ~]# blkid -t LABEL=/
/dev/md1: UUID="d15371a0-1488-40b9-8b3b-81eb6b2613f6" TYPE="ext3" LABEL="/" SEC_TYPE="ext2" 

Here's everything that I have:

[root@octopus blkid]# blkid /dev/md0
/dev/md0: UUID="435c46f4-8bbf-4cb8-8272-43ee862e3a23" TYPE="ext3" LABEL="/boot" SEC_TYPE="ext2" 
[root@octopus blkid]# blkid /dev/md1
/dev/md1: UUID="d15371a0-1488-40b9-8b3b-81eb6b2613f6" TYPE="ext3" LABEL="/" SEC_TYPE="ext2" 
[root@octopus blkid]# blkid /dev/md2
/dev/md2: TYPE="swap" LABEL="swap" UUID="5f582ab3-8a31-441d-b0a8-b690b1698a2a" 
[root@octopus blkid]# blkid /dev/md3
/dev/md3: UUID="3dd140f4-a78f-49a7-9a26-1f622ae82aab" TYPE="ext3" LABEL="/home" SEC_TYPE="ext2" 

The only other thing I can tell you is that initially Fedora was installed without partition labels. I e2label-ed and re-mkswap-ed all of the above, and fixed up fstab and kernel boot command line, accordingly. But that was a while ago.

hmm, what kind of raid array is md1?  IIRC, some softraid modes simply copy labels to each of their separate partitions, while some do not.  I wonder if, during install, anaconda takes that into account and labels the underlying devices rather than the softraid device itself, to be safe.  Perhaps, if you relabeled post-install, you only labeled one device.  If the raid array gets assembeled in a different order during kdump boot than it does on regular boot, you might see a problem like this.  I wonder if you would run blkid on the physical devices that make up /dev/md1 in your system.  I would be interested to know if both produce the label "/", or if only one does

and, regarding my use of the term partition, I'm using it in its strict sense.  as in /dev/hda1 and /dev/hda2 are separate partitions on the same device

Neither. All the raid volumes are raid-1:

[root@octopus ~]# mdadm --detail /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Fri Aug 15 11:57:11 2008
     Raid Level : raid1
     Array Size : 20972736 (20.00 GiB 21.48 GB)
  Used Dev Size : 20972736 (20.00 GiB 21.48 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Mon Mar 30 20:43:40 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : dc2d6543:e9a8ac2d:623ad5dc:9716c58c
         Events : 0.32

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2


[root@octopus ~]# blkid /dev/sda2
/dev/sda2: UUID="43652ddc-2dac-a8e9-dcd5-3a628cc51697" TYPE="mdraid" 
[root@octopus ~]# blkid /dev/sdb2
/dev/sdb2: UUID="43652ddc-2dac-a8e9-dcd5-3a628cc51697" TYPE="mdraid" 
[root@octopus ~]# 

/dev/md0 consists of /dev/sda1 and /dev/sdb1

[root@octopus ~]# blkid /dev/sda1
/dev/sda1: UUID="84fe2f1f-8a7e-94be-bc15-89126544626a" TYPE="mdraid" 
[root@octopus ~]# blkid /dev/sdb1
/dev/sdb1: UUID="84fe2f1f-8a7e-94be-bc15-89126544626a" TYPE="mdraid" 
[root@octopus ~]# 

I get the same results if I specify -c /dev/null to blkid (see its man page).

The purpose of specifying a label is to still end up getting the right partition, if they get assembled in a different order.

However, recall that specifying "ext3 /dev/md1" results in the dump being saved in the right partition. Furthermore, recall that after LABEL=/ was incorrectly located on /dev/md0, the partial crash was saved there before the partition quickly filled up. /dev/md0 is the small /boot partition. After rebooting, I manually cleared the partial crash dump from /boot. This tells me that initrd assembled at least /dev/md0 and /dev/md1 mapping them to the same underlying partitions that the regular kernel did.


I'm looking at the following code in busybox:

char *get_devname_from_label(const char *spec)
{
        struct uuidCache_s *uc;
        int spec_len = strlen(spec);

        uuidcache_init();
        uc = uuidCache;
        while (uc) {
// FIXME: empty label ("LABEL=") matches anything??!
                if (uc->label[0] && strncmp(spec, uc->label, spec_len) == 0) {
                        return xstrdup(uc->device);
                }
                uc = uc->next;
        }
        return NULL;
}


/dev/md0's label is "/boot". /dev/md1's label is "/", and that's what I'm asking for: "LABEL=/". Looks to me like that loop iterates over /dev/md0 first, and because strncmp("/", "/boot", 1) == 0, it incorrectly returns /dev/md0. Edge condition.

I looked at the code in e2fsprogs, and it looks different. The corresponding function is

extern blkid_dev blkid_find_dev_with_tag(blkid_cache cache,
                                         const char *type,
                                         const char *value)

type would be "LABEL" and value would be the value, in this case "/". That code is different.

To test this theory, I changed the label on /dev/md1 to "/root", fixed up fstab and grub, rebooted, and put "ext3 LABEL=/root" in kdump.conf. This worked.

Grabbed busybox 1.13.2 from rawhide. Put the original "/" back on /dev/md1, rolled back the changed to fstab, menu.lst, rebooted, put "ext3 LABEL=/" in kdump.conf. This time, busybox 1.13.2 wanted to save the crash dump in /dev/md3, whose label is "/home" :-). Bug still exists in busybox 1.13.2

So, at least this part of the bug needs to be fixed upstream in busybox. I opened a bug upstream.

But there's still a kexec issue here:

1) Couple of minor fixes to mkdumprd:

-                    emit "  DUMPDEV=\`findfs \$DUMPDEV\`"
+                    emit "  DUMPDEV=\`findfs \"\$DUMPDEV\"\`"

and

     #copy the binaries and their shared libraries to the archive
     for n in $bin $kdump_libs $k_extras; do
         mkdir -p $MNTIMAGE/`dirname $n`
+	rm -f $MNTIMAGE/$n
         cp $n $MNTIMAGE/$n
     done

2) I still need an explicit "extra_modules sata_nv". mkdumprd still can't find my actual block drivers by itself.

So, that's the kexec portion of this bug.

good find!  Ok, so it sounds like all the busybox bits have appropriate bugs open.  I'll check in the mkdumprd changes on this bug shortly.

fixed in 1.102pre-61.el5.  Thanks!

There's an upstream fix for findfs apparently, I'll try to test it sometime soon.

https://bugs.busybox.net/show_bug.cgi?id=249

The fsck.ext[23] command is WONTFIX, bug 492040, so a workaround in mkdumprd is needed for that one, too.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1258.html

Tested kexec-tools-2.0.0-14.fc11.x86_64 -- it does not fix this bug. I must still explicitly put "extra_modules sata_nv" in order to generate a working initrd.

Looks like you tested a fedora package. This issue should be resolve in RHEL 5.4 kexec-tools-1.102pre-77.el5

Please verify the issue using the latest errata release, which should be available on RHN today. See comment #47 above.

If you find the issue needs to be fixed in Fedora, please file a bug against Fedora. Thanks.

Well, I did file a bug against Fedora. Specifically, this bug. It was opened for Fedora. Somebody else changed it to an RHEL bug. See the bug history.

Okay, i will clone this bug back to fedora then. :-/