Description of problem: The part of detecting USB root filesystem in mkdumprd is not working at the moment. It used kudzu to find out USB device, but looks like the latest version of kudzu uses Python and there is no executable file /sbin/kudzu anymore. As a result, users have to add necessary USB modules to extra_modules option in kdump config file manually. Version-Release number of selected component (if applicable): kexec-tools-1.102pre-13.fc10.x86_64 kudzu-1.2.85 How reproducible: Always
Created attachment 312812 [details] patch to include usb_storage driver This should include the usb_storage driver automatically for any scsi-over usb devices. Please test & Confirm. Thank you!
Still no luck with this patch.
Created attachment 313889 [details] command traces for mkdumprd When generated Kdump initramfs file, I got withusb=yes, but it still did not include necessary usb modules. Tested with kexec-tools-1.102pre-15.fc10.x86_64.
Looking at the trace you just attached the findstoragedevice function seemed to not be able to find sda in /sys. Did you have the usb device plugged in when when you ran makedumpfile, and did it attach as device sda?
Yes, my rootfs is on a usb disk. Looking though /sys, there are the following entries associated with sda, $ find /sys -name sda* /sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda /sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1 /sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda2 /sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda3 /sys/class/block/sda /sys/class/block/sda1 /sys/class/block/sda2 /sys/class/block/sda3 /sys/block/sda
ok, I wonder if something is going wrong witht he findone command in mkdumprd then. Could you please on your system issue the following command: echo nash-find -type d /sys/block -name sda | /sbin/nash --force --quiet This should tell us if the findone command is having trouble with sysfs directories for some reason. Thanks!
Nothing returned.
Created attachment 313986 [details] patch to find block devices in /sys Funny, I wonder if something has gotten borked in nash recently. No matter, this patch (which whould apply on top of the oter previous patch) should use find to find things in sys like you normal would. I dont' have a usb stick handy at the moment, so could you please confirm that it works for me? Thanks!
Still no luck. From command traces, ++ /usr/bin/find -type d /sys/block -name sda /usr/bin/find: paths must precede expression: /sys/block Usage: /usr/bin/find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression] Then, I changed the patch slightly. /usr/bin/find /sys/block -name sda -type d Still not work, because of /sys/block/sda is a symbolic link, # ls -l /sys/block/sda lrwxrwxrwx 1 root root 0 2008-08-12 06:59 /sys/block/sda -> ../devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda Finally, I modified the patch this way, /usr/bin/find /sys/block -name sda -type l But it failed later at, ++ findone /sys/block -name sda -type l ++ /usr/bin/find /sys/block -name sda -type l ++ /bin/awk '{ print $1; exit; }' + sysfs=/sys/block/sda + '[' -z /sys/block/sda ']' + pushd /sys/block/sda + findstoragedriverinsys + '[' '!' -L device ']' ++ readlink device ++ echo nash-readlink device ++ grep usb ++ /sbin/nash --force --quiet error resolving symbolic link device/../../../0:0:0:0: No such file or directory + IS_USB= So, looks like still need more work.
Created attachment 314013 [details] command traces from with the original patch applied
Created attachment 314015 [details] command traces from with the modified patch applied
ugh, one more silly exception for device types in sysfs. grrr. Is the system you're working on here one I can get access to cai? It sounds like I'm going to trudge through a series of errors to fix this.
ugh, this isn't a usb issue at all (I don't think), this is a live image issue. All the block devices in /sys/block are symlinks on the system you emailed me about. Whats worse, take a look at this: [root@walsinh sda]# pwd /sys/block/sda [root@walsinh sda]# ls -l total 0 lrwxrwxrwx 1 root root 0 2008-08-13 00:34 bdi -> ../../../../../../../../../. ./virtual/bdi/8:0 -r--r--r-- 1 root root 4096 2008-08-13 00:34 capability drwxr-xr-x 2 root root 0 2008-08-13 00:34 cmd_filter -r--r--r-- 1 root root 4096 2008-08-13 00:34 dev lrwxrwxrwx 1 root root 0 2008-08-13 00:34 device -> ../../../0:0:0:0 the device link is how we determine if the device is USB or not. On a normal (non-live-image) system the device link looks like this for USB: [nhorman@hmsreliant ~]$ cd /sys/block/sdf [nhorman@hmsreliant sdf]$ ls -l device lrwxrwxrwx 1 root root 0 2008-08-12 12:51 device -> ../../devices/pci0000:00/0000:00:1d.0/usb1/1-2/1-2:1.0/host10/target10:0:0/10:0:0:0 It gets worse still, note the pwd from your live-image system. That link shouldn't work as ../../.. should return you to root, but it doesnt. In fact all of the symlinks look to be out of whack. I think we need to defer this bug until sysfs in live images gets sorted. I'll open a kernel bug on sysfs and make this bug dependent on that.
scratch that last comment, the symlinks work because the directories in /sys/block themselves are symlinks. grrrrr
Created attachment 314121 [details] new patch Heres a new patch that should roll all the others together and work properly for live images (in whcih everything in /sys/block is s symlink. I just tried it on your system and it started kdump without error (of course I didn 't test crash the system. Can you try that out and make sure everything works properly? Thanks!
Rebuilding /boot/initrd-2.6.27-0.183.rc0.git14.fc10.x86_64kdump.img find: error: invalid type 'l' usage: find [-type type] [path [-name file]] Starting kdump: [ OK ] Looks like nash-find did not support -type l.
Created attachment 314187 [details] new corrected patch stupid nash. I had to convert the nash-find calls to find calls and I missed a spot. Sorry, this is the corrected patch.
I am afraid that there are just too many problems. With your latest patch, + device=sda ++ findone /sys/block -type d -name sda ++ find /sys/block -type d -name sda ++ /bin/awk '{ print $1; exit; }' + sysfs= + '[' -z '' ']' ++ findone /sys/block -type l -name sda ++ find /sys/block -type l -name sda ++ /bin/awk '{ print $1; exit; }' + sysfs=/sys/block/sda + '[' -z /sys/block/sda ']' ++ readlink /sys/block/sda ++ echo nash-readlink /sys/block/sda ++ /sbin/nash --force --quiet + IS_LINK=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda + pushd /sys/block/sda + findstoragedriverinsys /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda + '[' '!' -L device ']' ++ echo /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda ++ grep usb + IS_USB=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda ++ readlink ./device ++ echo nash-readlink ./device ++ /sbin/nash --force --quiet + cd /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0 + '[' '!' -f modalias ']' ++ cat modalias + modalias=scsi:t-0x00 ++ modprobe --set-version 2.6.27-0.183.rc0.git14.fc10.x86_64 --show-depends scsi:t-0x00 ++ awk '{ print gensub(".*/","","g",$2) }' + popd + testdm= + '[' -n '' ']' I checked with Kdump initramfs afterwards, and did not find usb_storage.ko will be loaded there. It looks like we did not enter the following loop in mkdumprd at all, 280 for driver in $(modprobe --set-version $kernel --show-depends $modalias 2>/dev/null| awk '{ print gensub(".*/",""," g",$2) }'); do 281 findmodule ${driver%%.ko} 282 if [ -n "$IS_USB" ] 283 then 284 findmodule usb_storage.ko 285 fi 286 done Therefore, I have modified the code a little bit to 280 for i in 1; do 281 # findmodule ${driver%%.ko} 282 if [ -n "$IS_USB" ] 283 then 284 findmodule usb_storage.ko 285 fi 286 done Then, I got this error, ++ findone /lib/modules/2.6.27-0.183.rc0.git14.fc10.x86_64 -name usb_storage.ko.ko ++ find /lib/modules/2.6.27-0.183.rc0.git14.fc10.x86_64 -name usb_storage.ko.ko ++ /bin/awk '{ print $1; exit; }' + fmPath= + '[' -f '' ']' + '[' '!' -f '' ']' + '[' -n '' ']' + for n in '$PRESCSIMODS' + '[' sd_mod = usb_storage.ko ']' + '[' -n '' ']' + error 'No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.' + NONL= + '[' 'No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.' == -n ']' + echo 'No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.' No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting. + exit 1 Failed to run mkdumprd Therefore, I did further modification to, 284 findmodule usb_storage Then, this time Kdump service finally started.
Hmm, even after all those modification, and Kdump initramfs looks correctly include usb_storge module, capture Kernel still failed to find rootfs, and dropped to a shell. I have had the same testing machine for you if you would like to have a look. Feel free to crash the box if you want (no serial console and remote power control, so you might want to set Kdump default action to reboot).
Ok, please attach the modified patch. As for the finding of the rootfs, I'm not suprised. Theres just no way, given the current live image configuration that we're going to be able to locate the rootfs dynamically I think. You're just going to have to configure /etc/kdump.conf.
I don't have a general working patch yet. All I did were hard-coded modifications which only suit for this particular system. I might look at it in the near future to see if I could come out a working patch.
Created attachment 314309 [details] yet another corrected patch Here, give this a try, It looks like I had the check for IS_USB inside the loop when it probably should be outside. The findmodule that you commented out needs to be there still though, as that is what sucks in the actual specified module for the storage device itself (for usb devices that is scsi_mod)
Still no luck, because other required modules were not loaded, like uhci_hcd,ohci_hcd etc. Actually, it looks like this is the best part to modify, 652 needusb="" 653 if [ -n "$withusb" -a "x$PROBE" == "xyes" ]; then 654 # If / or /boot is on a USB device include the driver. With root by 655 # label we could still get some odd behaviors 656 for fs in / /boot ; do 657 esc=$(echo $fs | sed 's,/,\\/,g') 658 dev=$(mount | awk "/ on ${esc} / { print \$1 }" | sed 's/[0-9]*$//' | cut -d/ -f3) 659 if [ "$(echo $dev | cut -c1-2)" = sd ]; then 660 if [ `which kudzu 2>/dev/null` ]; then 661 host=$(kudzu --probe -b scsi | 662 gawk '/^device: '${dev}'/,/^host:/ { if (/^host/) { print $2; exit; } }') 663 if [ -d /proc/scsi/usb-storage-${host} -o -f /proc/scsi/usb-storage/${host } ]; then 664 needusb=1 665 fi 666 fi 667 fi 668 done If this part works, and needusb becomes 1, almost all necessary modules for USB have been loaded afterwards. As I said in bug description, this part is not working proper at the moment is that it used kudzu to find out USB device, but the latest version of kudzu (kudzu-1.2.85-1.x86_64) uses Python and there is no executable file /sbin/kudzu anymore. Maybe we should consider using HAL to detect USB rootfs?
Cai, So I made myself a live image to tinker with this, and I just realized that the root filesystem (which we are going to be dumping to isn't actually mounted on the usb stick, rather its a ram disk image that mounted on /dev/mapper/live-rw. Thats why we're not finding the usb storage drivers there, and I expect why we're not finding the rootfs as well. Please confirm that this is the case in your environment as well. If so, I expect what we'll need to do here is add some code like what we have above to do usb detection (which does seem to work for me btw, if I specify /mnt/live, which is where the usb stick is truly mounted for me in /etc/kdump.conf). Then we'll have to make kbase note somewhere indicating that you'll have to configure kdump properly if you want it to work using a live image
Neil, you just found another problem. I am not using a live image, but a normal system with only root filesystem mounted on a USB hard disk partition. $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 10317828 5627008 4166704 58% / tmpfs 1456200 0 1456200 0% /dev/shm /dev/sda3 26122692 7236556 17559168 30% /home/qcai $ dmesg ... usb-storage: device scan complete isa bounce pool size: 16 pages scsi 0:0:0:0: Direct-Access SAMSUNG MP0402H 0811 PQ: 0 ANSI: 0 sd 0:0:0:0: [sda] 78242976 512-byte hardware sectors (40060 MB) sd 0:0:0:0: [sda] Test WP failed, assume Write Enabled sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] 78242976 512-byte hardware sectors (40060 MB) sd 0:0:0:0: [sda] Test WP failed, assume Write Enabled sd 0:0:0:0: [sda] Assuming drive cache: write through sda: sda1 sda2 sda3 sd 0:0:0:0: [sda] Attached SCSI disk sd 0:0:0:0: Attached scsi generic sg0 type 0 ...
fantastic. Ok, back to the drawing board.
*** Bug 459542 has been marked as a duplicate of this bug. ***
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle. Changing version to '10'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
you, know, I'm prett sure at this point, I'm not going to roll these changes into the existing kdump config setup, but this can be covered by the new config setup. Closing as a dup of that *** This bug has been marked as a duplicate of bug 466392 ***