This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 456001 - mkdumprd unable to detect USB root filesystem
mkdumprd unable to detect USB root filesystem
Status: CLOSED DUPLICATE of bug 466392
Product: Fedora
Classification: Fedora
Component: kexec-tools (Show other bugs)
10
All Linux
medium Severity medium
: ---
: ---
Assigned To: Neil Horman
Fedora Extras Quality Assurance
:
: 459542 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-07-20 02:47 EDT by CAI Qian
Modified: 2009-07-28 10:09 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-07-28 10:09:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch to include usb_storage driver (984 bytes, patch)
2008-07-28 15:41 EDT, Neil Horman
no flags Details | Diff
command traces for mkdumprd (132.14 KB, text/plain)
2008-08-10 08:14 EDT, CAI Qian
no flags Details
patch to find block devices in /sys (476 bytes, patch)
2008-08-11 13:42 EDT, Neil Horman
no flags Details | Diff
command traces from with the original patch applied (132.09 KB, text/plain)
2008-08-11 19:20 EDT, CAI Qian
no flags Details
command traces from with the modified patch applied (132.59 KB, text/plain)
2008-08-11 19:21 EDT, CAI Qian
no flags Details
new patch (1.48 KB, patch)
2008-08-12 13:49 EDT, Neil Horman
no flags Details | Diff
new corrected patch (1.73 KB, patch)
2008-08-13 06:52 EDT, Neil Horman
no flags Details | Diff
yet another corrected patch (1.71 KB, patch)
2008-08-14 07:57 EDT, Neil Horman
no flags Details | Diff

  None (edit)
Description CAI Qian 2008-07-20 02:47:13 EDT
Description of problem:
The part of detecting USB root filesystem in mkdumprd is not working at the
moment. It used kudzu to find out USB device, but looks like the latest version
of kudzu uses Python and there is no executable file /sbin/kudzu anymore. As a
result, users have to add necessary USB modules to extra_modules option in kdump
config file manually.

Version-Release number of selected component (if applicable):
kexec-tools-1.102pre-13.fc10.x86_64
kudzu-1.2.85

How reproducible:
Always
Comment 1 Neil Horman 2008-07-28 15:41:30 EDT
Created attachment 312812 [details]
patch to include usb_storage driver

This should include the usb_storage driver automatically for any scsi-over usb
devices. Please test & Confirm.  Thank you!
Comment 2 CAI Qian 2008-08-10 08:05:15 EDT
Still no luck with this patch.
Comment 3 CAI Qian 2008-08-10 08:14:40 EDT
Created attachment 313889 [details]
command traces for mkdumprd

When generated Kdump initramfs file, I got withusb=yes, but it still did not include necessary usb modules. Tested with kexec-tools-1.102pre-15.fc10.x86_64.
Comment 4 Neil Horman 2008-08-10 09:24:50 EDT
Looking at the trace you just attached the findstoragedevice function seemed to not be able to find sda in /sys.  Did you have the usb device plugged in when when you ran makedumpfile, and did it attach as device sda?
Comment 5 CAI Qian 2008-08-10 10:02:52 EDT
Yes, my rootfs is on a usb disk. Looking though /sys, there are the following entries associated with sda,

$ find /sys -name sda*
/sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda
/sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1
/sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda2
/sys/devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda3
/sys/class/block/sda
/sys/class/block/sda1
/sys/class/block/sda2
/sys/class/block/sda3
/sys/block/sda
Comment 6 Neil Horman 2008-08-11 08:48:02 EDT
ok, I wonder if something is going wrong witht he findone command in mkdumprd then.  Could you please on your system issue the following command:
echo nash-find -type d /sys/block -name sda | /sbin/nash --force --quiet

This should tell us if the findone command is having trouble with sysfs directories for some reason.  Thanks!
Comment 7 CAI Qian 2008-08-11 10:04:20 EDT
Nothing returned.
Comment 8 Neil Horman 2008-08-11 13:42:14 EDT
Created attachment 313986 [details]
patch to find block devices in /sys

Funny, I wonder if something has gotten borked in nash recently.  No matter, this patch (which whould apply on top of the oter previous patch) should use find to find things in sys like you normal would.  I dont' have a usb stick handy at the moment, so could you please confirm that it works for me?  Thanks!
Comment 9 CAI Qian 2008-08-11 19:12:29 EDT
Still no luck. From command traces,

++ /usr/bin/find -type d /sys/block -name sda
/usr/bin/find: paths must precede expression: /sys/block
Usage: /usr/bin/find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]

Then, I changed the patch slightly.

/usr/bin/find /sys/block -name sda -type d

Still not work, because of /sys/block/sda is a symbolic link,

# ls -l /sys/block/sda
lrwxrwxrwx 1 root root 0 2008-08-12 06:59 /sys/block/sda -> ../devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host0/target0:0:0/0:0:0:0/block/sda

Finally, I modified the patch this way,

/usr/bin/find /sys/block -name sda -type l

But it failed later at,
++ findone /sys/block -name sda -type l
++ /usr/bin/find /sys/block -name sda -type l
++ /bin/awk '{ print $1; exit; }'
+ sysfs=/sys/block/sda
+ '[' -z /sys/block/sda ']'
+ pushd /sys/block/sda
+ findstoragedriverinsys
+ '[' '!' -L device ']'
++ readlink device
++ echo nash-readlink device
++ grep usb
++ /sbin/nash --force --quiet
error resolving symbolic link device/../../../0:0:0:0: No such file or directory
+ IS_USB=

So, looks like still need more work.
Comment 10 CAI Qian 2008-08-11 19:20:47 EDT
Created attachment 314013 [details]
command traces from with the original patch applied
Comment 11 CAI Qian 2008-08-11 19:21:59 EDT
Created attachment 314015 [details]
command traces from with the modified patch applied
Comment 12 Neil Horman 2008-08-12 08:36:08 EDT
ugh, one more silly exception for device types in sysfs.  grrr.
Is the system you're working on here one I can get access to cai?  It sounds like I'm going  to trudge through a series of errors to fix this.
Comment 13 Neil Horman 2008-08-12 13:01:50 EDT
ugh, this isn't a usb issue at all (I don't think), this is a live image issue.  All the block devices in /sys/block are symlinks on the system you emailed me about.  Whats worse, take a look at this:

[root@walsinh sda]# pwd
/sys/block/sda
[root@walsinh sda]# ls -l
total 0
lrwxrwxrwx 1 root root    0 2008-08-13 00:34 bdi -> ../../../../../../../../../.
./virtual/bdi/8:0
-r--r--r-- 1 root root 4096 2008-08-13 00:34 capability
drwxr-xr-x 2 root root    0 2008-08-13 00:34 cmd_filter
-r--r--r-- 1 root root 4096 2008-08-13 00:34 dev
lrwxrwxrwx 1 root root    0 2008-08-13 00:34 device -> ../../../0:0:0:0

the device link is how we determine if the device is USB or not.  On a normal (non-live-image) system the device link looks like this for USB:

[nhorman@hmsreliant ~]$ cd /sys/block/sdf
[nhorman@hmsreliant sdf]$ ls -l device
lrwxrwxrwx 1 root root 0 2008-08-12 12:51 device -> ../../devices/pci0000:00/0000:00:1d.0/usb1/1-2/1-2:1.0/host10/target10:0:0/10:0:0:0


It gets worse still, note the pwd from your live-image system.   That link shouldn't  work  as ../../.. should return you to root, but it doesnt.  In fact all of the symlinks look to be out of whack.

I think we need to defer this bug until sysfs in live images gets sorted.  I'll open a kernel bug on sysfs and make this bug dependent on that.
Comment 14 Neil Horman 2008-08-12 13:12:28 EDT
scratch that last comment, the symlinks work because the directories in /sys/block themselves are symlinks.   grrrrr
Comment 15 Neil Horman 2008-08-12 13:49:04 EDT
Created attachment 314121 [details]
new patch

Heres a new patch that should roll all the others together and work properly for live images (in whcih everything in /sys/block is s symlink.  I just tried it on your system and it started kdump without error (of course I didn 't test crash the system.  Can you try that out and make sure everything works properly?  Thanks!
Comment 16 CAI Qian 2008-08-12 23:16:51 EDT
Rebuilding /boot/initrd-2.6.27-0.183.rc0.git14.fc10.x86_64kdump.img
find: error: invalid type 'l'
usage: find [-type type] [path [-name file]]
Starting kdump:                                            [  OK  ]

Looks like nash-find did not support -type l.
Comment 17 Neil Horman 2008-08-13 06:52:08 EDT
Created attachment 314187 [details]
new corrected patch

stupid nash.  I had to convert the nash-find calls to find calls and I missed a spot.  Sorry, this is the corrected patch.
Comment 18 CAI Qian 2008-08-13 08:23:17 EDT
I am afraid that there are just too many problems. With your latest patch, 

+ device=sda
++ findone /sys/block -type d -name sda
++ find /sys/block -type d -name sda
++ /bin/awk '{ print $1; exit; }'
+ sysfs=
+ '[' -z '' ']'
++ findone /sys/block -type l -name sda
++ find /sys/block -type l -name sda
++ /bin/awk '{ print $1; exit; }'
+ sysfs=/sys/block/sda
+ '[' -z /sys/block/sda ']'
++ readlink /sys/block/sda
++ echo nash-readlink /sys/block/sda
++ /sbin/nash --force --quiet
+ IS_LINK=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda
+ pushd /sys/block/sda
+ findstoragedriverinsys /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda
+ '[' '!' -L device ']'
++ echo /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda
++ grep usb
+ IS_USB=/sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0/block/sda
++ readlink ./device
++ echo nash-readlink ./device
++ /sbin/nash --force --quiet
+ cd /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host0/target0:0:0/0:0:0:0
+ '[' '!' -f modalias ']'
++ cat modalias
+ modalias=scsi:t-0x00
++ modprobe --set-version 2.6.27-0.183.rc0.git14.fc10.x86_64 --show-depends scsi:t-0x00
++ awk '{ print gensub(".*/","","g",$2) }'
+ popd
+ testdm=
+ '[' -n '' ']'

I checked with Kdump initramfs afterwards, and did not find usb_storage.ko will be loaded there. It looks like we did not enter the following loop in mkdumprd at all,

    280     for driver in $(modprobe --set-version $kernel --show-depends $modalias 2>/dev/null| awk '{ print gensub(".*/","","        g",$2) }'); do
    281         findmodule ${driver%%.ko}
    282         if [ -n "$IS_USB" ]
    283         then
    284             findmodule usb_storage.ko
    285         fi
    286     done

Therefore, I have modified the code a little bit to     

    280     for i in 1; do
    281         # findmodule ${driver%%.ko}
    282         if [ -n "$IS_USB" ]
    283         then
    284             findmodule usb_storage.ko
    285         fi
    286     done

Then, I got this error,

++ findone /lib/modules/2.6.27-0.183.rc0.git14.fc10.x86_64 -name usb_storage.ko.ko
++ find /lib/modules/2.6.27-0.183.rc0.git14.fc10.x86_64 -name usb_storage.ko.ko
++ /bin/awk '{ print $1; exit; }'
+ fmPath=
+ '[' -f '' ']'
+ '[' '!' -f '' ']'
+ '[' -n '' ']'
+ for n in '$PRESCSIMODS'
+ '[' sd_mod = usb_storage.ko ']'
+ '[' -n '' ']'
+ error 'No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.'
+ NONL=
+ '[' 'No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.' == -n ']'
+ echo 'No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.'
No module usb_storage.ko found for kernel 2.6.27-0.183.rc0.git14.fc10.x86_64, aborting.
+ exit 1
Failed to run mkdumprd

Therefore, I did further modification to,

    284             findmodule usb_storage

Then, this time Kdump service finally started.
Comment 19 CAI Qian 2008-08-13 08:49:36 EDT
Hmm, even after all those modification, and Kdump initramfs looks correctly include usb_storge module, capture Kernel still failed to find rootfs, and dropped to a shell. I have had the same testing machine for you if you would like to have a look. Feel free to crash the box if you want (no serial console and remote power control, so you might want to set Kdump default action to reboot).
Comment 20 Neil Horman 2008-08-13 09:12:52 EDT
Ok, please attach the modified patch.

As for the finding of the rootfs, I'm not suprised.  Theres just no way, given the current live image configuration that we're going to be able to locate the rootfs dynamically I think.  You're just going to have to configure /etc/kdump.conf.
Comment 21 CAI Qian 2008-08-14 03:18:26 EDT
I don't have a general working patch yet. All I did were hard-coded modifications which only suit for this particular system. I might look at it in the near future to see if I could come out a working patch.
Comment 22 Neil Horman 2008-08-14 07:57:03 EDT
Created attachment 314309 [details]
yet another corrected patch

Here, give this a try, It looks like I had the check for IS_USB inside the loop when it probably should be outside.  The findmodule that you commented out needs to be there still though, as that is what sucks in the actual specified module for the storage device itself (for usb devices that is scsi_mod)
Comment 23 CAI Qian 2008-08-15 23:28:20 EDT
Still no luck, because other required modules were not loaded, like uhci_hcd,ohci_hcd etc. Actually, it looks like this is the best part to modify,

    652 needusb=""
    653 if [ -n "$withusb" -a "x$PROBE" == "xyes" ]; then
    654     # If / or /boot is on a USB device include the driver. With root by
    655     # label we could still get some odd behaviors
    656     for fs in / /boot ; do
    657         esc=$(echo $fs | sed 's,/,\\/,g')
    658         dev=$(mount | awk "/ on ${esc} / { print \$1 }" | sed 's/[0-9]*$//' | cut -d/ -f3)
    659         if [ "$(echo $dev | cut -c1-2)" = sd ]; then
    660             if [ `which kudzu 2>/dev/null` ]; then
    661                 host=$(kudzu --probe -b scsi |
    662                 gawk '/^device: '${dev}'/,/^host:/ { if (/^host/) { print $2; exit; } }')
    663                 if [ -d /proc/scsi/usb-storage-${host} -o -f /proc/scsi/usb-storage/${host        } ]; then
    664                     needusb=1
    665                 fi
    666             fi
    667         fi
    668     done

If this part works, and needusb becomes 1, almost all necessary modules for USB have been loaded afterwards.

As I said in bug description, this part is not working proper at the moment is that it used kudzu to find out USB device, but the latest version of kudzu (kudzu-1.2.85-1.x86_64) uses Python and there is no executable file /sbin/kudzu anymore. Maybe we should consider using HAL to detect USB rootfs?
Comment 24 Neil Horman 2008-08-20 15:47:01 EDT
Cai, So I made myself a live image to tinker with this, and I just realized that the root filesystem (which we are going to be dumping to isn't actually mounted on the usb stick, rather its a ram disk image that mounted on /dev/mapper/live-rw.  Thats why we're not finding the usb storage drivers there, and I expect why we're not finding the rootfs as well.  Please confirm that this is the case in your environment as well.

If so, I expect what we'll need to do here is add  some code like what we have above to do usb detection (which does seem to work for me btw, if I specify /mnt/live, which is where the usb stick is truly mounted for me in /etc/kdump.conf).  Then we'll have to make  kbase note somewhere indicating that you'll have to configure kdump properly if you want it to work using a live image
Comment 25 CAI Qian 2008-08-20 23:37:07 EDT
Neil, you just found another problem. I am not using a live image, but a normal system with only root filesystem mounted on a USB hard disk partition.

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10317828   5627008   4166704  58% /
tmpfs                  1456200         0   1456200   0% /dev/shm
/dev/sda3             26122692   7236556  17559168  30% /home/qcai

$ dmesg
...
usb-storage: device scan complete
isa bounce pool size: 16 pages
scsi 0:0:0:0: Direct-Access     SAMSUNG  MP0402H          0811 PQ: 0 ANSI: 0
sd 0:0:0:0: [sda] 78242976 512-byte hardware sectors (40060 MB)
sd 0:0:0:0: [sda] Test WP failed, assume Write Enabled
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] 78242976 512-byte hardware sectors (40060 MB)
sd 0:0:0:0: [sda] Test WP failed, assume Write Enabled
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2 sda3
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
...
Comment 26 Neil Horman 2008-08-21 07:06:55 EDT
fantastic.  Ok, back to the drawing board.
Comment 27 Neil Horman 2008-09-02 16:43:54 EDT
*** Bug 459542 has been marked as a duplicate of this bug. ***
Comment 28 Bug Zapper 2008-11-25 21:35:07 EST
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 29 Neil Horman 2009-07-28 10:09:20 EDT
you, know, I'm prett sure at this point, I'm not going to roll these changes into the existing kdump config setup, but this can be covered by the new config setup.  Closing as a dup of that

*** This bug has been marked as a duplicate of bug 466392 ***

Note You need to log in before you can comment on or make changes to this bug.