From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050302 Firefox/1.0.1 Fedora/1.0.1-1.3.2 Description of problem: On my Fujitsu P2120 (Transmeta Crusoe processor 933MHz, Ali chipset), booting up FC3 takes over 2 minutes from poweron to the console login prompt (Text mode). A significant amount of time (~ 1 minute) is spent on udev initialization and the hardware detection and initialization process (network audio ...). This does not occur with other processors, e.g., desktop P3 1 GHz. The bootup time increases to over 5 minutes if SELinux is enabled. Version-Release number of selected component (if applicable): udev-039-10.FC3.6 How reproducible: Always Steps to Reproduce: 1. Power on notebook 2. Time duration for bootup process 3. Additional info:
You could try a newer version of udev, which resolved some kind of timing problems. Not yet available: http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/Fedora/RPMS/udev-057-3.i386.rpm You may have to wait until tomorrow until the server is synced. How long is the time between "Starting udev..." and "Initializing hardware..." ??
I've installed udev-057-3, it does not make any significant difference. The timing was done from the "Starting udev..." to the [OK] after "Initializing hardware...". It is currently 49 seconds. However, the slight improvement I believe was due to upgrading the kernel from 2.6.10-xxx to 2.6.11-1.14_FC3. Udev initialization by itself usually takes 2-5 seconds (seems faster with udev-057). The delay is apparent during the Initializing hardware stage, each of the three categories of hardware devices probed took 5-10 seconds each, while the remaining wait was for the system to get to the "done" stage.
do you know shell syntax? If yes, you could modify /etc/rc.sysinit. change: load_module () { LC_ALL=C fgrep -xq "$1" /etc/hotplug/blacklist 2>/dev/null || modprobe $ 1 >/dev/null 2>&1 } to: load_module () { LC_ALL=C fgrep -xq "$1" /etc/hotplug/blacklist 2>/dev/null || (echo "Loading module $1"; modprobe $1) } This will display the "Initializing hardware..." section in more detail, so we may find a module, which takes very long to load.
I've made the changes, and these are the stopwatch readings I get: [Initializing] udev @ 0 s (takes about 2-3 s) [Hardware detection] floppy @ 5 s (no device) eth0 @ 10 s eth1 @ 11 s (no device) snd-drv @ 16 s (ali chipset driver) ohci-hcd @ 23 s ehci-hcd @ 25 s ohci-hcd @ 31 s ohci-hcd @ 34 s yenta @ 36 s ohci1394 @ 44 s done @ 49 s It would seem that the listed entries were those that were probed? Does udev spend time on unprobed modules as well? BTW, I have to start my kernel with acpi_irq_isa=10,11 as otherwise ACPI would assign the IRQs incorrectly for one of my USB ports. I don't know if it'll make any difference to the hardware probing time by udev though.
udev only runs in the background after module loading. To be sure "udev" is not the one wasting time, move /etc/hotplug.d/default/10-udev.hotplug to /etc/hotplug.d/default/10-udev.hotplug.bak and reboot. If there is no significant speedup, then it is no udev issue, but a kernel or kudzu one. So please do the renaming of that file temporarily (expect some missing devices for sound and floppy) and report, if that makes any difference. If not, this bug should be reassigned to component kudzu or kernel.
after the test you need to re-rename /etc/hotplug.d/default/10-udev.hotplug.bak back to /etc/hotplug.d/default/10-udev.hotplug . For clarification of the "Initializing hardware" stage: "kudzu" determines which module to load, "modprobe"s it, the "kernel" does the module stuff, "hotplug" gets called for any devices recognized, "hotplug" calls "udev", udev scans /sys for the drivers device node informations and 'mknod's the /dev/xxx device file. So much for the process. I doubt udev is the one which takes so long.
I tried the udev disabling test in Comment #5, and it shaved about 5 seconds off the total time. The process now completes in 43 - 44 s as opposed to 49 s previously. How would I check the boot timings for kudzu? 'time kudzu' from the console gave: real 0m5.944s user 0m2.183s sys 0m0.151s Subsequent timings range for real is from 5.225s - 5.752s. user 2.128s - 2.165s, sys 0.136s - 2.165s. From reading your description of the hardware initialization process, I'm not sure whether the HD in my notebook would have any impact. It's a default 2.5 inch 40GB Toshiba notebook HD (I think it's 5400 rpm). idebus=66 is passed to the kernel. hdparm /dev/hda shows that DMA is enabled. hdparm -t /dev/hda gives 22.39 MB/sec.
adding kudzu and kernel owners on CC to comment
There were some post-FC3 fixes for kudzu that should save that down to ~2 seconds. That doesn't account for the entirety of the slowness, though. Not sure why the module loading would be so slow.
I see a kudzu-1.1.116-1.i386.rpm available. However, the binary is dependent on Python 2.4. Would I be able to rebuild the SRPM for FC3?
Yes. You'll probably just want to install the new kudzu-devel, and then build kmodule (from initscripts) against it.
I've rebuilt and installed kudzu & kudzu-devel 1.1.116-1, as well as built initscripts 8.10-1. I copied kmodule from the BUILD directory to /sbin, and the udev startup + hardware detection time is now about 55 s (4 seconds less). This is for normal udev hotplug detection (i.e., normal operation). From the console, time kudzu gives: real 0m3.671s user 0m0.608s sys 0m0.197s So, there's some improvement, but still not overly dramatic. I wonder if the interpreted scripts used for hotplugging is a major cause of slowdown on Crusoe processors. (Is the hotplug-ng effort announced by Greg KH recently a viable replacement?) BTW, Is it safe for me to upgrade my initscripts on FC3 from 7.93.7 to 8.10? I'd prefer to keep my RPMS consistent if possible.
reassinging to kernel for more comments
are you seeing this with all kernel versions, including the latest errata ? if you boot with profile=1, you'll be able to run readprofile which will tell you where the kernel has been spending most of its time, please attach that to the bug, as that might give some clues.
Created attachment 114237 [details] Profile for FC3 bootup on P2120 (slow hardware initialization)
I experienced the slowdown since the first time I installed FC3. I sync up with FC3 updates pretty frequently (once a week minimum). I don't really know how to interpret the output from readprofile though. Is this the same output used by the graphical bootup visualization tool?
Comment on attachment 114237 [details] Profile for FC3 bootup on P2120 (slow hardware initialization) Profile for kernel 2.6.11-1.14_FC3
theres nothing obvious in that profile at all to finger the kernel as taking up excessive amounts of time. I think this is just down to a combination of udev opening a huge number of files coupled with old slow hardware.
While it is true that the P2120 is 'old slow hardware', the system was doing fine under RH9. My personal (unsustantiated) guess is that the x86 segment limits (NX prototection approximation) code does not run optimally on Transmeta Crusoe processors. I based this guess on my observation that the transition from WinXP -> WinXP SP2 resulted in a significant slowdown as well. However, the slowdown experienced in the transition from RH9 -> FC3 is much more apparent and (subjectively) to a much greater degree in comparison.
udev didn't exist at all on RH9, so the statement is certainly valid that that could be the cause.
the segment limitation shouldnt affect performance at all. This can be tested by booting with exec-sheild=0
(I'm assuming the correct spelling is exec-shield) Booting with exec-shield=0 cuts down the time to 25 s (udev startup & hardware detection time). The time taken by udev statup remains the same, but the hardware detection process is now twice as fast. Overall the system is faster as well (to be expected). It would seem that the Crusoe has pretty weak performance for handling segment limits checks. Darn. The dilemma of choosing between system security vs. speed & usability.
reassigning to kernel
wow, that is quite an impact, a lot more than I would have expected. Unfortunatly, I doubt there's anything we can do, other than recommend disabling exec-shield if speed is more important to you.
An update has been released for Fedora Core 3 (kernel-2.6.12-1.1372_FC3) which may contain a fix for your problem. Please update to this new kernel, and report whether or not it fixes your problem. If you have updated to Fedora Core 4 since this bug was opened, and the problem still occurs with the latest updates for that release, please change the version field of this bug to 'fc4'. Thank you.
I no longer have a FC3 installation on my P2120. In FC4 kernel-2.6.12-1.1390_FC4.i686.rpm, the hardware initialization time with exec-shield=0 is approximately 21 seconds. This is slightly improved over FC3 udev and hardware detection time.
Ok, as FC4 & FC3 share the same kernel sources, we can close this. At some point, we'll make it automatically disable parts of exec-shield that incur the performance hit, whilst leaving others enabled.