Bug 150545 - Crusoe slow with exec-shield
Crusoe slow with exec-shield
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
3
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Dave Jones
Brian Brock
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2005-03-07 23:28 EST by TC
Modified: 2015-01-04 17:17 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-08-26 01:47:22 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Profile for FC3 bootup on P2120 (slow hardware initialization) (93.79 KB, text/plain)
2005-05-11 01:10 EDT, TC
no flags Details

  None (edit)
Description TC 2005-03-07 23:28:39 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050302 Firefox/1.0.1 Fedora/1.0.1-1.3.2

Description of problem:
On my Fujitsu P2120 (Transmeta Crusoe processor 933MHz, Ali chipset), booting up FC3 takes over 2 minutes from poweron to the console login prompt (Text mode).
A significant amount of time (~ 1 minute) is spent on udev initialization and the hardware detection and initialization process (network audio ...).

This does not occur with other processors, e.g., desktop P3 1 GHz. 

The bootup time increases to over 5 minutes if SELinux is enabled.

Version-Release number of selected component (if applicable):
udev-039-10.FC3.6

How reproducible:
Always

Steps to Reproduce:
1. Power on notebook
2. Time duration for bootup process
3.
  

Additional info:
Comment 1 Harald Hoyer 2005-04-27 06:03:39 EDT
You could try a newer version of udev, which resolved some kind of timing problems.
Not yet available:
http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/Fedora/RPMS/udev-057-3.i386.rpm
You may have to wait until tomorrow until the server is synced.

How long is the time between "Starting udev..." and "Initializing hardware..." ??
Comment 2 TC 2005-04-28 00:03:05 EDT
I've installed udev-057-3, it does not make any significant difference.

The timing was done from the "Starting udev..." to the [OK] after "Initializing
hardware...". 

It is currently 49 seconds. However, the slight improvement I believe was due to
upgrading the kernel from 2.6.10-xxx to 2.6.11-1.14_FC3.

Udev initialization by itself usually takes 2-5 seconds (seems faster with
udev-057).

The delay is apparent during the Initializing hardware stage, each of the three
categories of hardware devices probed took 5-10 seconds each, while the
remaining wait was for the system to get to the "done" stage.

Comment 3 Harald Hoyer 2005-04-28 04:57:29 EDT
do you know shell syntax? If yes, you could modify /etc/rc.sysinit.

change:
load_module () {
        LC_ALL=C fgrep -xq "$1" /etc/hotplug/blacklist 2>/dev/null || modprobe $
1 >/dev/null 2>&1
}

to:
load_module () {
        LC_ALL=C fgrep -xq "$1" /etc/hotplug/blacklist 2>/dev/null || (echo
"Loading module $1"; modprobe $1) 
}

This will display the "Initializing hardware..." section in more detail, so we
may find a module, which takes very long to load.
Comment 4 TC 2005-04-29 04:24:55 EDT
I've made the changes, and these are the stopwatch readings I get:

[Initializing]
udev     @ 0 s  (takes about 2-3 s)
[Hardware detection]
floppy   @ 5 s  (no device)
eth0     @ 10 s
eth1     @ 11 s (no device)
snd-drv  @ 16 s (ali chipset driver)
ohci-hcd @ 23 s
ehci-hcd @ 25 s
ohci-hcd @ 31 s
ohci-hcd @ 34 s
yenta    @ 36 s
ohci1394 @ 44 s
done     @ 49 s

It would seem that the listed entries were those that were probed? Does udev
spend time on unprobed modules as well?

BTW, I have to start my kernel with acpi_irq_isa=10,11 as otherwise ACPI would
assign the IRQs incorrectly for one of my USB ports. I don't know if it'll make
any difference to the hardware probing time by udev though.
Comment 5 Harald Hoyer 2005-04-29 07:10:53 EDT
udev only runs in the background after module loading.
To be sure "udev" is not the one wasting time, move
/etc/hotplug.d/default/10-udev.hotplug to
/etc/hotplug.d/default/10-udev.hotplug.bak and reboot. If there is no
significant speedup, then it is no udev issue, but a kernel or kudzu one.

So please do the renaming of that file temporarily (expect some missing devices
for sound and floppy) and report, if that makes any difference.

If not, this bug should be reassigned to component kudzu or kernel.
Comment 6 Harald Hoyer 2005-04-29 07:21:36 EDT
after the test you need to re-rename /etc/hotplug.d/default/10-udev.hotplug.bak
 back to /etc/hotplug.d/default/10-udev.hotplug . 
For clarification of the "Initializing hardware" stage: "kudzu" determines which
module to load, "modprobe"s it, the "kernel" does the module stuff, "hotplug"
gets called for any devices recognized, "hotplug" calls "udev", udev scans /sys
for the drivers device node informations and 'mknod's the /dev/xxx device file.
So much for the process. I doubt udev is the one which takes so long.
Comment 7 TC 2005-04-29 22:49:55 EDT
I tried the udev disabling test in Comment #5, and it shaved about 5 seconds off
the total time. The process now completes in 43 - 44 s as opposed to 49 s
previously.

How would I check the boot timings for kudzu?

'time kudzu' from the console gave:
real 0m5.944s
user 0m2.183s
sys  0m0.151s

Subsequent timings range for real is from 5.225s - 5.752s. user 2.128s - 2.165s,
sys 0.136s - 2.165s.

From reading your description of the hardware initialization process, I'm not
sure whether the HD in my notebook would have any impact. It's a default 2.5
inch 40GB Toshiba notebook HD (I think it's 5400 rpm). idebus=66 is passed to
the kernel.

hdparm /dev/hda shows that DMA is enabled. hdparm -t /dev/hda gives 22.39 MB/sec.

Comment 8 Harald Hoyer 2005-05-02 04:57:33 EDT
adding kudzu and kernel owners on CC to comment
Comment 9 Bill Nottingham 2005-05-02 12:14:18 EDT
There were some post-FC3 fixes for kudzu that should save that down to ~2
seconds. That doesn't account for the entirety of the slowness, though. Not sure
why the
module loading would be so slow.
Comment 10 TC 2005-05-04 00:36:59 EDT
I see a kudzu-1.1.116-1.i386.rpm available. However, the binary is dependent on
Python 2.4. Would I be able to rebuild the SRPM for FC3?
Comment 11 Bill Nottingham 2005-05-04 00:56:03 EDT
Yes. You'll probably just want to install the new kudzu-devel, and then build
kmodule (from initscripts) against it.
Comment 12 TC 2005-05-04 05:05:06 EDT
I've rebuilt and installed kudzu & kudzu-devel 1.1.116-1, as well as
built initscripts 8.10-1. I copied kmodule from the BUILD directory to /sbin,
and the udev startup + hardware detection time is now about 55 s (4 seconds
less). This is for normal udev hotplug detection (i.e., normal operation).

From the console, time kudzu gives:
real 0m3.671s
user 0m0.608s
sys  0m0.197s

So, there's some improvement, but still not overly dramatic. I wonder if the
interpreted scripts used for hotplugging is a major cause of slowdown on Crusoe
processors. (Is the hotplug-ng effort announced by Greg KH recently a viable
replacement?)

BTW, Is it safe for me to upgrade my initscripts on FC3 from 7.93.7 to 8.10? I'd
prefer to keep my RPMS consistent if possible.
Comment 13 Harald Hoyer 2005-05-04 08:00:26 EDT
reassinging to kernel for more comments
Comment 14 Dave Jones 2005-05-09 20:06:24 EDT
are you seeing this with all kernel versions, including the latest errata ?

if you boot with profile=1, you'll be able to run readprofile which will tell
you where the kernel has been spending most of its time, please attach that to
the bug, as that might give some clues.
Comment 15 TC 2005-05-11 01:10:31 EDT
Created attachment 114237 [details]
Profile for FC3 bootup on P2120 (slow hardware initialization)
Comment 16 TC 2005-05-11 01:12:45 EDT
I experienced the slowdown since the first time I installed FC3.
I sync up with FC3 updates pretty frequently (once a week minimum).

I don't really know how to interpret the output from readprofile though.
Is this the same output used by the graphical bootup visualization tool?
Comment 17 TC 2005-05-11 01:14:18 EDT
Comment on attachment 114237 [details]
Profile for FC3 bootup on P2120 (slow hardware initialization)

Profile for kernel 2.6.11-1.14_FC3
Comment 18 Dave Jones 2005-05-11 01:54:50 EDT
theres nothing obvious in that profile at all to finger the kernel as taking up
excessive amounts of time.  I think this is just down to a combination of udev
opening a huge number of files coupled with old slow hardware.
Comment 19 TC 2005-05-11 02:07:59 EDT
While it is true that the P2120 is 'old slow hardware', the system was doing
fine under RH9. My personal (unsustantiated) guess is that the x86 segment
limits (NX prototection approximation) code does not run optimally on Transmeta
Crusoe processors.

I based this guess on my observation that the transition from WinXP -> WinXP SP2
resulted in a significant slowdown as well. However, the slowdown experienced in
the transition from RH9 -> FC3 is much more apparent and (subjectively) to a
much greater degree in comparison.
Comment 20 Bill Nottingham 2005-05-11 15:21:18 EDT
udev didn't exist at all on RH9, so the statement is certainly valid that that
could be the cause.
Comment 21 Dave Jones 2005-05-15 21:17:09 EDT
the segment limitation shouldnt affect performance at all. This can be tested by
booting with exec-sheild=0
Comment 22 TC 2005-05-16 00:01:52 EDT
(I'm assuming the correct spelling is exec-shield)

Booting with exec-shield=0 cuts down the time to 25 s (udev startup & hardware
detection time). The time taken by udev statup remains the same, but the
hardware detection process is now twice as fast.

Overall the system is faster as well (to be expected).
 
It would seem that the Crusoe has pretty weak performance for handling segment
limits checks.

Darn. The dilemma of choosing between system security vs. speed & usability.
Comment 23 Harald Hoyer 2005-05-17 05:50:52 EDT
reassigning to kernel 
Comment 24 Dave Jones 2005-05-17 21:25:28 EDT
wow, that is quite an impact, a lot more than I would have expected.

Unfortunatly, I doubt there's anything we can do, other than recommend disabling
exec-shield if speed is more important to you.
Comment 25 Dave Jones 2005-07-15 15:45:26 EDT
An update has been released for Fedora Core 3 (kernel-2.6.12-1.1372_FC3) which
may contain a fix for your problem.   Please update to this new kernel, and
report whether or not it fixes your problem.

If you have updated to Fedora Core 4 since this bug was opened, and the problem
still occurs with the latest updates for that release, please change the version
field of this bug to 'fc4'.

Thank you.
Comment 26 TC 2005-07-17 22:45:40 EDT
I no longer have a FC3 installation on my P2120. 

In FC4 kernel-2.6.12-1.1390_FC4.i686.rpm, the hardware initialization time with
exec-shield=0 is approximately 21 seconds. This is slightly improved over FC3
udev and hardware detection time.
Comment 27 Dave Jones 2005-08-26 01:47:22 EDT
Ok, as FC4 & FC3 share the same kernel sources, we can close this.
At some point, we'll make it automatically disable parts of exec-shield that
incur the performance hit, whilst leaving others enabled.

Note You need to log in before you can comment on or make changes to this bug.