Description of problem:
On my Dual 200MHz PPro testbox, with SELinux enabled and using a vanilla
kernel (approximately FC6's kernel), the system will almost always wind up in
the filesystem repair shell.
Version-Release number of selected component (if applicable):
linux-2.6.18-rc2 up to date with git to 27th July 2006.
Almost 100% (it appears timing related), but it requires a slow system to show
Steps to Reproduce:
1. Install vanilla kernel
2. Boot to it
3. Wait for fsck to fail or complete.
System jumps to filesystem repair shell.
System should boot normally.
The problem appears to be udev takes such a long time to run that it gets
backgrounded by the boot procedure. However, shortly after it is
backgrounded, fsck is run. What appears to be happening is that udev hasn't
actually created any block device references at this point, and so fsck goes
searching through all the chardev lists in /sys - of which there are a lot,
since each tty dev entry points back to the list of tty dev entries.
Eventually fsck dies on SIGKILL.
From examining things with strace, I can say:
(1) rc.sysinit isn't SIGKILL'ing fsck.
(2) stracing fsck will cause fsck to succeed - I think because it slows fsck
and thus allows udev to catch up.
(3) fsck goes and does a lot of stat64'ing of chardevs in /sys, indirectly
via /dev/.udev. The paths look like this:
Note that each of four tty:ttyS[0-3] is iterated through, as are all
584 "?ty??" in the first subsystem directory, and also in the second
subdirectory (that's recursive through symlinkage). This is on the order of
1.4 million chardevs.
Using gdb shows the search is being conducted in libblkid from
I've upgraded by testbox from an early FC5 to the latest FC5 and that doesn't
change the problem.
As I said above, I think the root of the problem is that udev takes so long to
run (judging by the way the PIDs crank it's running nearly 2000 programs), and
this is a problem on a slow machine.
Running fsck with the same parameter list once the repair shell is available
works almost instantly.
fsck -T -t noopts=_netdev -A -a -C
I'm not sure whether this belongs against the udev, initscripts or e2fsprogs
packages, but I think the first has to be the major culprit: udev needs to be
faster or optional.
Searching /dev for device nodes that way can't really work, it's a weird
concept, and in this implementation obviously broken. (For that reason, on SUSE,
we patched mount and fsck to use libvolume_id provided by udev.)
However, shouldn't having udevsettle in the udev start procedure handle this
for any reasonably local devices (IDE, SCSI)?
Hmm, isn't this the problem:
"3) fsck goes and does a lot of stat64'ing of chardevs in /sys"
"This is on the order of 1.4 million chardevs."
What do you mean? What should udevsettle handle?
Maybe I misread; I though the problem was:
udev starts, starts coldplug, start scripts exit
<loads scsi, sata, etc driver>
<disk scan starts>
fsck runs, can't find device
<disk scan finishes>
and udevsettle would help with that. Perhaps I missed the actual problem here.
Oh right, looks like. You require a newer udev, or backport udevsettle for that.
Udevd needs to export the current seqnum, which udevsettle can compare against
the actual kernel number to see if the kernel has events in the queue and not
only in the udev daemon queue. Maybe that's the reason for the failure?