--- Additional comment from dledford on 2009-03-20 11:24:32 EDT --- Made some progress on this. If I change rc.sysinit to have a udevadm trigger call before the udevadm settle call, all works (the udevadm settle call in rc.sysinit is not in the official package yet, but it replaces the mdadm -As line in rc.sysinit right before lvm startup). However, the call to start_udev a few lines above *also* has a udevadm trigger call in it, and that *should* be sufficient. So there is a question as to why calling udevadm trigger *again* only a few lines later works. I'm going to clone this bug against udev in rawhide to see if we can address that part of the issue. The specific thing to note is that the mdadm supplied 65-md-incremental.rules file is *not* being run by the udevadm trigger call in start_udev, but when we make the call a few lines later in rc.sysinit, it does get run and arrays get started properly.
OK, I've isolated the problem, and udev is *not* the culprit, sorry for the false bug report. (The problem is that the / fs is ro at the time udev_start runs and this screws up mdadm which wants to write to /var/run/mdadm/mdadm.map)