Bug 466534 - can't find /dev/root - scsi + smp ???
Summary: can't find /dev/root - scsi + smp ???
Status: CLOSED DUPLICATE of bug 466607
Alias: None
Product: Fedora
Classification: Fedora
Component: mkinitrd
Version: 10
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
Depends On:
Blocks: F10Target
TreeView+ depends on / blocked
Reported: 2008-10-10 19:18 UTC by John Ellson
Modified: 2013-01-10 04:50 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2008-12-10 20:27:03 UTC
Type: ---

Attachments (Terms of Use)
final screen of failed boot sequence (729.80 KB, image/jpeg)
2008-10-11 01:13 UTC, John Ellson
no flags Details
Full text capture of failed boot (17.98 KB, text/plain)
2008-10-11 18:57 UTC, John Ellson
no flags Details
Full text capture of failed boot - mkinitrd-6.0.70-1.fc10.i386 (35.67 KB, text/plain)
2008-11-06 17:11 UTC, John Ellson
no flags Details
make scsi_scan_wait be used by default (395 bytes, text/plain)
2008-11-14 00:55 UTC, Charlie Moschel
no flags Details
Fix to mkinitrd to wait for scsi (872 bytes, patch)
2008-11-18 16:16 UTC, Lubomir Bulej
no flags Details | Diff

Description John Ellson 2008-10-10 19:18:35 UTC
Description of problem:
I have two systems now that won't boot recent kernels.  They hang with:
  Can't find /dev/root

My two sytems are:
  - A quad-cpu, x86_64, with 6 SCSI drives and / on /dev/sda1.
  - A dual-cpu, i686, with / on software raid over 2 SCSI drives.

I think there is an attempt to mount / before the SCSI initialization has completed?

There is also BZ #462233 which looks like the same problem, also SCSI + SMP.

BZ #459109 might be the same problem, but I can't tell if its SMP.

Version-Release number of selected component (if applicable):

The kernel on the F10-Beta-x86_64-Live_CD released Oct 8th, 2008.
Any kernel after about kernel-

How reproducible:

Steps to Reproduce:
1. reboot
Actual results:
Boot hangs looking for /dev/root

Expected results:

Additional info:

Comment 1 Peter Jones 2008-10-10 19:25:49 UTC
Can you add a more complete log, please?

Comment 2 John Ellson 2008-10-10 19:42:32 UTC
Yes.. give me few hours.

BTW  This might be caused by:

   BZ #461850

Comment 3 John Ellson 2008-10-11 01:12:52 UTC
Well that was a really frustrating exercise!  Whats the trick to getting a serial connection running these days!  All the Howtos are useless. /etc/initrd has changed, /var/lock has changed, ....   I had minicom to minicom working briefly, then nothing!  I couldn't get them to do it again.  Is something else locking up /dev/ttyS0?

Anyway, I took some pics instead.   The attached shows boot failing .. followed by sdb waking up.  /dev/sdb is the second drive in the striped pair forming /dev/md0 with the root
filesystem on it.

Perhaps something is waiting only for the first scsi drive response?

My other system at work isn't raid, but it does have 6 scsi drives, so again if something is waiting for the first response only it would fail.

Comment 4 John Ellson 2008-10-11 01:13:52 UTC
Created attachment 320082 [details]
final screen of failed boot sequence

Comment 5 John Ellson 2008-10-11 18:57:52 UTC
Created attachment 320104 [details]
Full text capture of failed boot

[Isn't there a standard that requires ttyS0 on the top connector? !!!  And where is my DB9 breakout box? ]

Finally got the serail cable to work.  Here is the capture of the failed boot.

Comment 6 John Ellson 2008-10-12 00:23:10 UTC
bug #466607 might be the same problem

Comment 7 John Ellson 2008-10-12 00:42:46 UTC
[OK, thats how to get bugs to hyperlink...]

   Possibly the same problem:
        bug #466607
        bug #464636
        bug #462233
        bug #461850
        bug #459109
        bug #454663

Comment 8 John Ellson 2008-10-13 17:55:41 UTC
Going by <https://fedoraproject.org/wiki/QA/ReleaseCriteria>
I propose that this bug is an F10 blocker.

Comment 9 John Ellson 2008-10-17 19:33:43 UTC
Still doesn't boot:

Comment 10 John Ellson 2008-10-30 02:33:34 UTC
Still doesn't boot:

Comment 11 Thomas J. Baker 2008-10-31 14:01:26 UTC
I just did a clean install of rawhide on a Dell Precision 670 and had the same problem. Booting rescue and remaking the initrd worked around the problem.

Comment 12 Thomas J. Baker 2008-10-31 19:54:21 UTC
I should have said remaking the initrd with the --with=scsi_wait-scan worked around the problem.

Comment 13 John Ellson 2008-11-01 01:38:33 UTC
Confirming that:
    mkinitrd --with=scsi_wait-scan ...
worked for me too.

How does this help someone installing from a Fedora-10-Live DVD ?  Can this option be provided as a kernel option?

Comment 14 John Ellson 2008-11-06 17:10:05 UTC
I was hoping that this would the problem:

    * Tue Nov  4 17:00:00 2008 Peter Jones <pjones redhat com> - 6.0.70-1
    - Make scsi waiting happen on any device with a scsi modalias.

but no such luck.


The "mkinitrd --with=scsi_wait-scan ..." still provides a workaround.

Capture of failed boot coming next...

Comment 15 John Ellson 2008-11-06 17:11:50 UTC
Created attachment 322760 [details]
Full text capture of failed boot - mkinitrd-6.0.70-1.fc10.i386

Comment 16 John Ellson 2008-11-06 17:36:08 UTC
I don't fully understand the implications of ".. with a scsi modalias" ?

In case its relevant, my /etc/modprobe.conf contains:

    alias eth0 e100
    alias scsi_hostadapter aic7xxx
    install snd-emu10k1 /sbin/modprobe --ignore-install snd-emu10k1 && /usr/sbin/alsactl restore >/dev/null 2>&1 || :
    alias char-major-81 bttv
    alias usb-controller uhci-hcd

Comment 17 Jesse Keating 2008-11-10 22:51:39 UTC
I'm kicking this one over to F10Target.

Comment 18 Thomas J. Baker 2008-11-11 21:00:19 UTC
I just had the problem again with the latest kernel update. Is this not a blocker because it's not happening to everyone with scsi/lvm?

Comment 19 Jesse Keating 2008-11-11 21:48:34 UTC
That's correct.  So far, very few people are reporting this.  We'd take a fix for it, but I don't believe we'd delay the release for it.

Comment 20 Charlie Moschel 2008-11-14 00:55:55 UTC
Created attachment 323519 [details]
make scsi_scan_wait be used by default

> That's correct.  So far, very few people are reporting this.  We'd take a fix
> for it, but I don't believe we'd delay the release for it.

I respectfully disagree; turning on SCSI_SCSAN_ASYNC and removing the scsi_wait_scan from mkinitrd is a bad combination that has caused quite a few problems in Fedora:

bug #471903
bug #470726
bug #466607
bug #466071
bug #466534
bug #465225
bug #454663

bug #464636
bug #461850
bug #459109

These are only the ones assigned to mkinitrd; there are probably others that didn't (yet) get correctly assigned.  Some bugs above have many 'me too's.

Attached patch will add scsi_scan_wait by default.  This is how it used to be done, before scsi_mod was built into the kernel (see bug #454663 for the first report).

Another way to fix this is to add more tests to trigger the use of nash's stabilized() call.  But LKML guidance suggests scsi_scan_wait is the right way (TM), and the current stabilized() call is not long enough in at least a couple of cases (bug #461850 and bug #466607)

Comment 21 Charlie Moschel 2008-11-14 00:59:31 UTC
(In reply to comment #20)
> bug #471903

sorry, that should be bug #471093

Comment 22 Lubomir Bulej 2008-11-18 16:16:38 UTC
Created attachment 323920 [details]
 Fix to mkinitrd to wait for scsi


the attached patch fixes mkinitrd for me, but serves mainly to show the main
reason, which is twofold. First, there is a case typo which results in scsi
devices not being recognized and the consequently variable wait_for_scsi is not
set to "yes". 

The second cause is due to emitmodules() being called twice (once for
GRAPHICSMODS modlist, and once for MODULES modlist -- the default). The problem
is that when the GRAPHICSMODS modlist is being processed, the wait_for_scsi
variable will get unset and will not be used later, when MODULES modlist is
being handled. 

The patch fixes the first typo and prevents wait_for_scsi variable to be unset
during handling the GRAPHICSMODS modlist.

I consider it rather dirty, but well, so is the emitmodules() function that has
side effect on global variables and is suddenly (I assume with advent of
plymouth) called twice.

Comment 23 Dr. Tilmann Bubeck 2008-11-19 18:07:11 UTC
Please fix this problem. Keep in mind, that an unfixed version of fedora is not able to be installed on an SCSI system and probably other systems. Offering an updated version through a online repository is useless, because the system ist not able to boot after the primary installation. I suffered from this problem too and found "scsi_wait-scan" to be a workaround to use the system at last.

This will affect quiete a few people.

Comment 24 Andrew Mann 2008-11-23 19:25:07 UTC
I just went through a several hour process of diagnosing and hacking around this on an install of the fedora 10 pre-release. Similar system configuration to what has already been reported:  dual core proc, scsi controller (3ware 9550SX), lvm.

I'd like to add in the factors that make this hard to diagnose:
- The new graphical boot that's enabled by default goes to 100% (or a full white progress bar) and stops there.  Hitting esc to bring up the text display just shows a blank screen through the whole process - I presume due to the 'quiet' kernel parameter.
- The default grub.conf has a timeout of 0, so the kernel boots immediately giving you no time to alter the kernel boot parameters (there might be a key you can hold during boot to stop this, but my grub-fu is weak).
- Once you do strip rhgb and quiet from the boot parameters, it's not very obvious that it's the delayed scsi device identification that causes lvm to fail. Typically the first checks are to make sure the lvm structures are ok disk, and that the appropriate drivers are being loaded in the initrd.
- The real difficulty here will be that there is no obvious string to search on to find this problem. Only after diagnosing the problem was I able to relate it to this bug.  To most, this will be "fedora 10 doesn't boot after install," a rather vague problem.

On the up side, I now know more about the fedora boot process :)

Comment 25 Bug Zapper 2008-11-26 03:45:24 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:

Comment 26 Eric Harrison 2008-12-10 01:40:25 UTC
This fixes this issue for Adaptec SCSI cards:

# rpm -q mkinitrd

--- /sbin/mkinitrd.orig	2008-12-09 17:00:49.000000000 -0800
+++ /sbin/mkinitrd	2008-12-09 17:30:54.000000000 -0800
@@ -1518,6 +1518,7 @@
                 -o "BusLogic" == "$module" \
                 -o "mptbase" == "$module" \
                 -o "pata_" == "${module::5}" \
+                -o "aic7" == "${module::4}" \
                 -o "qla" == "${module::3}" \
                 -o "sata_" == "${module::5}" \
                 ]; then

Comment 27 Warren Togami 2008-12-10 01:49:10 UTC
This was fixed in a more generic way (without hard coding controller names) in this build.  Please test it and report back.

Comment 28 John Ellson 2008-12-10 07:02:24 UTC
Works for me.

Do you want the console log?

Comment 29 Charlie Moschel 2008-12-10 13:19:10 UTC
Easiest path forward I see for a new install affected by this:

* Symptom: F10 installs OK, but your SCSI system won't boot after install is done.

* Workaround: 
  - Hit ESC (?) early enough to interrupt the boot; 
  - Add "scsi_mod.scan=sync" to the kernel command line,
  - After boot and firstboot complete, update mkinitrd
  - After updating mkinitrd, you must rebuild your /boot/initrd (run /usr/libexec/plymouth/plymouth-update-initrd as root).

Could this be added to the common bugs page at fedoraproject.org?

Comment 30 Eric Harrison 2008-12-10 16:32:22 UTC
(In reply to comment #27)
> http://kojipkgs.fedoraproject.org/packages/mkinitrd/6.0.71/3.fc10/
> This was fixed in a more generic way (without hard coding controller names) in
> this build.  Please test it and report back.

Confirmed that mkinitrd-6.0.71-3.fc10 works for me.

Comment 31 Dave Jones 2008-12-10 20:27:03 UTC

*** This bug has been marked as a duplicate of bug 470628 ***

Comment 32 Hans de Goede 2008-12-16 16:06:16 UTC

*** This bug has been marked as a duplicate of bug 466607 ***

Note You need to log in before you can comment on or make changes to this bug.