Bug 163407 (fc3mkinitrd) - mkinitrd needs update to fix race condition in sh
Summary: mkinitrd needs update to fix race condition in sh
Keywords:
Status: CLOSED ERRATA
Alias: fc3mkinitrd
Product: Fedora
Classification: Fedora
Component: mkinitrd
Version: 3
Hardware: i386
OS: Linux
medium
high
Target Milestone: ---
Assignee: Peter Jones
QA Contact: David Lawrence
URL:
Whiteboard:
: 151176 162800 162859 163308 163448 163917 164358 164604 164709 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2005-07-15 21:48 UTC by Ed
Modified: 2007-11-30 22:11 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-07-29 19:29:55 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Picture of boot panic screen (802.88 KB, image/jpeg)
2005-07-15 23:32 UTC, Ed
no flags Details
kernel-smp-2.6.12-1.1372_FC3 panic during boot (483 bytes, text/plain)
2005-07-18 12:53 UTC, John Horne
no flags Details

Description Ed 2005-07-15 21:48:30 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5

Description of problem:
did upgrade via yum, all went well.  Rebooted to load new kernel and got panic.  Was able to catch grub on next boot and booted old version no problem.

This happened on both smp servers, but worked fine on the single cpu boxes.

Version-Release number of selected component (if applicable):
kernel-smp-2.6.12-1.1372_FC3

How reproducible:
Always

Steps to Reproduce:
1.make kernel-smp-2.6.12-1.1372_FC3 the default boot kernel and reboot
2.
3.
  

Actual Results:  kernel panic not syncing

Expected Results:  system should have booted.

Additional info:

Comment 1 Dave Jones 2005-07-15 23:12:53 UTC
we'll need more of the messages to be able to diagnose this.

tips:

1. make sure you don't have 'quiet' in your boot command line in grub
2. adding vga=791 will put your console into a higher resolution so you get more
lines of text

Even if you have to resort to attaching a digital camera picture, it'd help.
As it stands, there's nothing in this report to work with. Sorry.

Comment 2 Ed 2005-07-15 23:32:35 UTC
Created attachment 116826 [details]
Picture of boot panic screen

From YUM :

Dependencies Resolved
Transaction Listing:
  Install: kernel.i686 0:2.6.12-1.1372_FC3 - updates-released
  Install: kernel-smp.i686 0:2.6.12-1.1372_FC3 - updates-released
Total download size: 28 M
Downloading Packages:
(1/2): kernel-2.6.12-1.13 100% |=========================|  14 MB    00:22     

(2/2): kernel-smp-2.6.12- 100% |=========================|  14 MB    00:22     

Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing: kernel-smp 100 % done 1/2 
Installing: kernel 100 % done 2/2 

Installed: kernel.i686 0:2.6.12-1.1372_FC3 kernel-smp.i686 0:2.6.12-1.1372_FC3
Complete!
===========================

As noted previously, this only happens on the SMP boxes, the single cpu boxes
are fine.    All are supermicro motherboards with on board ati video, scsi,
intel ethernet.

Let me know what else you need.  The attached picture shjows most of the boot.

Comment 3 Dave Jones 2005-07-15 23:48:14 UTC
this looks like part of your initrd is missing. (In particular the jbd part of ext3)

what does your /boot/initrd-2.6.12-1.1372_FC3 look like (ls -l)


Can you run (as root)

mkinitrd /boot/initrd-2.6.12-1.1372_FC3 2.6.12-1.1372_FC3

and try to boot again ?

Comment 4 Ed 2005-07-16 00:41:47 UTC
> ls -l /boot/initrd-2.6.12-1.1372_FC3
> ls: /boot/initrd-2.6.12-1.1372_FC3: No such file or directory

> mkinitrd /boot/initrd-2.6.12-1.1372_FC3 2.6.12-1.1372_FC3

>ls -l /boot/initrd-2.6.12-1.1372_FC3                     
  -rw-r--r--  1 root root 978581 Jul 15 20:32 /boot/initrd-2.6.12-1.1372_FC3

rebooted and got the same exact same panic.

rebooted again same panic

rebooted 3rd time caught grub and loaded older version, worked fine.


Comment 5 Dave Jones 2005-07-16 02:12:35 UTC
argh, my bad. that command line should be..

mkinitrd /boot/initrd-2.6.12-1.1372_FC3.img 2.6.12-1.1372_FC3

(I missed off the .img)
You can delete the /boot/initrd-2.6.12-1.1372_FC3 that got created after the
previous comment.


Comment 6 Ed 2005-07-16 03:07:41 UTC
just to be sure we are working with the same environemnt, I removed then
re-installed kernel.  Verified all old .1372 files were removed before
re-installing.  Ran the following command which produced and error :

> mkinitrd /boot/initrd-2.6.12-1.1372_FC3.img 2.6.12-1.1372_FC3
  /boot/initrd-2.6.12-1.1372_FC3.img already exists.

>




Comment 7 Dave Jones 2005-07-16 03:21:13 UTC
you can safely rm or mv it out of the way, the mkinitrd command will recreate it.


Comment 8 Ed 2005-07-16 11:46:52 UTC
Since this is an smp box, I ran you command as well as modifying it slightly to
regen the smp file also :
mkinitrd /boot/initrd-2.6.12-1.1372_FC3smp.img 2.6.12-1.1372_FC3smp

Tried a reboot and got the exact same panic again

==============
interesting the new build is smaller than the old one.

drwxr-xr-x   4 root root    4096 Jul 16 07:39 .
drwxr-xr-x  23 root root    4096 Jul 16 07:43 ..
-rw-r--r--   1 root root   55504 Jun 13 01:10 config-2.6.11-1.35_FC3
-rw-r--r--   1 root root   55276 Jun 13 01:36 config-2.6.11-1.35_FC3smp
-rw-r--r--   1 root root   56834 Jul 15 01:18 config-2.6.12-1.1372_FC3
-rw-r--r--   1 root root   56701 Jul 15 01:46 config-2.6.12-1.1372_FC3smp
drwxr-xr-x   2 root root    1024 Jul 15 23:03 grub
-rw-r--r--   1 root root 1162633 Jun 25 10:38 initrd-2.6.11-1.35_FC3.img
-rw-r--r--   1 root root 1143297 Jun 25 10:38 initrd-2.6.11-1.35_FC3smp.img
-rw-r--r--   1 root root  978577 Jul 16 07:29 initrd-2.6.12-1.1372_FC3.img
-rw-r--r--   1 root root  968390 Jul 16 07:39 initrd-2.6.12-1.1372_FC3smp.img
drwx------   2 root root   12288 Nov 20  2004 lost+found
-rw-r--r--   1 root root  747046 Jun 13 01:10 System.map-2.6.11-1.35_FC3
-rw-r--r--   1 root root  774796 Jun 13 01:36 System.map-2.6.11-1.35_FC3smp
-rw-r--r--   1 root root  760080 Jul 15 01:18 System.map-2.6.12-1.1372_FC3
-rw-r--r--   1 root root  790080 Jul 15 01:46 System.map-2.6.12-1.1372_FC3smp
-rw-r--r--   1 root root 1635072 Jun 13 01:10 vmlinuz-2.6.11-1.35_FC3
-rw-r--r--   1 root root 1553144 Jun 13 01:36 vmlinuz-2.6.11-1.35_FC3smp
-rw-r--r--   1 root root 1652295 Jul 15 01:18 vmlinuz-2.6.12-1.1372_FC3
-rw-r--r--   1 root root 1575017 Jul 15 01:46 vmlinuz-2.6.12-1.1372_FC3smp


Comment 9 Jason Tibbitts 2005-07-16 15:23:43 UTC
OK, here's a question: does everyone with this problem have ATI chipsets?  I
happen to have a whole load of ASUS Pundit-R machines lying around my office, so
that's what I've been doing all of my testing on.  They have ATI integrated
everything.  Unfortunately they're all at work and all of them are dead due to
this update, so I can't provide lspci or lsmod output right now.

Is anyone seeing this problem on a machine that does not have an ATI chipset?

I can provide full access to one of these boxes if it would help.  Heck, I'll
even overnight one to RedHat if necessary.

Comment 10 Ed 2005-07-16 15:44:57 UTC
I have many, many boxes with ATI video chipsets, it's only the dual cpu boxes
that have the problem, the single cpu boxes are just peachy with this update. 
These are all supermicro mb's with the scsi 160/320 [adaptec] and intel 10/100
or 1000 and the ATI stuff built in.



Comment 11 John Horne 2005-07-18 11:08:36 UTC
This is a me too - smp server fails to boot, ati graphics. Have rebuilt initrd
(as per comment 5) for smp kernel and will try again.



Comment 12 David Timms 2005-07-18 12:39:00 UTC
Hi, seeing this with SMP kernel only on Dell PE2650, dual cpu, PERC SCSI raid,
ATI rageXL. The non SMP boots (but selinux policy seems to be blocking various
apps like: yum, mozilla + others). Boot is not helped by removing rhgb + quiet
and adding acpi=off noacpi (can't remember which is the correct param :)

Also, seeing it on a Dell PE750, dual cpu, 2x SATA drives (software raid), ATI
rageXL. SMP fails at (about) the same spot as posted error image. This does
start SMP if I remove rhgb + quiet, and add acpi=off noacpi.

Comment 13 John Horne 2005-07-18 12:53:48 UTC
Created attachment 116866 [details]
kernel-smp-2.6.12-1.1372_FC3 panic during boot

Okay, using the newly created initrd made no difference. Using the
uni-processor kernel worked fine (as reported by others).

However, I'm not sure this is the same problem. I've attached the last part of
the panic output I got when trying to boot and it doesn't report anything about
unknown symbols, in fact jbd.ko loads fine. It seems to have trouble locating
the root device. It all works fine with the kernel-smp-2.6.11-1.35_FC3 kernel.

If you want me to submit this as a different bug let me know.



John.

Comment 14 Alexandre Oliva 2005-07-18 19:10:58 UTC
Same here, all of 1.137[012] panic during boot on 5 different boxes with HT
processors.  Booting the up kernel works; booting the smp kernel fails to mount
the root fs after failing to load ata_piix.ko, ext3.ko and dm-mirror.ko.  All
boxes have Intel D865 mobos.

Comment 15 Dan Carpenter 2005-07-18 20:19:19 UTC
I'm pretty sure this is the same as bug 160652

It's hard to be certain without seeing some more context from the kernel panic
in comment #13...  There were some other initrd bugs with scsi modules loading
but I think those have been fixed.  So I'm pretty sure this is a dup of bug 160652



Comment 16 Dan Carpenter 2005-07-18 20:22:25 UTC
BTW.  The solution is to downgrade to the last working kernel until a new
mkinitrd package is released for FC3.  It's a mkinitrd bug really, not a kernel bug.

A lot of people have been hit by this so it won't take long.


Comment 17 Alexandre Oliva 2005-07-19 07:04:11 UTC
This is definitely not a bug in mkinitrd.  I was able to boot the SMP kernel
using the apparently-broken initrd by simply disabling HT in the BIOS
configuration.  It sounds more like a race condition in the kernel.

FWIW, the latest FC4 kernel (1.1398) boots correctly on these boxes, after an
upgrade to FC4 (or rather full install in separate partitions, to preserve FC3
just in case).

Comment 18 John Horne 2005-07-19 15:54:18 UTC
1) Re comment 15 (re comment 13 output) - sorry but all I had was pen/paper to
write down the last part of the output. Being a 2x3GHz 2GB memory system you can
imagine that the rest of the output flies off the top of the (14" monitor)
screen extremely quickly! I'll see what the 'vga=791' boot option does.

2) Re comment 17 - I disabled HT in the bios but it still didn't work. Oddly
enough the first reboot (with HT disabled) *did* show a load of unresolved
symbol errors in the ext3 module and then paniced. Upon rebooting again there
were no such errors and jdb.ko and ext3.ko loaded fine, but it still had
problems finding/mounting root (as per comment 13). I'm back running on the
previous kernel.

Comment 19 Gaspar Bakos 2005-07-20 23:01:03 UTC
I have a dual opteron CPU machine running FC3 and
vmlinuz-2.6.11-1.14_FC3smp kernel.

With the recent upgrade to new FC3 2.6.12 kernels, i encounter the
following crash:

Only the smp version crashes (vmlinuz-2.6.12-1.1372_FC3smp). The single
processor kernel (vmlinuz-2.6.12-1.1372_FC3) boots in properly.

HW:
AMD Opteron(tm) Processor 246
Motherboard: Tyan
RAM: 4Gb
filesystem: XFS
Crash:
...
Creating root device
umount /sys failed: 16
Mounting root filesystem
mount: error 19 mounting xfs
mount: error 2 mounting none
switch to new root
switchroot: mount failed: 22
umount /initrd /dev/ failed: 2
kernel panic - not syncing: Attempted to kill init!




Comment 20 Bill Maloy 2005-07-21 21:06:52 UTC
Similar situation here.

$ uname -a
Linux machine 2.6.12-1.1372_FC3smp #1 SMP Fri Jul 15 01:08:54 EDT 2005 x86_64
x86_64 x86_64 GNU/Linux

but from other discussions I knew to precede the smp-kernel
install with an install of the latest available mkinitrd
(built from source)

$ mkinitrd --version
mkinitrd: version 4.2.18

I was skeptical, but it worked.  No joy on dual-IA32 machines, yet.


Comment 21 Dave Jones 2005-07-28 06:16:41 UTC
Peter, for more info on this, see Prarit's mail archived in this thread here..

http://www.adras.com/Race-condition-in-module-load-causing-undefined-symbols.t6648-141-3.html

It's quite likely that 163437 is also a dupe of this bug.


Comment 22 Dave Jones 2005-07-28 06:18:39 UTC
*** Bug 164358 has been marked as a duplicate of this bug. ***

Comment 23 Dave Jones 2005-07-28 06:22:15 UTC
*** Bug 163917 has been marked as a duplicate of this bug. ***

Comment 24 Dave Jones 2005-07-28 19:00:09 UTC
*** Bug 163308 has been marked as a duplicate of this bug. ***

Comment 25 Dave Jones 2005-07-28 19:01:30 UTC
*** Bug 163448 has been marked as a duplicate of this bug. ***

Comment 26 Dave Jones 2005-07-28 19:02:13 UTC
*** Bug 162859 has been marked as a duplicate of this bug. ***

Comment 27 Dave Jones 2005-07-28 22:18:36 UTC
An update to mkinitrd just got pushed out to fc3-updates-testing.
After installing this update, remove the latest kernel update, and reinstall it.
The initrd should then be recreated correctly. Hopefully closing a slew of these
bugs.


Comment 28 Jason Tibbitts 2005-07-28 23:24:49 UTC
The update has fixed the problem for me.


Comment 29 Stuart Anderson 2005-07-29 00:40:49 UTC
Fixed my problem on a small 5 node dual-P4 Xeon test cluster.

Comment 30 Damian Menscher 2005-07-29 00:59:56 UTC
For those of us who saw the warning *before* attempting an upgrade (and 
therefore have not yet installed the 2.6.12 kernel yet), what is the proper 
upgrade path?  Can we just run up2date and have everything work now?  (Possibly 
after waiting for the new mkinitrd to escape testing -- when will that happen 
and what's its version number so we will recognize it?)  Or do we need to update 
mkinitrd first in one up2date session, and the kernel in a second one?

Comment 31 Paul Watson 2005-07-29 01:59:35 UTC
Please be explicit about what to do to cause this to happen or point to a URL
that describes the process.  I'm sorry if I am the only one that does not know
all the specific commands and implications of these steps.

"
After installing this update, // with up2date?
remove the latest kernel update, // with what command? rm initrd-*?  rm vmlinuz-*?
and reinstall it.  // with up2date?
"

Comment 32 Jason Tibbitts 2005-07-29 02:12:21 UTC
This is better discussed on the mailing list.  Still:

If you're not up to pulling from the testing repository then perhaps it's best
to wait until the update is actually released.  But if you can't wait, then
download
http://download.fedora.redhat.com/pub/fedora/linux/core/updates/testing/3/i386/mkinitrd-4.1.18.1-1.i386.rpm
(assuming you are on i386 and not x86_64) and also grab the latest kernel from
http://download.fedora.redhat.com/pub/fedora/linux/core/updates/3/i386/kernel-smp-2.6.12-1.1372_FC3.i686.rpm

Then install the new mkinitrd:

rpm -Fvh mkinitrd-4.1.18.1-1.i386.rpm

Then uninstall the bad kernel:

rpm -e kernel-smp-2.6.12-1.1372_FC3

and install it again:

rpm -ivh kernel-smp-2.6.12-1.1372_FC3.i686.rpm

then reboot and enjoy.

Comment 33 Ed 2005-07-29 02:43:35 UTC
This fixed it for me. Supermicro dual intel xeon mb with ati / adaptec scsi /
intel gig ethernet ...

Thank you !

Comment 34 John Horne 2005-07-29 10:39:11 UTC
Confirmed. This fixed it for me too using mkinitrd-4.1.18.1-1 rpm from testing.
System is an HP DL360 (dual xeon cpus).


Many thanks,

John.

Comment 35 Dave Jones 2005-07-29 19:29:55 UTC
this now got moved to -final updates.

Thanks for testing folks, and apologies for the time this took to get fixed.


Comment 36 Paul Watson 2005-07-30 04:28:29 UTC
The new mkinitrd will be installed with up2date.  However, the kernel must still
be erased and installed in order for the change to be complete.

Per Jason:

rpm -e kernel-smp-2.6.12-1.1372_FC3

and install it again:

rpm -ivh kernel-smp-2.6.12-1.1372_FC3.i686.rpm

Comment 37 Dave Jones 2005-07-31 05:21:30 UTC
*** Bug 164709 has been marked as a duplicate of this bug. ***

Comment 38 Dave Jones 2005-08-03 22:50:53 UTC
*** Bug 164604 has been marked as a duplicate of this bug. ***

Comment 39 Dave Jones 2005-08-04 18:00:26 UTC
*** Bug 151176 has been marked as a duplicate of this bug. ***

Comment 40 Dave Jones 2005-08-26 08:44:04 UTC
*** Bug 162800 has been marked as a duplicate of this bug. ***

Comment 41 Shiroh Nakanishi 2005-12-13 17:18:33 UTC
I had same trouble in updating from 2.6.9 to 2.6.12-1.1456_FC4.src.rpm. 

Insmod error : ata-piix.ko -1 unknown symbol in module
.............................

mkinitrd version is 4.1.18-2.i386.
M/C configuration is 
MOtherboard Supermicro P8Sci
CPU P4 3Ghz, DDR2 1GB, SATA 160GB

Anyone can help me?


Note You need to log in before you can comment on or make changes to this bug.