From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5 Description of problem: did upgrade via yum, all went well. Rebooted to load new kernel and got panic. Was able to catch grub on next boot and booted old version no problem. This happened on both smp servers, but worked fine on the single cpu boxes. Version-Release number of selected component (if applicable): kernel-smp-2.6.12-1.1372_FC3 How reproducible: Always Steps to Reproduce: 1.make kernel-smp-2.6.12-1.1372_FC3 the default boot kernel and reboot 2. 3. Actual Results: kernel panic not syncing Expected Results: system should have booted. Additional info:
we'll need more of the messages to be able to diagnose this. tips: 1. make sure you don't have 'quiet' in your boot command line in grub 2. adding vga=791 will put your console into a higher resolution so you get more lines of text Even if you have to resort to attaching a digital camera picture, it'd help. As it stands, there's nothing in this report to work with. Sorry.
Created attachment 116826 [details] Picture of boot panic screen From YUM : Dependencies Resolved Transaction Listing: Install: kernel.i686 0:2.6.12-1.1372_FC3 - updates-released Install: kernel-smp.i686 0:2.6.12-1.1372_FC3 - updates-released Total download size: 28 M Downloading Packages: (1/2): kernel-2.6.12-1.13 100% |=========================| 14 MB 00:22 (2/2): kernel-smp-2.6.12- 100% |=========================| 14 MB 00:22 Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing: kernel-smp 100 % done 1/2 Installing: kernel 100 % done 2/2 Installed: kernel.i686 0:2.6.12-1.1372_FC3 kernel-smp.i686 0:2.6.12-1.1372_FC3 Complete! =========================== As noted previously, this only happens on the SMP boxes, the single cpu boxes are fine. All are supermicro motherboards with on board ati video, scsi, intel ethernet. Let me know what else you need. The attached picture shjows most of the boot.
this looks like part of your initrd is missing. (In particular the jbd part of ext3) what does your /boot/initrd-2.6.12-1.1372_FC3 look like (ls -l) Can you run (as root) mkinitrd /boot/initrd-2.6.12-1.1372_FC3 2.6.12-1.1372_FC3 and try to boot again ?
> ls -l /boot/initrd-2.6.12-1.1372_FC3 > ls: /boot/initrd-2.6.12-1.1372_FC3: No such file or directory > mkinitrd /boot/initrd-2.6.12-1.1372_FC3 2.6.12-1.1372_FC3 >ls -l /boot/initrd-2.6.12-1.1372_FC3 -rw-r--r-- 1 root root 978581 Jul 15 20:32 /boot/initrd-2.6.12-1.1372_FC3 rebooted and got the same exact same panic. rebooted again same panic rebooted 3rd time caught grub and loaded older version, worked fine.
argh, my bad. that command line should be.. mkinitrd /boot/initrd-2.6.12-1.1372_FC3.img 2.6.12-1.1372_FC3 (I missed off the .img) You can delete the /boot/initrd-2.6.12-1.1372_FC3 that got created after the previous comment.
just to be sure we are working with the same environemnt, I removed then re-installed kernel. Verified all old .1372 files were removed before re-installing. Ran the following command which produced and error : > mkinitrd /boot/initrd-2.6.12-1.1372_FC3.img 2.6.12-1.1372_FC3 /boot/initrd-2.6.12-1.1372_FC3.img already exists. >
you can safely rm or mv it out of the way, the mkinitrd command will recreate it.
Since this is an smp box, I ran you command as well as modifying it slightly to regen the smp file also : mkinitrd /boot/initrd-2.6.12-1.1372_FC3smp.img 2.6.12-1.1372_FC3smp Tried a reboot and got the exact same panic again ============== interesting the new build is smaller than the old one. drwxr-xr-x 4 root root 4096 Jul 16 07:39 . drwxr-xr-x 23 root root 4096 Jul 16 07:43 .. -rw-r--r-- 1 root root 55504 Jun 13 01:10 config-2.6.11-1.35_FC3 -rw-r--r-- 1 root root 55276 Jun 13 01:36 config-2.6.11-1.35_FC3smp -rw-r--r-- 1 root root 56834 Jul 15 01:18 config-2.6.12-1.1372_FC3 -rw-r--r-- 1 root root 56701 Jul 15 01:46 config-2.6.12-1.1372_FC3smp drwxr-xr-x 2 root root 1024 Jul 15 23:03 grub -rw-r--r-- 1 root root 1162633 Jun 25 10:38 initrd-2.6.11-1.35_FC3.img -rw-r--r-- 1 root root 1143297 Jun 25 10:38 initrd-2.6.11-1.35_FC3smp.img -rw-r--r-- 1 root root 978577 Jul 16 07:29 initrd-2.6.12-1.1372_FC3.img -rw-r--r-- 1 root root 968390 Jul 16 07:39 initrd-2.6.12-1.1372_FC3smp.img drwx------ 2 root root 12288 Nov 20 2004 lost+found -rw-r--r-- 1 root root 747046 Jun 13 01:10 System.map-2.6.11-1.35_FC3 -rw-r--r-- 1 root root 774796 Jun 13 01:36 System.map-2.6.11-1.35_FC3smp -rw-r--r-- 1 root root 760080 Jul 15 01:18 System.map-2.6.12-1.1372_FC3 -rw-r--r-- 1 root root 790080 Jul 15 01:46 System.map-2.6.12-1.1372_FC3smp -rw-r--r-- 1 root root 1635072 Jun 13 01:10 vmlinuz-2.6.11-1.35_FC3 -rw-r--r-- 1 root root 1553144 Jun 13 01:36 vmlinuz-2.6.11-1.35_FC3smp -rw-r--r-- 1 root root 1652295 Jul 15 01:18 vmlinuz-2.6.12-1.1372_FC3 -rw-r--r-- 1 root root 1575017 Jul 15 01:46 vmlinuz-2.6.12-1.1372_FC3smp
OK, here's a question: does everyone with this problem have ATI chipsets? I happen to have a whole load of ASUS Pundit-R machines lying around my office, so that's what I've been doing all of my testing on. They have ATI integrated everything. Unfortunately they're all at work and all of them are dead due to this update, so I can't provide lspci or lsmod output right now. Is anyone seeing this problem on a machine that does not have an ATI chipset? I can provide full access to one of these boxes if it would help. Heck, I'll even overnight one to RedHat if necessary.
I have many, many boxes with ATI video chipsets, it's only the dual cpu boxes that have the problem, the single cpu boxes are just peachy with this update. These are all supermicro mb's with the scsi 160/320 [adaptec] and intel 10/100 or 1000 and the ATI stuff built in.
This is a me too - smp server fails to boot, ati graphics. Have rebuilt initrd (as per comment 5) for smp kernel and will try again.
Hi, seeing this with SMP kernel only on Dell PE2650, dual cpu, PERC SCSI raid, ATI rageXL. The non SMP boots (but selinux policy seems to be blocking various apps like: yum, mozilla + others). Boot is not helped by removing rhgb + quiet and adding acpi=off noacpi (can't remember which is the correct param :) Also, seeing it on a Dell PE750, dual cpu, 2x SATA drives (software raid), ATI rageXL. SMP fails at (about) the same spot as posted error image. This does start SMP if I remove rhgb + quiet, and add acpi=off noacpi.
Created attachment 116866 [details] kernel-smp-2.6.12-1.1372_FC3 panic during boot Okay, using the newly created initrd made no difference. Using the uni-processor kernel worked fine (as reported by others). However, I'm not sure this is the same problem. I've attached the last part of the panic output I got when trying to boot and it doesn't report anything about unknown symbols, in fact jbd.ko loads fine. It seems to have trouble locating the root device. It all works fine with the kernel-smp-2.6.11-1.35_FC3 kernel. If you want me to submit this as a different bug let me know. John.
Same here, all of 1.137[012] panic during boot on 5 different boxes with HT processors. Booting the up kernel works; booting the smp kernel fails to mount the root fs after failing to load ata_piix.ko, ext3.ko and dm-mirror.ko. All boxes have Intel D865 mobos.
I'm pretty sure this is the same as bug 160652 It's hard to be certain without seeing some more context from the kernel panic in comment #13... There were some other initrd bugs with scsi modules loading but I think those have been fixed. So I'm pretty sure this is a dup of bug 160652
BTW. The solution is to downgrade to the last working kernel until a new mkinitrd package is released for FC3. It's a mkinitrd bug really, not a kernel bug. A lot of people have been hit by this so it won't take long.
This is definitely not a bug in mkinitrd. I was able to boot the SMP kernel using the apparently-broken initrd by simply disabling HT in the BIOS configuration. It sounds more like a race condition in the kernel. FWIW, the latest FC4 kernel (1.1398) boots correctly on these boxes, after an upgrade to FC4 (or rather full install in separate partitions, to preserve FC3 just in case).
1) Re comment 15 (re comment 13 output) - sorry but all I had was pen/paper to write down the last part of the output. Being a 2x3GHz 2GB memory system you can imagine that the rest of the output flies off the top of the (14" monitor) screen extremely quickly! I'll see what the 'vga=791' boot option does. 2) Re comment 17 - I disabled HT in the bios but it still didn't work. Oddly enough the first reboot (with HT disabled) *did* show a load of unresolved symbol errors in the ext3 module and then paniced. Upon rebooting again there were no such errors and jdb.ko and ext3.ko loaded fine, but it still had problems finding/mounting root (as per comment 13). I'm back running on the previous kernel.
I have a dual opteron CPU machine running FC3 and vmlinuz-2.6.11-1.14_FC3smp kernel. With the recent upgrade to new FC3 2.6.12 kernels, i encounter the following crash: Only the smp version crashes (vmlinuz-2.6.12-1.1372_FC3smp). The single processor kernel (vmlinuz-2.6.12-1.1372_FC3) boots in properly. HW: AMD Opteron(tm) Processor 246 Motherboard: Tyan RAM: 4Gb filesystem: XFS Crash: ... Creating root device umount /sys failed: 16 Mounting root filesystem mount: error 19 mounting xfs mount: error 2 mounting none switch to new root switchroot: mount failed: 22 umount /initrd /dev/ failed: 2 kernel panic - not syncing: Attempted to kill init!
Similar situation here. $ uname -a Linux machine 2.6.12-1.1372_FC3smp #1 SMP Fri Jul 15 01:08:54 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux but from other discussions I knew to precede the smp-kernel install with an install of the latest available mkinitrd (built from source) $ mkinitrd --version mkinitrd: version 4.2.18 I was skeptical, but it worked. No joy on dual-IA32 machines, yet.
Peter, for more info on this, see Prarit's mail archived in this thread here.. http://www.adras.com/Race-condition-in-module-load-causing-undefined-symbols.t6648-141-3.html It's quite likely that 163437 is also a dupe of this bug.
*** Bug 164358 has been marked as a duplicate of this bug. ***
*** Bug 163917 has been marked as a duplicate of this bug. ***
*** Bug 163308 has been marked as a duplicate of this bug. ***
*** Bug 163448 has been marked as a duplicate of this bug. ***
*** Bug 162859 has been marked as a duplicate of this bug. ***
An update to mkinitrd just got pushed out to fc3-updates-testing. After installing this update, remove the latest kernel update, and reinstall it. The initrd should then be recreated correctly. Hopefully closing a slew of these bugs.
The update has fixed the problem for me.
Fixed my problem on a small 5 node dual-P4 Xeon test cluster.
For those of us who saw the warning *before* attempting an upgrade (and therefore have not yet installed the 2.6.12 kernel yet), what is the proper upgrade path? Can we just run up2date and have everything work now? (Possibly after waiting for the new mkinitrd to escape testing -- when will that happen and what's its version number so we will recognize it?) Or do we need to update mkinitrd first in one up2date session, and the kernel in a second one?
Please be explicit about what to do to cause this to happen or point to a URL that describes the process. I'm sorry if I am the only one that does not know all the specific commands and implications of these steps. " After installing this update, // with up2date? remove the latest kernel update, // with what command? rm initrd-*? rm vmlinuz-*? and reinstall it. // with up2date? "
This is better discussed on the mailing list. Still: If you're not up to pulling from the testing repository then perhaps it's best to wait until the update is actually released. But if you can't wait, then download http://download.fedora.redhat.com/pub/fedora/linux/core/updates/testing/3/i386/mkinitrd-4.1.18.1-1.i386.rpm (assuming you are on i386 and not x86_64) and also grab the latest kernel from http://download.fedora.redhat.com/pub/fedora/linux/core/updates/3/i386/kernel-smp-2.6.12-1.1372_FC3.i686.rpm Then install the new mkinitrd: rpm -Fvh mkinitrd-4.1.18.1-1.i386.rpm Then uninstall the bad kernel: rpm -e kernel-smp-2.6.12-1.1372_FC3 and install it again: rpm -ivh kernel-smp-2.6.12-1.1372_FC3.i686.rpm then reboot and enjoy.
This fixed it for me. Supermicro dual intel xeon mb with ati / adaptec scsi / intel gig ethernet ... Thank you !
Confirmed. This fixed it for me too using mkinitrd-4.1.18.1-1 rpm from testing. System is an HP DL360 (dual xeon cpus). Many thanks, John.
this now got moved to -final updates. Thanks for testing folks, and apologies for the time this took to get fixed.
The new mkinitrd will be installed with up2date. However, the kernel must still be erased and installed in order for the change to be complete. Per Jason: rpm -e kernel-smp-2.6.12-1.1372_FC3 and install it again: rpm -ivh kernel-smp-2.6.12-1.1372_FC3.i686.rpm
*** Bug 164709 has been marked as a duplicate of this bug. ***
*** Bug 164604 has been marked as a duplicate of this bug. ***
*** Bug 151176 has been marked as a duplicate of this bug. ***
*** Bug 162800 has been marked as a duplicate of this bug. ***
I had same trouble in updating from 2.6.9 to 2.6.12-1.1456_FC4.src.rpm. Insmod error : ata-piix.ko -1 unknown symbol in module ............................. mkinitrd version is 4.1.18-2.i386. M/C configuration is MOtherboard Supermicro P8Sci CPU P4 3Ghz, DDR2 1GB, SATA 160GB Anyone can help me?