Bug 730007 - Kernel Panic after upgrade to kernel-PAE-2.6.40-4.fc15.i686
Summary: Kernel Panic after upgrade to kernel-PAE-2.6.40-4.fc15.i686
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 15
Hardware: i686
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-11 14:12 UTC by Eddie Lania
Modified: 2012-03-06 19:29 UTC (History)
11 users (show)

Fixed In Version: kernel-3.2.9-1.fc16
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-03-06 19:29:47 UTC


Attachments (Terms of Use)
Screenshot of kernel panic during boot process (1.66 MB, image/jpeg)
2011-08-11 14:12 UTC, Eddie Lania
no flags Details
test 1 (21.90 KB, text/plain)
2011-09-14 13:41 UTC, Eddie Lania
no flags Details
test 2 (21.54 KB, text/plain)
2011-09-14 13:41 UTC, Eddie Lania
no flags Details
test 3 (with rdinitdebug kernel parameter) (17.76 KB, text/plain)
2011-09-14 13:47 UTC, Eddie Lania
no flags Details
output of acpidump (80.82 KB, text/plain)
2011-10-06 07:09 UTC, Eddie Lania
no flags Details
boot.log of working kernel (6.70 KB, text/plain)
2011-10-06 07:10 UTC, Eddie Lania
no flags Details
/var/log/messages of working kernel (67.01 KB, text/plain)
2011-10-06 07:11 UTC, Eddie Lania
no flags Details


Links
System ID Priority Status Summary Last Updated
Launchpad 876359 None None None Never

Description Eddie Lania 2011-08-11 14:12:07 UTC
Created attachment 517818 [details]
Screenshot of kernel panic during boot process

Description of problem:

I recently performed a yum update to update my system to kernel-PAE 2.6.40-4.

The yum update appeared to run just fine,but upon rebooting I receive the following error message:

/bin/sh: error while loading shared libraries: libgcc_s.so.1: cannot open
shared object file: No such file or directory
[    5.229739] Kernel panic - not syncing: Attempted to kill init!
[    5.229806] Pid: 1, comm: init Not tainted 2.6.40-4.fc15.x86_64 #1
... call trace ...

Version-Release number of selected component (if applicable):

kernel-PAE-2.6.40-4.fc15.i686

How reproducible: Always


Steps to Reproduce:
1. Yum update to new kernel
2. Reboot
3. Observe you are unable to boot.
  
Actual results: System does not boot but stops with a kernel panic error message.


Expected results: Normal boot.


Additional info:

See the attached image.

Maybe it's not working for me because I use some older hardware?
I have a regular 3.0 GHz Intel Pentium 4 processor on a Intel D865GLC
mainboard.

I erased and reinstalled this kernel twice but i still get kernel panics during
boot.
I also used yumdownloader to download the kernel rpm amd did a "yum
localinstall" but it doesn't help either.

the older kernels boot fine.

Comment 1 Doug Enright 2011-08-11 14:29:39 UTC
BTW, in relation to BUG #729526 (https://bugzilla.redhat.com/show_bug.cgi?id=729526), while erasing kernel-2.6.40-4.fc15.x86_64, yum also removed kmod-nvidia-2.6.40-4.fc15.x86_64 which was not reinstalled.  I do have the akmod-nvidia package installed.

There may be an issue with the 2.6.40 kernel and adding modules to it.

Comment 2 Josh Boyer 2011-08-11 16:50:11 UTC
> Created attachment 517818 [details]
> Screenshot of kernel panic during boot process
> 
> Description of problem:
> 
> I recently performed a yum update to update my system to kernel-PAE 2.6.40-4.
> 
> The yum update appeared to run just fine,but upon rebooting I receive the
> following error message:
> 
> /bin/sh: error while loading shared libraries: libgcc_s.so.1: cannot open
> shared object file: No such file or directory

Something is going wrong in the creation of your initramfs, and dracut is exiting which panics the kernel (pid 1 can't ever exit).  If you look in /boot, do the initramfs-<version>.img files all seem of comparable size (around 14MB) or is the bad one really small, etc?

Comment 3 Eddie Lania 2011-08-11 17:21:46 UTC
I checked that.
They all seem of comparable size.

Comment 4 Dmitri Berdashov 2011-08-13 08:26:24 UTC
Hi!
Sorry for my bad english.
I have the same problem.
I have a regular 2.6 GHz Intel Pentium 4 processor on a Intel D865GBF
mainboard.
After upgrade  to kernel-PAE-2.6.40-4.fc15.i686 system does not boot.
Fedora 15

Comment 5 Eddie Lania 2011-09-08 13:42:44 UTC
Just updated to kernel 2.6.40.4-5 in the hope that it would solve this problem. But unfortunaly, it didn't. The same problem is still there!

Is someone working on it yet?

Comment 6 Josh Boyer 2011-09-08 13:53:32 UTC
(In reply to comment #5)
> Just updated to kernel 2.6.40.4-5 in the hope that it would solve this problem.
> But unfortunaly, it didn't. The same problem is still there!
> 
> Is someone working on it yet?

There isn't much information to go on in the bug.  We'll need to see more of the boot process in order to determine what might be happening.  Do you have a serial port on this machine that could be used as the console to capture the boot logs?

Comment 7 Eddie Lania 2011-09-08 14:54:21 UTC
Yes, the system has a serial port.

How do I have to do that?

Comment 8 Chuck Ebbert 2011-09-09 09:22:20 UTC
(In reply to comment #7)
> Yes, the system has a serial port.
> 
> How do I have to do that?

http://fedoraproject.org/wiki/How_to_debug_Dracut_problems

Juast adding rdinitdebug to the kernel boot options might be enough to see what's happening.

Comment 9 Eddie Lania 2011-09-14 13:39:46 UTC
I was able to capture boot process via ttyS0.

Please see the attached test1.txt, test2.txt and test3.txt.

Comment 10 Eddie Lania 2011-09-14 13:41:05 UTC
Created attachment 523152 [details]
test 1

Comment 11 Eddie Lania 2011-09-14 13:41:38 UTC
Created attachment 523153 [details]
test 2

Comment 12 Eddie Lania 2011-09-14 13:47:21 UTC
Created attachment 523158 [details]
test 3 (with rdinitdebug kernel parameter)

Comment 13 Eddie Lania 2011-09-27 13:53:53 UTC
I provided logs.

Is somebody working on this issue now?

Comment 14 Eddie Lania 2011-09-30 08:30:07 UTC
When is this going to be solved?
I can't install newer kernels now.

Comment 15 Josh Boyer 2011-09-30 11:48:03 UTC
Can you post the boot log of a working kernel, as well as providing the output of acpidump from that kernel?

Also, have you run memtest86+ on this machine to eliminate bad memory?

Comment 16 Eddie Lania 2011-10-06 07:09:30 UTC
Created attachment 526628 [details]
output of acpidump

Comment 17 Eddie Lania 2011-10-06 07:10:27 UTC
Created attachment 526629 [details]
boot.log of working kernel

Comment 18 Eddie Lania 2011-10-06 07:11:25 UTC
Created attachment 526630 [details]
/var/log/messages of working kernel

Comment 19 Eddie Lania 2011-10-06 15:03:37 UTC
(In reply to comment #15)
> Can you post the boot log of a working kernel, as well as providing the output
> of acpidump from that kernel?
> 
> Also, have you run memtest86+ on this machine to eliminate bad memory?

I have created the log files as you requested. Memtest did not produce errors.

Regards,

Eddie.

Comment 20 Josh Boyer 2011-10-10 19:43:11 UTC
I've tried to reproduce this on some of the P4 machines we have here, but none of them hit this issue.  Bug 727865 also shows similar errors for similar machines, so at the moment we suspect it's isolated to either a few particular models of Pentium 4 machines, or a particular BIOS version for them, or similar.

We know 2.6.38 works and 2.6.40 (3.0) does not.  It would be helpful to know if 2.6.39 worked.  Would you be willing to install this older kernel and see if it boots?

http://koji.fedoraproject.org/koji/buildinfo?buildID=244663

Until we can either recreate here or find someone with an afflicted machine that is willing to do a git bisect, this is going to be somewhat hard to narrow down.

Comment 21 Eddie Lania 2011-10-21 08:50:29 UTC
> We know 2.6.38 works and 2.6.40 (3.0) does not.  It would be helpful to know if
> 2.6.39 worked.  Would you be willing to install this older kernel and see if it
> boots?
> 
> http://koji.fedoraproject.org/koji/buildinfo?buildID=244663

This is a F16 kernel, my distr is F15 (current). It seems to me it is impossible to install F16 kernel on F15, is it?

Comment 22 Josh Boyer 2011-10-21 12:57:22 UTC
(In reply to comment #21)
> > We know 2.6.38 works and 2.6.40 (3.0) does not.  It would be helpful to know if
> > 2.6.39 worked.  Would you be willing to install this older kernel and see if it
> > boots?
> > 
> > http://koji.fedoraproject.org/koji/buildinfo?buildID=244663
> 
> This is a F16 kernel, my distr is F15 (current). It seems to me it is
> impossible to install F16 kernel on F15, is it?

No. Just download it and use:

yum localinstall <path to rpm>

Comment 23 Eddie Lania 2011-10-21 14:16:04 UTC
Okay, I downloaded and installed kernel 2.6.39 per as in yor comment #21.

It gives the same result, it crashes on boot.

So far, the kernels up to 2.6.38 work, up to 2.6.38.8-35 to be exactly.

Comment 24 Josh Boyer 2011-10-21 14:23:32 UTC
(In reply to comment #23)
> Okay, I downloaded and installed kernel 2.6.39 per as in yor comment #21.
> 
> It gives the same result, it crashes on boot.

And it crashes with the same backtrace as in comment #12?

> So far, the kernels up to 2.6.38 work, up to 2.6.38.8-35 to be exactly.

If it's the same crash, that's somewhat good news.  At least we've narrowed it down a bit.

Comment 26 Eddie Lania 2011-10-24 11:47:21 UTC
The first 2.6.39 kernel release (rc0 - 236996) in the links from comment #25 doesn't already work, so, do I have to continue testing them?

Comment 27 Eddie Lania 2011-11-05 22:08:31 UTC
Well, that's it for me guys.

I give it up.

I have only P4 machines running here and they all have problems with latest kernels so I am going to change to some other distro or perhaps say goodbye to linux.

To bad though. I always was convinced linux being far more stable then Microsoft's Windows but it seems it's the other way around now these days.

Can somebody please close this bug for me?

Bye bye RedHat/Fedora, Had some fun for more then 15 years.

Comment 28 Josh Boyer 2011-11-06 15:20:49 UTC
(In reply to comment #27)
> Well, that's it for me guys.
> 
> I give it up.
> 
> I have only P4 machines running here and they all have problems with latest
> kernels so I am going to change to some other distro or perhaps say goodbye to
> linux.

That's somewhat unfortunate.

I would honestly be very surprised if any distro running a 2.6.39 or newer kernel worked on that machine.  We really aren't carrying any patches in Fedora that would impact this.

If you do find a distro that works using a 3.x kernel, it would be greatly appreciated if you let us know.

Comment 29 Doug Enright 2011-11-06 19:01:17 UTC
As a someone who encountered an issue with my Fedora 15 machine and was successfully helped by the Redhat team, I say thank you.

To the original poster, I would say that the P4 is quickly (in computer time) becoming a "dusty deck" system given that the last P4 was shipped in 2008 and the architecture was first introduced in 2000.  

The ISA is still supported in newer chips which after 12 years are probably at least 6x more powerful.  You could potentially consolidate your P4 systems into a single newer Intel multicore processor based system and save on SWaP (size, weight and power) costs.  This cost savings could provide reinvested into future upgrades or other pressing needs.

Comment 30 Doug Enright 2011-11-06 19:03:21 UTC
the last line should read:

"This cost savings could be reinvested into future upgrades or other pressing needs."

Comment 31 Eddie Lania 2011-11-06 20:41:03 UTC
LOL: On the Fedora project homepage:


Download Requirements

    400 MHz Pentium Pro or better processor (comparable AMD, Cyrix, and VIA processors also acceptable)
    At least 768 MB memory (RAM), 1 GB recommended for best performance
    10 GB disk space / storage

Comment 32 Doug Enright 2011-11-07 01:04:24 UTC
Well I think the Fedora folks will need to update this information.

I think as with most FOSS projects ... caveat emptor or should it be caveat venditor? (either way, let the user beware).

Comment 33 Eddie Lania 2011-11-28 09:07:24 UTC
I give this one more try.

With kernel parameter maxcpus=0 every kernel boots fine.

Comment 34 Eddie Lania 2011-11-29 10:56:53 UTC
PS: A new machine is ordered already.

I only want this bug to be kept open because it must be solved for other P4 users as well.

I have now a P4 (Intel D865PERL + 3.2 GHz processor) especially running for this bug only.

Please continue the investagation,

Thank you.

Eddie.

Comment 35 anari.jalakas 2011-12-01 21:08:52 UTC
I am not Fedora user but I am VERY interested in that this bug is going to be fixed.

I have definitely the same problem: Intel P4 3GHz, Intel D865PERL.

My bug report in Launchpad is here:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/876359 

I have tested almost all main series after 2.6.38 and none of them work for me.

I am also compiling kernels by myself to find the revision that broke the kernel.
By now it is sure that the revision is between 2.6.38 and 2.6.39rc1.

Feel free to contact me it it is needed.

Comment 36 Josh Boyer 2011-12-01 21:37:25 UTC
(In reply to comment #35)
> I am not Fedora user but I am VERY interested in that this bug is going to be
> fixed.
> 
> I have definitely the same problem: Intel P4 3GHz, Intel D865PERL.
> 
> My bug report in Launchpad is here:
> 
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/876359 
> 
> I have tested almost all main series after 2.6.38 and none of them work for me.
> 
> I am also compiling kernels by myself to find the revision that broke the
> kernel.
> By now it is sure that the revision is between 2.6.38 and 2.6.39rc1.

Excellent.  A git bisect is probably the best way forward on this bug.  If you find the particular commit that causes the issues, please let us know.

Comment 37 Josh Boyer 2012-01-06 15:08:58 UTC
Matthew Garrett pointed me at a patch for a regression in ACPI yesterday.  I've started a scratch build with this patch applied.  Could those with an impacted machine please try this kernel when it finishes building and let us know the results?

http://koji.fedoraproject.org/koji/taskinfo?taskID=3624930

Comment 38 Josh Boyer 2012-01-06 16:05:17 UTC
My apologies, I pasted the wrong link to the scratch build.  This is the one that should be tested:

http://koji.fedoraproject.org/koji/taskinfo?taskID=3625177

Comment 39 Eddie Lania 2012-01-09 10:14:00 UTC
I installed the kernel-PAE, kernel-headers en perf from this koji build.

This kernel does not boot but hangs with the follwing messages:

swapper[1]: segfault at 9ba00 ip   (null) sp f4499ca0 error 4
BUG: unable to handle kernel no vm86_info: BAD
note: swapper[1] exited with preempt_count 1

Comment 40 Eddie Lania 2012-01-09 10:19:53 UTC
Hmmm, and onother try gives complete different error.

With kernel parameter maxcpus=1 everything boots fine and there is no problem at all with this kernel.

Comment 41 Josh Boyer 2012-02-26 16:31:05 UTC
Matthew pointed me to:

http://marc.info/?l=linux-acpi&m=133002974918284&w=2

That seems like a rather plausible fix for this.

Comment 42 Eddie Lania 2012-02-26 22:08:18 UTC
Great,

If somebody's able to apply the patch in an new kernel' I'd be happy to test this. I still am keeping a separate older P4 machine alive for this purpose.

Comment 43 Josh Boyer 2012-02-27 14:08:24 UTC
This scratch build should have the patch mentioned above:

http://koji.fedoraproject.org/koji/taskinfo?taskID=3822374

Testing when it completes would be much appreciated.

Comment 44 anari.jalakas 2012-02-27 21:04:37 UTC
(In reply to comment #41)
> Matthew pointed me to:
> 
> http://marc.info/?l=linux-acpi&m=133002974918284&w=2
> 
> That seems like a rather plausible fix for this.

I can confirm, this patch fixed the problem for me.

Comment 45 Fedora Update System 2012-02-29 01:34:37 UTC
kernel-3.2.8-3.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/kernel-3.2.8-3.fc16

Comment 46 Eddie Lania 2012-02-29 09:34:24 UTC
I confirm too, kernel-PAE-2.6.42.7-1.1.fc15.i686.rpm installed and working good with both cpu's enabled.

Comment 47 Fedora Update System 2012-03-01 09:29:44 UTC
Package kernel-3.2.8-3.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.2.8-3.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-2745/kernel-3.2.8-3.fc16
then log in and leave karma (feedback).

Comment 48 Fedora Update System 2012-03-01 22:52:56 UTC
kernel-3.2.9-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/kernel-3.2.9-1.fc16

Comment 49 Eddie Lania 2012-03-05 09:06:27 UTC
(In reply to comment #43)
> This scratch build should have the patch mentioned above:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=3822374
> 
> Testing when it completes would be much appreciated.

It is running fine here now, thank you.

Comment 50 Fedora Update System 2012-03-06 19:29:47 UTC
kernel-3.2.9-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.