Bug 669531

Summary: Kernel panic on random yum install or update commands
Product: [Fedora] Fedora Reporter: Matt Darcy <matt>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 14CC: dekacy10, ffesti, gansalmon, itamar, james.antill, jonathan, kernel-maint, madhu.chinakonda, maxamillion, pmatilai, tla
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-29 20:56:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Kernel Crash Dump from yum install
none
log output from kernel crash file
none
Boot log from a stable clean Fedora 13 x86_64 base install
none
Boot log from a stable updated Fedora 13 x86_64 (2.6.34.7-66.fc13.x86_64)
none
Boot log from a stable clean Fedora 14 x86_64 base install
none
Boot log from an updated Fedora 14 x86_64 (2.6.35.10-74.fc14.x86_64)
none
/var/log/dmesg output from the stable F14 kernel none

Description Matt Darcy 2011-01-13 21:52:13 UTC
Description of problem:
Using a Fedora 14 x86_64 on an intel core 2 cpu after a specific update (I'm assuming a kernel update as I'm struggling to be exact with reproduction conditions) yum install $component or $yum update causes a kernel panic. This issue does not happen with every update or install command, and not with specific packages as sometimes it can work fine for a test package and other times panic, although Samba and its dependencies appear to be a package that causes the panic more than others.
This is a stock Fedora 14 install, no external repos or packages. A clean install does not appear to have this problem, but after an update to current version (I am unable to be clear on exactly what version) I get random kernel panics from yum install/update commands. I can install packages with rpm without an issue, this only happens through yum.

The kernel panics and auto reboot kicks in.


Version-Release number of selected component (if applicable):
yum-2.7.1.fc14.1.x86_64
inux berger 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
pm-4.8.1-5.fc14.x86_64
rpm-libs-4.8.1-5.fc14.x86_64
rpm-python-4.8.1-5.fc14.x86_64
python-2.7-8.fc14.1.x86_64



How reproducible:
very regular but not tied to a specific package or on every install command


Steps to Reproduce:
1. install Fedora 14 x64_64
2. update OS and base products to 2.6.35.10-74-fc14 level
3.install via yum packages and use the yum update commands until kernel panic
  
Actual results:
kernel panic, stack trace and reboot


Expected results:
install packages and dependencies

Additional info:

Comment 1 Matt Darcy 2011-01-14 11:06:39 UTC
I should also point out this machine is in a standard configuration, no overclocking, fancy cooling, etc.

It has passed 4 compelte cycles on memtest without error.


I would be happy to provide login credentials to bug resolvers to assist with debugging.

Comment 2 Chuck Ebbert 2011-01-15 03:16:22 UTC
(In reply to comment #2)
> Actual results:
> kernel panic, stack trace and reboot

Did you forget to attach the panic messages and stack trace?

Comment 3 Matt Darcy 2011-01-15 12:28:05 UTC
no, the complete message scrolls off screen quite fast, and the stack trace to screen is quite long and garbled, even disabling the auto reboot I have no method of dumping the message and trace to file as the machine is totally hung.

If there is a process to capture this (maybe using netdump ?) I'd be happy to capture the information for you.

Comment 4 Matt Darcy 2011-01-15 13:07:08 UTC
I've now got the kdump service running so I'll attatch some files on the next crash. 

beyond a vmcore from kdump is there anything else you'd like ?

Comment 5 Matt Darcy 2011-01-17 16:15:31 UTC
Created attachment 473872 [details]
Kernel Crash Dump from yum install

this come from a 1.9GB vmcore file dumped on the panic, I can provide this but need a method of getting 1.9GB to you in bugzilla

Comment 6 Chuck Ebbert 2011-01-19 13:23:55 UTC
PANIC: "[183212.409984] Kernel panic - not syncing: Fatal Machine check"

This is most likely a real hardware problem, though it could also be a bug in the MCE code.  You should be able to get a lot more information from the 'crash' program, e.g. the log command will dump the contents of the kernel log. The last few lines may have additional information about the error. You may want to dump the entire log and attach it to this bug report.

Comment 7 Matt Darcy 2011-01-19 21:51:55 UTC
Created attachment 474364 [details]
log output from kernel crash file

details of the crash log do seem to reference a hardware problem as suggested. 

There are a few references to the video card, and then the panic.

While I agree this looks like a hardware issue, I cannot get this fault to occur on a Fedora 13 machine, and it never happened when this machine ran Fedora 12. I cannot get this issue to happen on Fedora 14 before I do an update to the current kernel, which suggests to me at a high level that there is something in one of the updates to the kernel that does not like a hardware device, rather than a phyiscal hardware failure.

I've attatched the log for viewing and I'll attempt to get more information out the panic.

Comment 8 Chuck Ebbert 2011-01-20 15:53:21 UTC
Can you attach the boot log from a kernel that doesn't hit this problem?

Comment 9 Matt Darcy 2011-01-20 16:04:38 UTC
I'll need to rebuild this machine as a Fedora 13 or non-updated Fedora 14 machine.

this will take a little time.

Comment 10 Matt Darcy 2011-01-30 13:50:59 UTC
Created attachment 476031 [details]
Boot log from a stable clean Fedora 13 x86_64 base install

this is a clean re-install of Fedora 13 x86_64 on the same machine, no updates, fresh from the install. This is the boot log as requested

Comment 11 Matt Darcy 2011-01-30 13:52:53 UTC
Created attachment 476032 [details]
Boot log from a stable updated Fedora 13 x86_64 (2.6.34.7-66.fc13.x86_64)

As requested a boot log from a fully up to date Fedora 13 install updated to the latest current Fedora 13 package set as of the time of writing.
2.6.34.7-66.fc13.x86_64

Fedora 14 logs will be coming shortly.

Comment 12 Matt Darcy 2011-01-30 15:37:28 UTC
Created attachment 476044 [details]
Boot log from a stable clean Fedora 14 x86_64 base install

as requested a boot log from a clean Fedora 14 install

Comment 13 Matt Darcy 2011-01-30 15:39:25 UTC
Created attachment 476045 [details]
Boot log from an updated Fedora 14 x86_64 (2.6.35.10-74.fc14.x86_64)

a boot look from the updated to current (as of time of posting) Fedora 14 box. 

This machine is now back in the configuration where the yum install commands randomly cause the panic

Comment 14 DemetrisK 2011-01-31 05:15:58 UTC
I am wondering is this is related with the problem I faced just know.
yum will return the following and then exit: 
 sudo yum update
[sudo] password for dimitris: 
Loaded plugins: fastestmirror, langpacks, presto, refresh-packagekit
Adding el_CY to language list
Loading mirror speeds from cached hostfile
updates/metalink                                         | 6.3 kB     00:00     
 * fedora: ftp.itu.edu.tr
 * rpmfusion-free: mirror.yandex.ru
 * rpmfusion-free-updates: mirror.yandex.ru
 * rpmfusion-nonfree: mirror.yandex.ru
 * rpmfusion-nonfree-updates: mirror.yandex.ru
 * updates: ftp.rhd.ru
adobe-linux-i386                                         |  951 B     00:00     
home_Gnurou                                              | 1.7 kB     00:00     
rpmfusion-free-updates                                   | 3.3 kB     00:00     
rpmfusion-nonfree-updates                                | 3.3 kB     00:00     
updates                                                  | 4.7 kB     00:00     

Message from syslogd@innovator at Jan 31 07:05:26 ...
 kernel:[384473.894492] Oops: 0000 [#3] SMP 

Message from syslogd@innovator at Jan 31 07:05:26 ...
 kernel:[384473.894495] last sysfs file: /sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_map

Message from syslogd@innovator at Jan 31 07:05:26 ...
 kernel:[384473.894590] Stack:

Message from syslogd@innovator at Jan 31 07:05:26 ...
 kernel:[384473.894603] Call Trace:

Message from syslogd@innovator at Jan 31 07:05:26 ...
 kernel:[384473.894641] Code: ff 74 1a 4c 89 06 49 2b 4b 08 eb 05 49 39 d8 72 b4 48 89 0a 31 c0 eb 05 b8 ea ff ff ff 5b 41 5c c9 c3 55 48 89 e5 0f 1f 44 00 00 <8b> 57 08 48 8d 4f 08 eb 02 89 c2 85 d2 74 14 8d 72 01 89 d0 f0 

Message from syslogd@innovator at Jan 31 07:05:26 ...
 kernel:[384473.894681] CR2: ffffea400110fdd8

Comment 15 Matt Darcy 2011-01-31 21:47:08 UTC
less than 24 hours after updating to the current Fedora 14 kernel (2.6.35.10-74.fc14.x86_64) a yup update on cifs-tools that was recently released (just the one package was being downloaded) caused a kernel panic and reboot as before

Comment 16 Chuck Ebbert 2011-02-01 06:55:27 UTC
(In reply to comment #13)
> Created attachment 476045 [details]
> Boot log from an updated Fedora 14 x86_64 (2.6.35.10-74.fc14.x86_64)
> 
> a boot look from the updated to current (as of time of posting) Fedora 14 box. 
> 
> This machine is now back in the configuration where the yum install commands
> randomly cause the panic

We need the kernel boot log (/var/log/dmesg). No need to upload the log from the current broken kernel, we have that from kdump, and there's no need for anything from F13 just yet. Just boot the working F14 kernel and then upload /var/log/dmesg from that.

Comment 17 Matt Darcy 2011-02-01 15:51:30 UTC
Created attachment 476423 [details]
/var/log/dmesg output from the stable F14 kernel

boot log from /var/log/dmesg from the stable F14 kernel.

Comment 18 Matt Darcy 2011-02-05 17:57:43 UTC
after more time on the stable F14 kernel I've managed to reproduce this crash with a yum install.

I should point out that this is on a Fedora 14 machine that is fully up to date, but using the earlier stable Fedora 14. I assume you'd like this re-installed to a Fedora 14 stock CD install and re-tested.

The problem I have is that when I do yum install from the earlier Fedora 14 base install it will update dependencies.

Comment 19 Matt Darcy 2011-03-16 16:46:46 UTC
is there more information I can provide on this as this machine is still crashing on a regular basis and is now out of use.

Comment 20 Matt Darcy 2011-07-12 07:39:46 UTC
suggest closing this bu as it doesn't apper to be going anywhere.

As the bug raiser I am happy for it to beclosed

Comment 21 Josh Boyer 2011-08-29 20:56:50 UTC
Closing per comment #20