Bug 681112

Summary: Opps 0002 [#11] SMP, possible bug - 2.6.35.11-83.fc14.x86_64
Product: [Fedora] Fedora Reporter: Naoki <naoki>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: 14CC: gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-29 17:35:14 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Naoki 2011-03-01 06:31:24 UTC 
Description of problem:
[100451.852475] BUG: unable to handle kernel paging request at ffff8801d0033000
[100451.853379] IP: [<ffffffff81220717>] clear_page_c+0x7/0x10
[100451.853379] PGD 1a43063 PUD f067 PMD 80000001d0000100 
[100451.853379] Oops: 0002 [#11] SMP 
[100451.853379] last sysfs file: /sys/devices/pci0000:00/0000:00:02.1/host0/target0:0:0/0:0:0:0/block/sr0/stat
[100451.853379] CPU 3 
[100451.853379] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf]
[100451.853379] 
[100451.853379] Pid: 1890, comm: agetty Tainted: G      D     2.6.35.11-83.fc14.x86_64 #1 7984/IBM System  x3455-[798474J]-
[100451.853379] RIP: 0010:[<ffffffff81220717>]  [<ffffffff81220717>] clear_page_c+0x7/0x10
[100451.853379] RSP: 0018:ffff8802a48978b0  EFLAGS: 00010246
[100451.853379] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000200
[100451.853379] RDX: 0000000006580b28 RSI: 6db6db6db6db6db7 RDI: ffff8801d0033000
[100451.853379] RBP: ffff8802a48979c8 R08: ffffea0006580b50 R09: 00000000000c982a
[100451.853379] R10: 0000000000000002 R11: 0000000000000a99 R12: 0000000000000030
[100451.853379] R13: ffff880100001c00 R14: ffffea0006580b28 R15: ffff8802a4896000
[100451.853379] FS:  00007fe6771a2720(0000) GS:ffff880108500000(0000) knlGS:0000000000000000
[100451.853379] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[100451.853379] CR2: ffff8801d0033000 CR3: 00000002a54a3000 CR4: 00000000000006e0
[100451.853379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[100451.853379] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[100451.853379] Process agetty (pid: 1890, threadinfo ffff8802a4896000, task ffff8802a4a20000)
[100451.853379] Stack:
[100451.853379]  ffffffff810d9be9 0000000006580b28 0000000000000000 0000000000000001
[100451.853379] <0> ffff8802a4897fd8 0000000008510630 ffff8802a4a20000 0000000000000001
[100451.853379] <0> ffffffff00015500 ffff8802a48979f8 0000004081048a7e 0000000000000002
[100451.853379] Call Trace:
[100451.853379]  [<ffffffff810d9be9>] ? get_page_from_freelist+0x4c7/0x674
[100451.853379]  [<ffffffff810d9ee8>] __alloc_pages_nodemask+0x152/0x776
[100451.853379]  [<ffffffff810780c5>] ? __raw_local_irq_save+0x1d/0x23
[100451.853379]  [<ffffffff81469f7a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[100451.853379]  [<ffffffff81059f58>] ? lock_timer_base.clone.24+0x2b/0x50
[100451.853379]  [<ffffffff8103c14a>] ? need_resched+0x23/0x2d
[100451.853379]  [<ffffffff81101f59>] alloc_page_vma+0xce/0xd3
[100451.853379]  [<ffffffff810ebf4c>] handle_mm_fault+0x277/0x84d
[100451.853379]  [<ffffffff81109c4c>] ? kmem_cache_alloc+0x92/0x105
[100451.853379]  [<ffffffff810d8145>] ? get_pageblock_flags_group+0x4a/0x81
[100451.853379]  [<ffffffff810d9558>] ? free_hot_cold_page+0x144/0x153
[100451.853379]  [<ffffffff8146d375>] do_page_fault+0x250/0x265
[100451.853379]  [<ffffffff8146a6f5>] page_fault+0x25/0x30
[100451.853379]  [<ffffffff810d3337>] ? file_read_actor+0x39/0x10c
[100451.853379]  [<ffffffff8121c2c6>] ? radix_tree_lookup_slot+0xe/0x10
[100451.853379]  [<ffffffff810d4ee0>] generic_file_aio_read+0x391/0x5b6
[100451.853379]  [<ffffffff81116e3e>] do_sync_read+0xcb/0x108
[100451.853379]  [<ffffffff811e3d62>] ? selinux_file_permission+0x5a/0xb9
[100451.853379]  [<ffffffff811dc789>] ? security_file_permission+0x16/0x18
[100451.853379]  [<ffffffff8111750d>] vfs_read+0xa9/0xfd
[100451.853379]  [<ffffffff811175ab>] sys_read+0x4a/0x6e
[100451.853379]  [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b
[100451.853379] Code: 89 f3 48 83 ec 08 e8 ca fa ff ff 8d 53 ff 48 63 c8 48 39 d9 41 5a 0f 43 c2 5b c9 c3 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 <f3> 48 ab c3 0f 1f 44 00 00 eb ee 66 66 66 90 66 66 66 90 66 66 
[100451.853379] RIP  [<ffffffff81220717>] clear_page_c+0x7/0x10
[100451.853379]  RSP <ffff8802a48978b0>
[100451.853379] CR2: ffff8801d0033000
[100451.853379] ---[ end trace e75160529f0356b3 ]---
[100451.853379] note: agetty[1890] exited with preempt_count 1
[100454.115000] BUG: scheduling while atomic: agetty/1890/0x10000001
[100454.151494] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf]
[100454.369556] Pid: 1890, comm: agetty Tainted: G      D     2.6.35.11-83.fc14.x86_64 #1
[100454.416974] Call Trace:
[100454.432156]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
[100454.465535]  [<ffffffff8146845e>] schedule+0xd9/0x5cb
[100454.496317]  [<ffffffff81049246>] __cond_resched+0x2a/0x35
[100454.529693]  [<ffffffff81468a11>] _cond_resched+0x1b/0x22
[100454.562553]  [<ffffffff81469639>] down_read+0x29/0x3b
[100454.593334]  [<ffffffff81050e80>] exit_mm+0x3c/0x127
[100454.623593]  [<ffffffff810511ba>] do_exit+0x24f/0x74f
[100454.654373]  [<ffffffff81469fcf>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[100454.695555]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
[100454.725814]  [<ffffffff810324dc>] no_context+0x1f9/0x208
[100454.758155]  [<ffffffff8103d357>] ? update_curr+0xd9/0xe2
[100454.791015]  [<ffffffff8103267a>] __bad_area_nosemaphore+0x18f/0x1b2
[100454.829594]  [<ffffffff81031cce>] ? pud_page_vaddr+0xe/0x2a
[100454.863497]  [<ffffffff81031d09>] ? pmd_offset+0x1f/0x25
[100454.895838]  [<ffffffff810326b0>] bad_area_nosemaphore+0x13/0x15
[100454.932333]  [<ffffffff8146d25f>] do_page_fault+0x13a/0x265
[100454.966232]  [<ffffffff8146a6f5>] page_fault+0x25/0x30
[100454.997534]  [<ffffffff81220717>] ? clear_page_c+0x7/0x10
[100455.030391]  [<ffffffff810d9be9>] ? get_page_from_freelist+0x4c7/0x674
[100455.070016]  [<ffffffff810d9ee8>] __alloc_pages_nodemask+0x152/0x776
[100455.108588]  [<ffffffff810780c5>] ? __raw_local_irq_save+0x1d/0x23
[100455.146128]  [<ffffffff81469f7a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[100455.184709]  [<ffffffff81059f58>] ? lock_timer_base.clone.24+0x2b/0x50
[100455.224326]  [<ffffffff8103c14a>] ? need_resched+0x23/0x2d
[100455.257704]  [<ffffffff81101f59>] alloc_page_vma+0xce/0xd3
[100455.291083]  [<ffffffff810ebf4c>] handle_mm_fault+0x277/0x84d
[100455.326023]  [<ffffffff81109c4c>] ? kmem_cache_alloc+0x92/0x105
[100455.362000]  [<ffffffff810d8145>] ? get_pageblock_flags_group+0x4a/0x81
[100455.402138]  [<ffffffff810d9558>] ? free_hot_cold_page+0x144/0x153
[100455.439673]  [<ffffffff8146d375>] do_page_fault+0x250/0x265
[100455.473573]  [<ffffffff8146a6f5>] page_fault+0x25/0x30
[100455.504874]  [<ffffffff810d3337>] ? file_read_actor+0x39/0x10c
[100455.540336]  [<ffffffff8121c2c6>] ? radix_tree_lookup_slot+0xe/0x10
[100455.578393]  [<ffffffff810d4ee0>] generic_file_aio_read+0x391/0x5b6
[100455.616453]  [<ffffffff81116e3e>] do_sync_read+0xcb/0x108
[100455.649311]  [<ffffffff811e3d62>] ? selinux_file_permission+0x5a/0xb9
[100455.688411]  [<ffffffff811dc789>] ? security_file_permission+0x16/0x18
[100455.728030]  [<ffffffff8111750d>] vfs_read+0xa9/0xfd
[100455.758290]  [<ffffffff811175ab>] sys_read+0x4a/0x6e
[100455.788546]  [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b
[100455.825235] BUG: scheduling while atomic: agetty/1890/0x10000001
[100455.861708] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf]
[100456.079760] Pid: 1890, comm: agetty Tainted: G      D     2.6.35.11-83.fc14.x86_64 #1
[100456.127179] Call Trace:
[100456.142361]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
[100456.175738]  [<ffffffff8146845e>] schedule+0xd9/0x5cb
[100456.206515]  [<ffffffff81049246>] __cond_resched+0x2a/0x35
[100456.239897]  [<ffffffff81468a11>] _cond_resched+0x1b/0x22
[100456.272752]  [<ffffffff81050bd6>] put_files_struct+0x86/0xd5
[100456.307170]  [<ffffffff81050cb6>] exit_files+0x41/0x46
[100456.338470]  [<ffffffff81051200>] do_exit+0x295/0x74f
[100456.369251]  [<ffffffff81469fcf>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[100456.410431]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
[100456.440695]  [<ffffffff810324dc>] no_context+0x1f9/0x208
[100456.473034]  [<ffffffff8103d357>] ? update_curr+0xd9/0xe2
[100456.505891]  [<ffffffff8103267a>] __bad_area_nosemaphore+0x18f/0x1b2
[100456.544469]  [<ffffffff81031cce>] ? pud_page_vaddr+0xe/0x2a
[100456.578369]  [<ffffffff81031d09>] ? pmd_offset+0x1f/0x25
[100456.610707]  [<ffffffff810326b0>] bad_area_nosemaphore+0x13/0x15
[100456.647210]  [<ffffffff8146d25f>] do_page_fault+0x13a/0x265
[100456.681108]  [<ffffffff8146a6f5>] page_fault+0x25/0x30
[100456.712409]  [<ffffffff81220717>] ? clear_page_c+0x7/0x10
[100456.745263]  [<ffffffff810d9be9>] ? get_page_from_freelist+0x4c7/0x674
[100456.784882]  [<ffffffff810d9ee8>] __alloc_pages_nodemask+0x152/0x776
[100456.823464]  [<ffffffff810780c5>] ? __raw_local_irq_save+0x1d/0x23
[100456.861004]  [<ffffffff81469f7a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[100456.899579]  [<ffffffff81059f58>] ? lock_timer_base.clone.24+0x2b/0x50
[100456.939195]  [<ffffffff8103c14a>] ? need_resched+0x23/0x2d
[100456.972579]  [<ffffffff81101f59>] alloc_page_vma+0xce/0xd3
[100457.005956]  [<ffffffff810ebf4c>] handle_mm_fault+0x277/0x84d
[100457.040895]  [<ffffffff81109c4c>] ? kmem_cache_alloc+0x92/0x105
[100457.076871]  [<ffffffff810d8145>] ? get_pageblock_flags_group+0x4a/0x81
[100457.117002]  [<ffffffff810d9558>] ? free_hot_cold_page+0x144/0x153
[100457.154537]  [<ffffffff8146d375>] do_page_fault+0x250/0x265
[100457.188432]  [<ffffffff8146a6f5>] page_fault+0x25/0x30
[100457.219729]  [<ffffffff810d3337>] ? file_read_actor+0x39/0x10c
[100457.255192]  [<ffffffff8121c2c6>] ? radix_tree_lookup_slot+0xe/0x10
[100457.293245]  [<ffffffff810d4ee0>] generic_file_aio_read+0x391/0x5b6
[100457.331304]  [<ffffffff81116e3e>] do_sync_read+0xcb/0x108
[100457.364164]  [<ffffffff811e3d62>] ? selinux_file_permission+0x5a/0xb9
[100457.403265]  [<ffffffff811dc789>] ? security_file_permission+0x16/0x18
[100457.442880]  [<ffffffff8111750d>] vfs_read+0xa9/0xfd
[100457.473136]  [<ffffffff811175ab>] sys_read+0x4a/0x6e
[100457.503395]  [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b
[100457.548372] BUG: unable to handle kernel paging request at ffff8801d0034000
[100457.549357] IP: [<ffffffff81220765>] copy_page_c+0x5/0x10
[100457.549357] PGD 1a43063 PUD f067 PMD 80000001d0000100 
[100457.54 R14: ffff8800272060d8 R15: 00007f9f037b89f0
[100457.549357] FS:  00007f9f037b8720(0000) GS:ffff880108500000(0000) knlGS:0000000000000000
[100457.549357] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[100457.549357] CR2: ffff8801d0034000 CR3: 0000000026d05000 CR4: 00000000000006e0
[100457.549357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[100457.549357] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[100457.549357] Process init (pid: 24595, threadinfo ffff88003bf54000, task ffff880027210000)
[100457.549357] Stack:
[100457.549357]  ffffffff810e9b41 ffffea0000ceda20 ffffea0006580b60 ffff88003bf55d78
[100457.549357] <0> ffffffff810eaa07 0000000000000000 0000000000000000 ffff880026ed0a80
[100457.549357] <0> ffff88003d374dc0 0000000000000000 0000000000000000 0000000000000000
[100457.549357] Call Trace:
[100457.549357]  [<ffffffff810e9b41>] ? copy_user_highpage.clone.41+0x2d/0x3c
[100457.549357]  [<ffffffff810eaa07>] do_wp_page+0x327/0x58e
[100457.549357]  [<ffffffff810e9519>] ? pmd_offset+0x19/0x40
[100457.549357]  [<ffffffff810ec4c4>] handle_mm_fault+0x7ef/0x84d
[100457.549357]  [<ffffffff8146d375>] do_page_fault+0x250/0x265
[100457.549357]  [<ffffffff8146a6f5>] page_fault+0x25/0x30
[100457.549357]  [<ffffffff8122183d>] ? __put_user_4+0x1d/0x30
[100457.549357]  [<ffffffff810494cb>] ? schedule_tail+0x64/0x68
[100457.549357]  [<ffffffff81009bf3>] ret_from_fork+0x13/0x80
[100457.549357] Code: 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 90 90 90 90 90 90 90 90 b9 00 02 00 00 <f3> 48 a5 c3 0f 1f 80 00 00 00 00 eb ee 66 66 66 90 66 66 66 90 
[100457.549357] RIP  [<ffffffff81220765>] copy_page_c+0x5/0x10
[100457.549357]  RSP <ffff88003bf55ce0>
[100457.549357] CR2: ffff8801d0034000
[100457.549357] ---[ end trace e75160529f0356b4 ]---
[100457.549357] note: init[24595] exited with preempt_count 2
[100459.319223] BUG: scheduling while atomic: init/24595/0x10000002
[100459.355164] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf]
[100459.573210] Pid: 24595, comm: init Tainted: G      D     2.6.35.11-83.fc14.x86_64 #1
[100459.620108] Call Trace:
[100459.635291]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
[100459.668669]  [<ffffffff8146845e>] schedule+0xd9/0x5cb
[100459.699450]  [<ffffffff81049246>] __cond_resched+0x2a/0x35
[100459.732826]  [<ffffffff81468a11>] _cond_resched+0x1b/0x22
[100459.765687]  [<ffffffff81469639>] down_read+0x29/0x3b
[100459.796464]  [<ffffffff81050e80>] exit_mm+0x3c/0x127
[100459.826725]  [<ffffffff810511ba>] do_exit+0x24f/0x74f
[100459.857505]  [<ffffffff81469fcf>] ? ~. [terminated ipmitool]


Version-Release number of selected component (if applicable):
2.6.35.11-83.fc14.x86_64

How reproducible:
So far unknown.
Comment 1 Chuck Ebbert 2011-03-01 23:10:46 UTC 
(In reply to comment #1)
> [100451.853379] Oops: 0002 [#11] SMP 
                              ^^^ 
This is the 11th oops the machine has encountered since it was booted. What did the other oops messages look like?
Comment 2 Naoki 2011-03-07 01:59:23 UTC 
Hi Chuck, some more info below. But I should mention I've seen many, many, ECC errors coming up 

[255413.887870] ECC/ChipKill ECC error.
[255413.909287] EDAC amd64 MC1: CE ERROR_ADDRESS= 0x29a98f4f0
[255413.942145] EDAC amd64 MC1: Failed to translate InputAddr to csrow for address 0x29a98f4f0
[255413.992149] EDAC MC1: CE - no information available: amd64_edac
[255419.036310]  Northbridge Error, node 1, core: 0
[255419.064142] ECC/ChipKill ECC error.
[255419.085562] EDAC amd64 MC1: CE ERROR_ADDRESS= 0x292b8f4f0
[255419.118425] EDAC amd64 MC1: Failed to translate InputAddr to csrow for address 0x292b8f4f0
[255419.168433] EDAC MC1: CE - no information available: amd64_edac
[255420.206042]  Northbridge Error, node 1, core: 1
[255420.233892] ECC/ChipKill ECC error.
[255420.255315] EDAC amd64 MC1: CE ERROR_ADDRESS= 0x2a118f4f8
[255420.288178] EDAC amd64 MC1: Failed to translate InputAddr to csrow for address 0x2a118f4f8
[255420.338184] EDAC MC1: CE - no information available: amd64_edac

I'd like to figure out which DIMM is the problem but am stumped at the "Failed to translate" which leaves me in the dark a bit.

# grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count
/sys/devices/system/edac/mc/mc0/csrow4/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow4/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow2/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow2/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow3/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow3/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow4/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow4/ch1_ce_count:0


[root.prod ~]# zgrep -i Oops /var/log/messages*gz
/var/log/messages-20110306.gz:Mar  2 14:56:33 pdbsearch11 kernel: [84164.552150] Oops: 0002 [#1] SMP 
/var/log/messages-20110306.gz:Mar  2 14:56:33 pdbsearch11 kernel: [84167.120384]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 14:57:38 pdbsearch11 kernel: [84230.093109] Oops: 0002 [#2] SMP 
/var/log/messages-20110306.gz:Mar  2 14:57:39 pdbsearch11 kernel: [84232.580624]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 14:59:31 pdbsearch11 kernel: [84342.718259] Oops: 0002 [#3] SMP 
/var/log/messages-20110306.gz:Mar  2 14:59:31 pdbsearch11 kernel: [84345.349656]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.416134] Oops: 0002 [#4] SMP 
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.462131] Oops: 0002 [#5] SMP 
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84377.032740]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84378.492327] Oops: 0002 [#6] SMP 
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.416134]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:00:10 pdbsearch11 kernel: [84383.558089]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:01:03 pdbsearch11 kernel: [84435.091327] Oops: 0002 [#7] SMP 
/var/log/messages-20110306.gz:Mar  2 15:01:03 pdbsearch11 kernel: [84437.345607]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:03 pdbsearch11 kernel: [84674.879419] Oops: 0002 [#8] SMP 
/var/log/messages-20110306.gz:Mar  2 15:05:03 pdbsearch11 kernel: [84677.133804]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84678.274255] Oops: 0002 [#9] SMP 
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84678.511385] Oops: 0002 [#10] SMP 
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84678.533136] Oops: 0002 [#11] SMP 
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84679.933911]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84679.934021]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84679.934372]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84679.936170]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:15 pdbsearch11 kernel: [84688.601699]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7
/var/log/messages-20110306.gz:Mar  2 15:05:16 pdbsearch11 kernel: [84689.805922]  [<ffffffff8146b22e>] oops_end+0xbf/0xc7



/var/log/messages-20110306.gz:Mar  2 14:56:33 pdbsearch11 kernel: [84164.527752] BUG: unable to handle kernel paging request at ffff8801d002c000
/var/log/messages-20110306.gz:Mar  2 14:56:33 pdbsearch11 kernel: [84166.547148] BUG: scheduling while atomic: snmpd/1382/0x10000001
/var/log/messages-20110306.gz:Mar  2 14:56:33 pdbsearch11 kernel: [84166.861164]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 14:57:38 pdbsearch11 kernel: [84230.092206] BUG: unable to handle kernel paging request at ffff8801d002d000
/var/log/messages-20110306.gz:Mar  2 14:57:38 pdbsearch11 kernel: [84231.969912] BUG: scheduling while atomic: glusterfsd/1445/0x10000001
/var/log/messages-20110306.gz:Mar  2 14:57:38 pdbsearch11 kernel: [84232.289088]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 14:59:31 pdbsearch11 kernel: [84342.717329] BUG: unable to handle kernel paging request at ffff8801d002e000
/var/log/messages-20110306.gz:Mar  2 14:59:31 pdbsearch11 kernel: [84344.777429] BUG: scheduling while atomic: ntpd/1406/0x10000001
/var/log/messages-20110306.gz:Mar  2 14:59:31 pdbsearch11 kernel: [84345.090420]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.410627] BUG: unable to handle kernel paging request at ffff8801d002f000
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.462131] BUG: unable to handle kernel paging request at ffff8801d0041000
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84377.032686] BUG: scheduling while atomic: munin-update/1698/0x10000001
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84377.032706]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84378.492306] BUG: unable to handle kernel paging request at ffff8801d0042000
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.416134] BUG: scheduling while atomic: crond/1693/0x10000002
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84375.416134]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84382.982250] BUG: scheduling while atomic: crond/1693/0x10000002
/var/log/messages-20110306.gz:Mar  2 15:00:09 pdbsearch11 kernel: [84383.296261]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 15:01:03 pdbsearch11 kernel: [84435.090381] BUG: unable to handle kernel paging request at ffff8801d0030000
/var/log/messages-20110306.gz:Mar  2 15:01:03 pdbsearch11 kernel: [84436.772360] BUG: scheduling while atomic: crond/1703/0x10000002
/var/log/messages-20110306.gz:Mar  2 15:01:03 pdbsearch11 kernel: [84437.086367]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 15:05:03 pdbsearch11 kernel: [84674.878472] BUG: unable to handle kernel paging request at ffff8801d0031000
/var/log/messages-20110306.gz:Mar  2 15:05:03 pdbsearch11 kernel: [84676.560558] BUG: scheduling while atomic: crond/1704/0x10000002
/var/log/messages-20110306.gz:Mar  2 15:05:03 pdbsearch11 kernel: [84676.874558]  [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84678.274136] BUG: unable to handle kernel paging request at ffff8801d0032000
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84678.511385] BUG: unable to handle kernel paging request at ffff8801d0043000
/var/log/messages-20110306.gz:Mar  2 15:05:14 pdbsearch11 kernel: [84678.533136] BUG: unable to handle kernel paging request at ffff8801d0044000
Comment 3 Josh Boyer 2011-08-29 17:35:14 UTC 
This appears to be hardware related.  There isn't much we can do about this.