Hide Forgot
Description of problem: [100451.852475] BUG: unable to handle kernel paging request at ffff8801d0033000 [100451.853379] IP: [<ffffffff81220717>] clear_page_c+0x7/0x10 [100451.853379] PGD 1a43063 PUD f067 PMD 80000001d0000100 [100451.853379] Oops: 0002 [#11] SMP [100451.853379] last sysfs file: /sys/devices/pci0000:00/0000:00:02.1/host0/target0:0:0/0:0:0:0/block/sr0/stat [100451.853379] CPU 3 [100451.853379] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf] [100451.853379] [100451.853379] Pid: 1890, comm: agetty Tainted: G D 2.6.35.11-83.fc14.x86_64 #1 7984/IBM System x3455-[798474J]- [100451.853379] RIP: 0010:[<ffffffff81220717>] [<ffffffff81220717>] clear_page_c+0x7/0x10 [100451.853379] RSP: 0018:ffff8802a48978b0 EFLAGS: 00010246 [100451.853379] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000200 [100451.853379] RDX: 0000000006580b28 RSI: 6db6db6db6db6db7 RDI: ffff8801d0033000 [100451.853379] RBP: ffff8802a48979c8 R08: ffffea0006580b50 R09: 00000000000c982a [100451.853379] R10: 0000000000000002 R11: 0000000000000a99 R12: 0000000000000030 [100451.853379] R13: ffff880100001c00 R14: ffffea0006580b28 R15: ffff8802a4896000 [100451.853379] FS: 00007fe6771a2720(0000) GS:ffff880108500000(0000) knlGS:0000000000000000 [100451.853379] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [100451.853379] CR2: ffff8801d0033000 CR3: 00000002a54a3000 CR4: 00000000000006e0 [100451.853379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [100451.853379] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [100451.853379] Process agetty (pid: 1890, threadinfo ffff8802a4896000, task ffff8802a4a20000) [100451.853379] Stack: [100451.853379] ffffffff810d9be9 0000000006580b28 0000000000000000 0000000000000001 [100451.853379] <0> ffff8802a4897fd8 0000000008510630 ffff8802a4a20000 0000000000000001 [100451.853379] <0> ffffffff00015500 ffff8802a48979f8 0000004081048a7e 0000000000000002 [100451.853379] Call Trace: [100451.853379] [<ffffffff810d9be9>] ? get_page_from_freelist+0x4c7/0x674 [100451.853379] [<ffffffff810d9ee8>] __alloc_pages_nodemask+0x152/0x776 [100451.853379] [<ffffffff810780c5>] ? __raw_local_irq_save+0x1d/0x23 [100451.853379] [<ffffffff81469f7a>] ? _raw_spin_lock_irqsave+0x12/0x2f [100451.853379] [<ffffffff81059f58>] ? lock_timer_base.clone.24+0x2b/0x50 [100451.853379] [<ffffffff8103c14a>] ? need_resched+0x23/0x2d [100451.853379] [<ffffffff81101f59>] alloc_page_vma+0xce/0xd3 [100451.853379] [<ffffffff810ebf4c>] handle_mm_fault+0x277/0x84d [100451.853379] [<ffffffff81109c4c>] ? kmem_cache_alloc+0x92/0x105 [100451.853379] [<ffffffff810d8145>] ? get_pageblock_flags_group+0x4a/0x81 [100451.853379] [<ffffffff810d9558>] ? free_hot_cold_page+0x144/0x153 [100451.853379] [<ffffffff8146d375>] do_page_fault+0x250/0x265 [100451.853379] [<ffffffff8146a6f5>] page_fault+0x25/0x30 [100451.853379] [<ffffffff810d3337>] ? file_read_actor+0x39/0x10c [100451.853379] [<ffffffff8121c2c6>] ? radix_tree_lookup_slot+0xe/0x10 [100451.853379] [<ffffffff810d4ee0>] generic_file_aio_read+0x391/0x5b6 [100451.853379] [<ffffffff81116e3e>] do_sync_read+0xcb/0x108 [100451.853379] [<ffffffff811e3d62>] ? selinux_file_permission+0x5a/0xb9 [100451.853379] [<ffffffff811dc789>] ? security_file_permission+0x16/0x18 [100451.853379] [<ffffffff8111750d>] vfs_read+0xa9/0xfd [100451.853379] [<ffffffff811175ab>] sys_read+0x4a/0x6e [100451.853379] [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b [100451.853379] Code: 89 f3 48 83 ec 08 e8 ca fa ff ff 8d 53 ff 48 63 c8 48 39 d9 41 5a 0f 43 c2 5b c9 c3 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 <f3> 48 ab c3 0f 1f 44 00 00 eb ee 66 66 66 90 66 66 66 90 66 66 [100451.853379] RIP [<ffffffff81220717>] clear_page_c+0x7/0x10 [100451.853379] RSP <ffff8802a48978b0> [100451.853379] CR2: ffff8801d0033000 [100451.853379] ---[ end trace e75160529f0356b3 ]--- [100451.853379] note: agetty[1890] exited with preempt_count 1 [100454.115000] BUG: scheduling while atomic: agetty/1890/0x10000001 [100454.151494] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf] [100454.369556] Pid: 1890, comm: agetty Tainted: G D 2.6.35.11-83.fc14.x86_64 #1 [100454.416974] Call Trace: [100454.432156] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 [100454.465535] [<ffffffff8146845e>] schedule+0xd9/0x5cb [100454.496317] [<ffffffff81049246>] __cond_resched+0x2a/0x35 [100454.529693] [<ffffffff81468a11>] _cond_resched+0x1b/0x22 [100454.562553] [<ffffffff81469639>] down_read+0x29/0x3b [100454.593334] [<ffffffff81050e80>] exit_mm+0x3c/0x127 [100454.623593] [<ffffffff810511ba>] do_exit+0x24f/0x74f [100454.654373] [<ffffffff81469fcf>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [100454.695555] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 [100454.725814] [<ffffffff810324dc>] no_context+0x1f9/0x208 [100454.758155] [<ffffffff8103d357>] ? update_curr+0xd9/0xe2 [100454.791015] [<ffffffff8103267a>] __bad_area_nosemaphore+0x18f/0x1b2 [100454.829594] [<ffffffff81031cce>] ? pud_page_vaddr+0xe/0x2a [100454.863497] [<ffffffff81031d09>] ? pmd_offset+0x1f/0x25 [100454.895838] [<ffffffff810326b0>] bad_area_nosemaphore+0x13/0x15 [100454.932333] [<ffffffff8146d25f>] do_page_fault+0x13a/0x265 [100454.966232] [<ffffffff8146a6f5>] page_fault+0x25/0x30 [100454.997534] [<ffffffff81220717>] ? clear_page_c+0x7/0x10 [100455.030391] [<ffffffff810d9be9>] ? get_page_from_freelist+0x4c7/0x674 [100455.070016] [<ffffffff810d9ee8>] __alloc_pages_nodemask+0x152/0x776 [100455.108588] [<ffffffff810780c5>] ? __raw_local_irq_save+0x1d/0x23 [100455.146128] [<ffffffff81469f7a>] ? _raw_spin_lock_irqsave+0x12/0x2f [100455.184709] [<ffffffff81059f58>] ? lock_timer_base.clone.24+0x2b/0x50 [100455.224326] [<ffffffff8103c14a>] ? need_resched+0x23/0x2d [100455.257704] [<ffffffff81101f59>] alloc_page_vma+0xce/0xd3 [100455.291083] [<ffffffff810ebf4c>] handle_mm_fault+0x277/0x84d [100455.326023] [<ffffffff81109c4c>] ? kmem_cache_alloc+0x92/0x105 [100455.362000] [<ffffffff810d8145>] ? get_pageblock_flags_group+0x4a/0x81 [100455.402138] [<ffffffff810d9558>] ? free_hot_cold_page+0x144/0x153 [100455.439673] [<ffffffff8146d375>] do_page_fault+0x250/0x265 [100455.473573] [<ffffffff8146a6f5>] page_fault+0x25/0x30 [100455.504874] [<ffffffff810d3337>] ? file_read_actor+0x39/0x10c [100455.540336] [<ffffffff8121c2c6>] ? radix_tree_lookup_slot+0xe/0x10 [100455.578393] [<ffffffff810d4ee0>] generic_file_aio_read+0x391/0x5b6 [100455.616453] [<ffffffff81116e3e>] do_sync_read+0xcb/0x108 [100455.649311] [<ffffffff811e3d62>] ? selinux_file_permission+0x5a/0xb9 [100455.688411] [<ffffffff811dc789>] ? security_file_permission+0x16/0x18 [100455.728030] [<ffffffff8111750d>] vfs_read+0xa9/0xfd [100455.758290] [<ffffffff811175ab>] sys_read+0x4a/0x6e [100455.788546] [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b [100455.825235] BUG: scheduling while atomic: agetty/1890/0x10000001 [100455.861708] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf] [100456.079760] Pid: 1890, comm: agetty Tainted: G D 2.6.35.11-83.fc14.x86_64 #1 [100456.127179] Call Trace: [100456.142361] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 [100456.175738] [<ffffffff8146845e>] schedule+0xd9/0x5cb [100456.206515] [<ffffffff81049246>] __cond_resched+0x2a/0x35 [100456.239897] [<ffffffff81468a11>] _cond_resched+0x1b/0x22 [100456.272752] [<ffffffff81050bd6>] put_files_struct+0x86/0xd5 [100456.307170] [<ffffffff81050cb6>] exit_files+0x41/0x46 [100456.338470] [<ffffffff81051200>] do_exit+0x295/0x74f [100456.369251] [<ffffffff81469fcf>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [100456.410431] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 [100456.440695] [<ffffffff810324dc>] no_context+0x1f9/0x208 [100456.473034] [<ffffffff8103d357>] ? update_curr+0xd9/0xe2 [100456.505891] [<ffffffff8103267a>] __bad_area_nosemaphore+0x18f/0x1b2 [100456.544469] [<ffffffff81031cce>] ? pud_page_vaddr+0xe/0x2a [100456.578369] [<ffffffff81031d09>] ? pmd_offset+0x1f/0x25 [100456.610707] [<ffffffff810326b0>] bad_area_nosemaphore+0x13/0x15 [100456.647210] [<ffffffff8146d25f>] do_page_fault+0x13a/0x265 [100456.681108] [<ffffffff8146a6f5>] page_fault+0x25/0x30 [100456.712409] [<ffffffff81220717>] ? clear_page_c+0x7/0x10 [100456.745263] [<ffffffff810d9be9>] ? get_page_from_freelist+0x4c7/0x674 [100456.784882] [<ffffffff810d9ee8>] __alloc_pages_nodemask+0x152/0x776 [100456.823464] [<ffffffff810780c5>] ? __raw_local_irq_save+0x1d/0x23 [100456.861004] [<ffffffff81469f7a>] ? _raw_spin_lock_irqsave+0x12/0x2f [100456.899579] [<ffffffff81059f58>] ? lock_timer_base.clone.24+0x2b/0x50 [100456.939195] [<ffffffff8103c14a>] ? need_resched+0x23/0x2d [100456.972579] [<ffffffff81101f59>] alloc_page_vma+0xce/0xd3 [100457.005956] [<ffffffff810ebf4c>] handle_mm_fault+0x277/0x84d [100457.040895] [<ffffffff81109c4c>] ? kmem_cache_alloc+0x92/0x105 [100457.076871] [<ffffffff810d8145>] ? get_pageblock_flags_group+0x4a/0x81 [100457.117002] [<ffffffff810d9558>] ? free_hot_cold_page+0x144/0x153 [100457.154537] [<ffffffff8146d375>] do_page_fault+0x250/0x265 [100457.188432] [<ffffffff8146a6f5>] page_fault+0x25/0x30 [100457.219729] [<ffffffff810d3337>] ? file_read_actor+0x39/0x10c [100457.255192] [<ffffffff8121c2c6>] ? radix_tree_lookup_slot+0xe/0x10 [100457.293245] [<ffffffff810d4ee0>] generic_file_aio_read+0x391/0x5b6 [100457.331304] [<ffffffff81116e3e>] do_sync_read+0xcb/0x108 [100457.364164] [<ffffffff811e3d62>] ? selinux_file_permission+0x5a/0xb9 [100457.403265] [<ffffffff811dc789>] ? security_file_permission+0x16/0x18 [100457.442880] [<ffffffff8111750d>] vfs_read+0xa9/0xfd [100457.473136] [<ffffffff811175ab>] sys_read+0x4a/0x6e [100457.503395] [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b [100457.548372] BUG: unable to handle kernel paging request at ffff8801d0034000 [100457.549357] IP: [<ffffffff81220765>] copy_page_c+0x5/0x10 [100457.549357] PGD 1a43063 PUD f067 PMD 80000001d0000100 [100457.54 R14: ffff8800272060d8 R15: 00007f9f037b89f0 [100457.549357] FS: 00007f9f037b8720(0000) GS:ffff880108500000(0000) knlGS:0000000000000000 [100457.549357] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [100457.549357] CR2: ffff8801d0034000 CR3: 0000000026d05000 CR4: 00000000000006e0 [100457.549357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [100457.549357] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [100457.549357] Process init (pid: 24595, threadinfo ffff88003bf54000, task ffff880027210000) [100457.549357] Stack: [100457.549357] ffffffff810e9b41 ffffea0000ceda20 ffffea0006580b60 ffff88003bf55d78 [100457.549357] <0> ffffffff810eaa07 0000000000000000 0000000000000000 ffff880026ed0a80 [100457.549357] <0> ffff88003d374dc0 0000000000000000 0000000000000000 0000000000000000 [100457.549357] Call Trace: [100457.549357] [<ffffffff810e9b41>] ? copy_user_highpage.clone.41+0x2d/0x3c [100457.549357] [<ffffffff810eaa07>] do_wp_page+0x327/0x58e [100457.549357] [<ffffffff810e9519>] ? pmd_offset+0x19/0x40 [100457.549357] [<ffffffff810ec4c4>] handle_mm_fault+0x7ef/0x84d [100457.549357] [<ffffffff8146d375>] do_page_fault+0x250/0x265 [100457.549357] [<ffffffff8146a6f5>] page_fault+0x25/0x30 [100457.549357] [<ffffffff8122183d>] ? __put_user_4+0x1d/0x30 [100457.549357] [<ffffffff810494cb>] ? schedule_tail+0x64/0x68 [100457.549357] [<ffffffff81009bf3>] ret_from_fork+0x13/0x80 [100457.549357] Code: 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90 66 66 90 90 90 90 90 90 90 90 b9 00 02 00 00 <f3> 48 a5 c3 0f 1f 80 00 00 00 00 eb ee 66 66 66 90 66 66 66 90 [100457.549357] RIP [<ffffffff81220765>] copy_page_c+0x5/0x10 [100457.549357] RSP <ffff88003bf55ce0> [100457.549357] CR2: ffff8801d0034000 [100457.549357] ---[ end trace e75160529f0356b4 ]--- [100457.549357] note: init[24595] exited with preempt_count 2 [100459.319223] BUG: scheduling while atomic: init/24595/0x10000002 [100459.355164] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 tg3 amd64_edac_mod k8temp i2c_piix4 edac_core edac_mce_amd shpchp serio_raw btrfs zlib_deflate libcrc32c raid1 pata_acpi ata_generic mptsas mptscsih mptbase scsi_transport_sas pata_serverworks sata_svw radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: mperf] [100459.573210] Pid: 24595, comm: init Tainted: G D 2.6.35.11-83.fc14.x86_64 #1 [100459.620108] Call Trace: [100459.635291] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 [100459.668669] [<ffffffff8146845e>] schedule+0xd9/0x5cb [100459.699450] [<ffffffff81049246>] __cond_resched+0x2a/0x35 [100459.732826] [<ffffffff81468a11>] _cond_resched+0x1b/0x22 [100459.765687] [<ffffffff81469639>] down_read+0x29/0x3b [100459.796464] [<ffffffff81050e80>] exit_mm+0x3c/0x127 [100459.826725] [<ffffffff810511ba>] do_exit+0x24f/0x74f [100459.857505] [<ffffffff81469fcf>] ? ~. [terminated ipmitool] Version-Release number of selected component (if applicable): 2.6.35.11-83.fc14.x86_64 How reproducible: So far unknown.
(In reply to comment #1) > [100451.853379] Oops: 0002 [#11] SMP ^^^ This is the 11th oops the machine has encountered since it was booted. What did the other oops messages look like?
Hi Chuck, some more info below. But I should mention I've seen many, many, ECC errors coming up [255413.887870] ECC/ChipKill ECC error. [255413.909287] EDAC amd64 MC1: CE ERROR_ADDRESS= 0x29a98f4f0 [255413.942145] EDAC amd64 MC1: Failed to translate InputAddr to csrow for address 0x29a98f4f0 [255413.992149] EDAC MC1: CE - no information available: amd64_edac [255419.036310] Northbridge Error, node 1, core: 0 [255419.064142] ECC/ChipKill ECC error. [255419.085562] EDAC amd64 MC1: CE ERROR_ADDRESS= 0x292b8f4f0 [255419.118425] EDAC amd64 MC1: Failed to translate InputAddr to csrow for address 0x292b8f4f0 [255419.168433] EDAC MC1: CE - no information available: amd64_edac [255420.206042] Northbridge Error, node 1, core: 1 [255420.233892] ECC/ChipKill ECC error. [255420.255315] EDAC amd64 MC1: CE ERROR_ADDRESS= 0x2a118f4f8 [255420.288178] EDAC amd64 MC1: Failed to translate InputAddr to csrow for address 0x2a118f4f8 [255420.338184] EDAC MC1: CE - no information available: amd64_edac I'd like to figure out which DIMM is the problem but am stumped at the "Failed to translate" which leaves me in the dark a bit. # grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count /sys/devices/system/edac/mc/mc0/csrow4/ch0_ce_count:0 /sys/devices/system/edac/mc/mc0/csrow4/ch1_ce_count:0 /sys/devices/system/edac/mc/mc1/csrow2/ch0_ce_count:0 /sys/devices/system/edac/mc/mc1/csrow2/ch1_ce_count:0 /sys/devices/system/edac/mc/mc1/csrow3/ch0_ce_count:0 /sys/devices/system/edac/mc/mc1/csrow3/ch1_ce_count:0 /sys/devices/system/edac/mc/mc1/csrow4/ch0_ce_count:0 /sys/devices/system/edac/mc/mc1/csrow4/ch1_ce_count:0 [root.prod ~]# zgrep -i Oops /var/log/messages*gz /var/log/messages-20110306.gz:Mar 2 14:56:33 pdbsearch11 kernel: [84164.552150] Oops: 0002 [#1] SMP /var/log/messages-20110306.gz:Mar 2 14:56:33 pdbsearch11 kernel: [84167.120384] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 14:57:38 pdbsearch11 kernel: [84230.093109] Oops: 0002 [#2] SMP /var/log/messages-20110306.gz:Mar 2 14:57:39 pdbsearch11 kernel: [84232.580624] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 14:59:31 pdbsearch11 kernel: [84342.718259] Oops: 0002 [#3] SMP /var/log/messages-20110306.gz:Mar 2 14:59:31 pdbsearch11 kernel: [84345.349656] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.416134] Oops: 0002 [#4] SMP /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.462131] Oops: 0002 [#5] SMP /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84377.032740] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84378.492327] Oops: 0002 [#6] SMP /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.416134] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:00:10 pdbsearch11 kernel: [84383.558089] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:01:03 pdbsearch11 kernel: [84435.091327] Oops: 0002 [#7] SMP /var/log/messages-20110306.gz:Mar 2 15:01:03 pdbsearch11 kernel: [84437.345607] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:03 pdbsearch11 kernel: [84674.879419] Oops: 0002 [#8] SMP /var/log/messages-20110306.gz:Mar 2 15:05:03 pdbsearch11 kernel: [84677.133804] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84678.274255] Oops: 0002 [#9] SMP /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84678.511385] Oops: 0002 [#10] SMP /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84678.533136] Oops: 0002 [#11] SMP /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84679.933911] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84679.934021] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84679.934372] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84679.936170] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:15 pdbsearch11 kernel: [84688.601699] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 15:05:16 pdbsearch11 kernel: [84689.805922] [<ffffffff8146b22e>] oops_end+0xbf/0xc7 /var/log/messages-20110306.gz:Mar 2 14:56:33 pdbsearch11 kernel: [84164.527752] BUG: unable to handle kernel paging request at ffff8801d002c000 /var/log/messages-20110306.gz:Mar 2 14:56:33 pdbsearch11 kernel: [84166.547148] BUG: scheduling while atomic: snmpd/1382/0x10000001 /var/log/messages-20110306.gz:Mar 2 14:56:33 pdbsearch11 kernel: [84166.861164] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 14:57:38 pdbsearch11 kernel: [84230.092206] BUG: unable to handle kernel paging request at ffff8801d002d000 /var/log/messages-20110306.gz:Mar 2 14:57:38 pdbsearch11 kernel: [84231.969912] BUG: scheduling while atomic: glusterfsd/1445/0x10000001 /var/log/messages-20110306.gz:Mar 2 14:57:38 pdbsearch11 kernel: [84232.289088] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 14:59:31 pdbsearch11 kernel: [84342.717329] BUG: unable to handle kernel paging request at ffff8801d002e000 /var/log/messages-20110306.gz:Mar 2 14:59:31 pdbsearch11 kernel: [84344.777429] BUG: scheduling while atomic: ntpd/1406/0x10000001 /var/log/messages-20110306.gz:Mar 2 14:59:31 pdbsearch11 kernel: [84345.090420] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.410627] BUG: unable to handle kernel paging request at ffff8801d002f000 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.462131] BUG: unable to handle kernel paging request at ffff8801d0041000 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84377.032686] BUG: scheduling while atomic: munin-update/1698/0x10000001 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84377.032706] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84378.492306] BUG: unable to handle kernel paging request at ffff8801d0042000 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.416134] BUG: scheduling while atomic: crond/1693/0x10000002 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84375.416134] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84382.982250] BUG: scheduling while atomic: crond/1693/0x10000002 /var/log/messages-20110306.gz:Mar 2 15:00:09 pdbsearch11 kernel: [84383.296261] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 15:01:03 pdbsearch11 kernel: [84435.090381] BUG: unable to handle kernel paging request at ffff8801d0030000 /var/log/messages-20110306.gz:Mar 2 15:01:03 pdbsearch11 kernel: [84436.772360] BUG: scheduling while atomic: crond/1703/0x10000002 /var/log/messages-20110306.gz:Mar 2 15:01:03 pdbsearch11 kernel: [84437.086367] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 15:05:03 pdbsearch11 kernel: [84674.878472] BUG: unable to handle kernel paging request at ffff8801d0031000 /var/log/messages-20110306.gz:Mar 2 15:05:03 pdbsearch11 kernel: [84676.560558] BUG: scheduling while atomic: crond/1704/0x10000002 /var/log/messages-20110306.gz:Mar 2 15:05:03 pdbsearch11 kernel: [84676.874558] [<ffffffff8103ffbe>] __schedule_bug+0x5f/0x64 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84678.274136] BUG: unable to handle kernel paging request at ffff8801d0032000 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84678.511385] BUG: unable to handle kernel paging request at ffff8801d0043000 /var/log/messages-20110306.gz:Mar 2 15:05:14 pdbsearch11 kernel: [84678.533136] BUG: unable to handle kernel paging request at ffff8801d0044000
This appears to be hardware related. There isn't much we can do about this.