Description of problem: Kernel panic - not syncing: Fatal exception in interrupt Version-Release number of selected component (if applicable): 2.6.21-14.el5rtvanilla How reproducible: Unknown Steps to Reproduce: 1. Using RHEL5 GA base distro, Adding the 2.6.21-14.el5rtvanilla. While running the /kernel/stress/racer the Panic occured Actual results: BUG: unable to handle kernel paging request at virtual address 3445516f printing eip: ee81f00f *pde = 00000000 Oops: 0002 [#1] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq video sbs i2c_ec dock button battery asus_acpi ac lp floppy sg ata_generic snd_hda_intel snd_hda_codec ata_piix snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm iTCO_wdt iTCO_vendor_support snd_timer snd soundcore tg3 snd_page_alloc pcspkr ide_cd i2c_i801 i2c_core parport_pc cdrom rtc_cmos parport serio_raw rtc_core rtc_lib dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<ee81f00f>] Not tainted VLI EFLAGS: 00010282 (2.6.21-14.el5rtvanilla #1) EIP is at 0xee81f00f eax: ee7f85b0 ebx: ee7f85c0 ecx: 0000000c edx: ee7f85c0 esi: c049236e edi: c0797da7 ebp: c0490c44 esp: c0797da4 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 Process swapper (pid: 0, ti=c0797000 task=c070c460 task.ti=c0750000) Stack: c0797da8 c0797dd0 c04926bf c0797db8 c0459fb0 c0797dd0 00000246 00000000 f7edbc80 0000019b d0ed47fc c0797e0c f88fb44c c045a01b 00000246 f7df9cc0 00000000 e18b04e8 c0797df4 00000000 00000001 0000111e 004f11a2 00000000 Call Trace: [<c0405e9e>] show_trace_log_lvl+0x1a/0x2f [<c0405f4e>] show_stack_log_lvl+0x9b/0xa3 [<c0406134>] show_registers+0x1de/0x2f6 [<c0406365>] die+0x119/0x22e [<c062b64b>] do_page_fault+0x435/0x506 [<c0629dd4>] error_code+0x7c/0x84 ======================= Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d8 f5 57 08 43 d6 01 00 18 48 13 08 78 2e 4f <08> 44 d6 01 00 0c 48 13 08 90 2e 4f 08 45 d6 01 00 00 48 13 08 EIP: [<ee81f00f>] 0xee81f00f SS:ESP 0068:c0797da4 Kernel panic - not syncing: Fatal exception in interrupt Additional info:
Assigning to Clark to incorporate patch.
oops, wrong bug.. no patch here ;-)
what is this 'racer' tool?
You can download a copy from here. It is a directory test. ftp://ftp.lustre.org/pub/benchmarks/racer-lustre.tar.gz
Ingo, Hold off on looking at this for right now. We have run into an issue this week with a system known as ibm-taroko. It may be correlated back to a hardware failure. Here is the failure from this weeks kernel: 2.6.21-20.el5rt mm/memory.c:117: bad pgd 2b12a001. stopped custom tracer. ------------[ cut here ]------------ kernel BUG at mm/mmap.c:2005! invalid opcode: 0000 [#1] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<c046a6b7>] Not tainted VLI EFLAGS: 00010202 (2.6.21-20.el5rt #1) EIP is at exit_mmap+0x104/0x110 eax: efb264c0 ebx: c1e28a00 ecx: c1e2a880 edx: f0b36630 esi: 00000000 edi: 00000000 ebp: f12dae18 esp: f12dadf4 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 preempt:00000001 Process file_link.sh (pid: 4735, ti=f12da000 task=f0b36630 task.ti=f12da000) Stack: 00000000 f12dae04 00000000 efb264c0 0000004f c1e28a00 efb264c0 efb26534 f0fa4a40 f12dae28 c0425f36 efb264f4 f0ed7a84 f12dae84 c047ddad 00000000 00000001 f7aaf538 f6f10c00 00000000 effda840 f0b36630 f0ed7580 00000000 Call Trace: [<c0425f36>] mmput+0x23/0x7a [<c047ddad>] flush_old_exec+0x569/0x7bb [<c04a1494>] load_elf_binary+0x4af/0x144d [<c047d1b2>] search_binary_handler+0xd7/0x220 [<c047e8a7>] do_execve+0x154/0x1e5 [<c0403541>] sys_execve+0x2e/0x79 [<c0404fc0>] syscall_call+0x7/0xb [<ffffe410>] pg0+0x3f71f410/0xfffff798 ======================= Code: 60 66 80 c0 03 04 95 80 4a 7c c0 e8 de 5b 1c 00 eb 09 89 f8 e8 d4 fe ff ff 89 c7 85 ff 75 f3 8b 45 e8 83 b8 a8 00 00 00 00 74 04 <0f> 0b eb fe 83 c4 18 5b 5e 5f 5d c3 55 89 e5 57 89 d7 56 89 c6 EIP: [<c046a6b7>] exit_mmap+0x104/0x110 SS:ESP 0068:f12dadf4 /usr/bin/rhts-test-runner.sh: line 92: [: =: unary operator expected security_compute_av: unrecognized class 23556 security_compute_av: unrecognized class 23556 security_compute_av: unrecognized class 23556 security_compute_av: unrecognized class 23556 BUG: unable to handle kernel paging request<1>BUG: unable to handle kernel paging request at virtual address 04da000f printing eip: c04f0d5c *pde = 00000000 Oops: 0000 [#2] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<c04f0d5c>] Not tainted VLI EFLAGS: 00010297 (2.6.21-20.el5rt #1) EIP is at vsnprintf+0x295/0x460 eax: 04da000f ebx: ffffffff ecx: 04da000f edx: fffffffe esi: e97648d2 edi: f0a98c70 ebp: f0a98c30 esp: f0a98be8 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068 preempt:00000001 Process ln (pid: 4861, ti=f0a98000 task=efedc8b0 task.ti=f0a98000) Stack: f0a98c14 c0444213 efe90d97 f0a98c00 c044422f 00000246 00000336 e97648ca 007c1980 00000202 e9764c00 ffffffff ffffffff 00000000 c06d6820 f0a98c70 f63412c0 00000336 f0a98c54 c0452506 f0a98c70 c06d6817 f0f07560 f0a98c70 Call Trace: [<c0452506>] audit_log_vformat+0x93/0x147 [<c04525d5>] audit_log_format+0x1b/0x1d [<c04ce2b5>] avc_audit+0xc25/0xc38 [<c04ce316>] avc_has_perm+0x4e/0x58 [<c04cf737>] inode_has_perm+0x64/0x6c [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7 [<c04803ae>] permission+0xcc/0xe0 [<c04812f1>] vfs_permission+0xf/0x11 [<c048186d>] __link_path_walk+0x143/0xc7a [<c04823ef>] link_path_walk+0x4b/0xc0 [<c0482719>] do_path_lookup+0x191/0x1e2 [<c0482f06>] __user_walk_fd+0x32/0x44 [<c047cb54>] vfs_stat_fd+0x1b/0x41 [<c047cbfc>] vfs_stat+0x11/0x13 [<c047cc12>] sys_stat64+0x14/0x28 [<c0404fc0>] syscall_call+0x7/0xb [<ffffe410>] pg0+0x3f71f410/0xfffff798 ======================= Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c 6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83 fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:f0a98be8 BUG: unable to handle kernel paging request at virtual address 04da000f printing eip: c04f0d5c *pde = 00000000 Oops: 0000 [#3] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<c04f0d5c>] Not tainted VLI EFLAGS: 00010297 (2.6.21-20.el5rt #1) EIP is at vsnprintf+0x295/0x460 eax: 04da000f ebx: ffffffff ecx: 04da000f edx: fffffffe esi: e97640d5 edi: efb51c70 ebp: efb51c30 esp: efb51be8 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068 preempt:00000001 Process mkdir (pid: 4859, ti=efb51000 task=f13f97b0 task.ti=efb51000) Stack: efb51c14 c0444213 f0f07897 efb51c00 c044422f 00000246 00000333 e97640cd 007c1980 00000202 e9764400 ffffffff ffffffff 00000000 c06d6820 efb51c70 f62d6a80 00000333 efb51c54 c0452506 efb51c70 c06d6817 f0786aa0 efb51c70 Call Trace: [<c0452506>] audit_log_vformat+0x93/0x147 [<c04525d5>] audit_log_format+0x1b/0x1d [<c04ce2b5>] avc_audit+0xc25/0xc38 [<c04ce316>] avc_has_perm+0x4e/0x58 [<c04cf737>] inode_has_perm+0x64/0x6c [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7 [<c04803ae>] permission+0xcc/0xe0 [<c04812f1>] vfs_permission+0xf/0x11 [<c048186d>] __link_path_walk+0x143/0xc7a [<c04823ef>] link_path_walk+0x4b/0xc0 [<c0482719>] do_path_lookup+0x191/0x1e2 [<c0482f06>] __user_walk_fd+0x32/0x44 [<c047cb54>] vfs_stat_fd+0x1b/0x41 [<c047cbfc>] vfs_stat+0x11/0x13 [<c047cc12>] sys_stat64+0x14/0x28 [<c0404fc0>] syscall_call+0x7/0xb [<ffffe410>] pg0+0x3f71f410/0xfffff798 ======================= Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c 6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83 fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:efb51be8 BUG: unable to handle kernel paging request at virtual address 04da000f printing eip: c04f0d5c *pde = 00000000 Oops: 0000 [#4] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<c04f0d5c>] Not tainted VLI EFLAGS: 00010297 (2.6.21-20.el5rt #1) EIP is at vsnprintf+0x295/0x460 eax: 04da000f ebx: ffffffff ecx: 04da000f edx: fffffffe esi: f63820d2 edi: efb4dc70 ebp: efb4dc30 esp: efb4dbe8 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068 preempt:00000001 Process ln (pid: 4862, ti=efb4d000 task=efa82db0 task.ti=efb4d000) Stack: efb4dc14 c0444213 f06df857 efb4dc00 c044422f 00000246 00000336 f63820ca 007c1980 00000202 f6382400 ffffffff ffffffff 00000000 c06d6820 efb4dc70 f168f180 00000336 efb4dc54 c0452506 efb4dc70 c06d6817 f165fec0 efb4dc70 Call Trace: [<c0452506>] audit_log_vformat+0x93/0x147 [<c04525d5>] audit_log_format+0x1b/0x1d [<c04ce2b5>] avc_audit+0xc25/0xc38 [<c04ce316>] avc_has_perm+0x4e/0x58 [<c04cf737>] inode_has_perm+0x64/0x6c [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7 [<c04803ae>] permission+0xcc/0xe0 [<c04812f1>] vfs_permission+0xf/0x11 [<c048186d>] __link_path_walk+0x143/0xc7a [<c04823ef>] link_path_walk+0x4b/0xc0 [<c0482719>] do_path_lookup+0x191/0x1e2 [<c0482f06>] __user_walk_fd+0x32/0x44 [<c047cb54>] vfs_stat_fd+0x1b/0x41 [<c047cbfc>] vfs_stat+0x11/0x13 [<c047cc12>] sys_stat64+0x14/0x28 [<c0404fc0>] syscall_call+0x7/0xb [<ffffe410>] pg0+0x3f71f410/0xfffff798 ======================= Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c 6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83 fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:efb4dbe8 swap_free: Unused swap offset entry 00000ff0 Eeek! page_mapcount(page) went negative! (-1) page pfn = 0 page->flags = 400 page->count = 1 page->mapping = 00000000 vma->vm_ops = generic_file_vm_ops+0x0/0x18 vma->vm_ops->nopage = filemap_nopage+0x0/0x329 vma->vm_file->f_op->mmap = generic_file_mmap+0x0/0x3b ------------[ cut here ]------------ kernel BUG at mm/rmap.c:596! invalid opcode: 0000 [#5] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<c046d080>] Not tainted VLI EFLAGS: 00010282 (2.6.21-20.el5rt #1) EIP is at page_remove_rmap+0xd7/0xf3 eax: 0000003b ebx: c1000000 ecx: 00000046 edx: 00000000 esi: eff9ce9c edi: ef7f6908 ebp: efb51a40 esp: efb51a30 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 preempt:00000001 Process mkdir (pid: 4859, ti=efb51000 task=f13f97b0 task.ti=efb51000) Stack: c06cf057 00000000 c1000000 00000000 efb51a98 c04679cc 00000000 eff9ce9c efb51ab4 00000000 00000001 403c4000 f557d400 f63b6b80 c1e28a00 00000000 fffffff6 c1a63ddc f557d400 40242000 00363e39 403c4000 00000000 efb51ab4 Call Trace: [<c04679cc>] unmap_vmas+0x25c/0x4e9 [<c046a63e>] exit_mmap+0x8b/0x110 [<c0425f36>] mmput+0x23/0x7a [<c0429a22>] exit_mm+0xef/0xf4 [<c042adb2>] do_exit+0x1e2/0x75a [<c0406669>] die+0x211/0x237 [<c06323b3>] do_page_fault+0x451/0x522 [<c06309fc>] error_code+0x7c/0x84 [<c04f0d5c>] vsnprintf+0x295/0x460 [<c0452506>] audit_log_vformat+0x93/0x147 [<c04525d5>] audit_log_format+0x1b/0x1d [<c04ce2b5>] avc_audit+0xc25/0xc38 [<c04ce316>] avc_has_perm+0x4e/0x58 [<c04cf737>] inode_has_perm+0x64/0x6c [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7 [<c04803ae>] permission+0xcc/0xe0 [<c04812f1>] vfs_permission+0xf/0x11 [<c048186d>] __link_path_walk+0x143/0xc7a [<c04823ef>] link_path_walk+0x4b/0xc0 [<c0482719>] do_path_lookup+0x191/0x1e2 [<c0482f06>] __user_walk_fd+0x32/0x44 [<c047cb54>] vfs_stat_fd+0x1b/0x41 [<c047cbfc>] vfs_stat+0x11/0x13 [<c047cc12>] sys_stat64+0x14/0x28 [<c0404fc0>] syscall_call+0x7/0xb [<ffffe410>] pg0+0x3f71f410/0xfffff798 ======================= Code: c0 74 0d 8b 50 08 b8 87 f0 6c c0 e8 06 bc fd ff 8b 46 48 85 c0 74 14 8b 40 30 85 c0 74 0d 8b 50 2c b8 a6 f0 6c c0 e8 eb bb fd ff <0f> 0b eb fe 8b 53 10 89 d8 83 e2 01 f7 da 83 c2 04 e8 d3 8a ff EIP: [<c046d080>] page_remove_rmap+0xd7/0xf3 SS:ESP 0068:efb51a30 Fixing recursive fault but reboot is needed! BUG: unable to handle kernel NULL pointer dereference at virtual address 00000020 printing eip: f89055f3 *pde = 00000000 Oops: 0000 [#6] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 0 EIP: 0060:[<f89055f3>] Not tainted VLI EFLAGS: 00010246 (2.6.21-20.el5rt #1) EIP is at clone_endio+0x2e/0xcb [dm_mod] eax: 00000000 ebx: 00000001 ecx: f8800080 edx: ef6353a8 esi: 00000000 edi: e9791b40 ebp: f7c27da4 esp: f7c27d80 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 preempt:00000001 Process IRQ-17 (pid: 439, ti=f7c27000 task=f7c56e70 task.ti=f7c27000) Stack: 14000000 00000050 f8800080 ef6353a8 00000001 00000000 e9791b40 00001000 f89055c5 f7c27dcc c0498d07 f796f800 f7c27db8 f88da3e1 00000000 00000000 e9791b40 00001000 00000000 f7c27e30 c04e27b4 00000000 c0808500 f58450c0 Call Trace: [<c0498d07>] bio_endio+0x5f/0x67 [<c04e27b4>] __end_that_request_first+0x220/0x484 [<c04e2a20>] end_that_request_chunk+0x8/0xa [<f888d8e8>] scsi_end_request+0x20/0xaa [scsi_mod] [<f888db26>] scsi_io_completion+0x168/0x33f [scsi_mod] [<f88645dc>] sd_rw_intr+0x227/0x250 [sd_mod] [<f888936b>] scsi_finish_command+0x87/0x8f [scsi_mod] [<f888e2fb>] scsi_softirq_done+0xd0/0xd8 [scsi_mod] [<c04e4860>] blk_done_softirq+0x51/0x60 [<c042cba6>] ___do_softirq+0xf3/0x1fa [<c042cfbc>] do_softirq_from_hardirq+0x4b/0x65 [<c0458f3c>] do_irqd+0x202/0x24e [<c0439bcf>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= Code: 57 89 c7 56 89 ce 53 bb 01 00 00 00 83 ec 18 8b 40 3c 83 7f 20 00 89 45 e8 89 c2 8b 00 8b 00 89 45 ec 8b 4a 04 89 4d e4 8b 41 04 <8b> 40 20 89 45 f0 0f 85 87 00 00 00 f6 47 10 01 75 0a 85 f6 b8 EIP: [<f89055f3>] clone_endio+0x2e/0xcb [dm_mod] SS:ESP 0068:f7c27d80 at virtual address 04da000f printing eip: c04f0d5c *pde = 00000000 Oops: 0000 [#7] PREEMPT SMP Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801 iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd CPU: 1 EIP: 0060:[<c04f0d5c>] Not tainted VLI EFLAGS: 00010297 (2.6.21-20.el5rt #1) EIP is at vsnprintf+0x295/0x460 eax: 04da000f ebx: ffffffff ecx: 04da000f edx: fffffffe esi: f63420d2 edi: ef77cc70 ebp: ef77cc30 esp: ef77cbe8 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068 preempt:00000001 Process mv (pid: 4856, ti=ef77c000 task=f0b16070 task.ti=ef77c000) Stack: ef77cc14 c0444213 f0f07697 ef77cc00 c044422f 00000246 00000336 f63420ca 007c1980 00000202 f6342400 ffffffff ffffffff 00000000 c06d6820 ef77cc70 f168f0c0 00000336 ef77cc54 c0452506 ef77cc70 c06d6817 f7ca89a0 ef77cc70 Call Trace: [<c0452506>] audit_log_vformat+0x93/0x147 [<c04525d5>] audit_log_format+0x1b/0x1d [<c04ce2b5>] avc_audit+0xc25/0xc38 [<c04ce316>] avc_has_perm+0x4e/0x58 [<c04cf737>] inode_has_perm+0x64/0x6c [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7 [<c04803ae>] permission+0xcc/0xe0 [<c04812f1>] vfs_permission+0xf/0x11 [<c048186d>] __link_path_walk+0x143/0xc7a [<c04823ef>] link_path_walk+0x4b/0xc0 [<c0482719>] do_path_lookup+0x191/0x1e2 [<c0482f06>] __user_walk_fd+0x32/0x44 [<c047cb54>] vfs_stat_fd+0x1b/0x41 [<c047cbfc>] vfs_stat+0x11/0x13 [<c047cc12>] sys_stat64+0x14/0x28 [<c0404fc0>] syscall_call+0x7/0xb [<ffffe410>] pg0+0x3f71f410/0xfffff798 ======================= Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c 6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83 fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:ef77cbe8
Konrad is supposed to be buying new DIMMS.
The engineering services ticket was just resolved. But it doesn't say what the resolution was/is. Let me know if teh system can be used for testing, I willschedule an RT-kernel test. https://engineering.redhat.com/rt3/Ticket/Display.html?id=12358
Closing this one out. Reopen if problem reoccurs with fixed memory.