Bug 241283 - [RHEL5 RT Vanilla] BUG: unable to handle kernel paging request at virtual address 3445516f
[RHEL5 RT Vanilla] BUG: unable to handle kernel paging request at virtual add...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel (Show other bugs)
1.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Red Hat Real Time Maintenance
http://rhts.lab.boston.redhat.com/cgi...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-05-24 16:13 EDT by Jeff Burke
Modified: 2008-02-27 14:58 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-07-26 09:09:42 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jeff Burke 2007-05-24 16:13:11 EDT
Description of problem:
 Kernel panic - not syncing: Fatal exception in interrupt

Version-Release number of selected component (if applicable):
 2.6.21-14.el5rtvanilla

How reproducible:
 Unknown

Steps to Reproduce:
1. Using RHEL5 GA base distro, Adding the 2.6.21-14.el5rtvanilla. While running
the /kernel/stress/racer the Panic occured
  
Actual results:

BUG: unable to handle kernel paging request at virtual address 3445516f
 printing eip:
ee81f00f
*pde = 00000000
Oops: 0002 [#1]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
video sbs i2c_ec dock button battery asus_acpi ac lp floppy sg ata_generic
snd_hda_intel snd_hda_codec ata_piix snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_wdt iTCO_vendor_support snd_timer snd soundcore tg3 snd_page_alloc pcspkr
ide_cd i2c_i801 i2c_core parport_pc cdrom rtc_cmos parport serio_raw rtc_core
rtc_lib dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<ee81f00f>]    Not tainted VLI
EFLAGS: 00010282   (2.6.21-14.el5rtvanilla #1)
EIP is at 0xee81f00f
eax: ee7f85b0   ebx: ee7f85c0   ecx: 0000000c   edx: ee7f85c0
esi: c049236e   edi: c0797da7   ebp: c0490c44   esp: c0797da4
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process swapper (pid: 0, ti=c0797000 task=c070c460 task.ti=c0750000)
Stack: c0797da8 c0797dd0 c04926bf c0797db8 c0459fb0 c0797dd0 00000246 00000000 
       f7edbc80 0000019b d0ed47fc c0797e0c f88fb44c c045a01b 00000246 f7df9cc0 
       00000000 e18b04e8 c0797df4 00000000 00000001 0000111e 004f11a2 00000000 
Call Trace:
 [<c0405e9e>] show_trace_log_lvl+0x1a/0x2f
 [<c0405f4e>] show_stack_log_lvl+0x9b/0xa3
 [<c0406134>] show_registers+0x1de/0x2f6
 [<c0406365>] die+0x119/0x22e
 [<c062b64b>] do_page_fault+0x435/0x506
 [<c0629dd4>] error_code+0x7c/0x84
 =======================
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 d8 f5 57 08 43 d6 01 00 18 48 13 08 78 2e 4f <08> 44 d6 01 00 0c 48 13
08 90 2e 4f 08 45 d6 01 00 00 48 13 08 
EIP: [<ee81f00f>] 0xee81f00f SS:ESP 0068:c0797da4
Kernel panic - not syncing: Fatal exception in interrupt

Additional info:
Comment 1 Tim Burke 2007-05-25 08:23:24 EDT
Assigning to Clark to incorporate patch.
Comment 2 Tim Burke 2007-05-25 08:25:55 EDT
oops, wrong bug.. no patch here ;-)
Comment 3 Ingo Molnar 2007-06-01 08:54:24 EDT
what is this 'racer' tool?
Comment 4 Jeff Burke 2007-06-01 09:52:37 EDT
You can download a copy from here. It is a directory test.
ftp://ftp.lustre.org/pub/benchmarks/racer-lustre.tar.gz

Comment 5 Jeff Burke 2007-06-01 13:38:14 EDT
Ingo,
     Hold off on looking at this for right now. We have run into an issue this
week with a system known as ibm-taroko. It may be correlated back to a hardware
failure.

Here is the failure from this weeks kernel: 2.6.21-20.el5rt
mm/memory.c:117: bad pgd 2b12a001.
stopped custom tracer.
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2005!
invalid opcode: 0000 [#1]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<c046a6b7>]    Not tainted VLI
EFLAGS: 00010202   (2.6.21-20.el5rt #1)
EIP is at exit_mmap+0x104/0x110
eax: efb264c0   ebx: c1e28a00   ecx: c1e2a880   edx: f0b36630
esi: 00000000   edi: 00000000   ebp: f12dae18   esp: f12dadf4
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068  preempt:00000001
Process file_link.sh (pid: 4735, ti=f12da000 task=f0b36630 task.ti=f12da000)
Stack: 00000000 f12dae04 00000000 efb264c0 0000004f c1e28a00 efb264c0 efb26534 
       f0fa4a40 f12dae28 c0425f36 efb264f4 f0ed7a84 f12dae84 c047ddad 00000000 
       00000001 f7aaf538 f6f10c00 00000000 effda840 f0b36630 f0ed7580 00000000 
Call Trace:
 [<c0425f36>] mmput+0x23/0x7a
 [<c047ddad>] flush_old_exec+0x569/0x7bb
 [<c04a1494>] load_elf_binary+0x4af/0x144d
 [<c047d1b2>] search_binary_handler+0xd7/0x220
 [<c047e8a7>] do_execve+0x154/0x1e5
 [<c0403541>] sys_execve+0x2e/0x79
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff798
 =======================
Code: 60 66 80 c0 03 04 95 80 4a 7c c0 e8 de 5b 1c 00 eb 09 89 f8 e8 d4 fe ff ff
89 c7 85 ff 75 f3 8b 45 e8 83 b8 a8 00 00 00 00 74 04 <0f> 0b eb fe 83 c4 18 5b
5e 5f 5d c3 55 89 e5 57 89 d7 56 89 c6 
EIP: [<c046a6b7>] exit_mmap+0x104/0x110 SS:ESP 0068:f12dadf4
/usr/bin/rhts-test-runner.sh: line 92: [: =: unary operator expected
security_compute_av:  unrecognized class 23556
security_compute_av:  unrecognized class 23556
security_compute_av:  unrecognized class 23556
security_compute_av:  unrecognized class 23556
BUG: unable to handle kernel paging request<1>BUG: unable to handle kernel
paging request at virtual address 04da000f
 printing eip:
c04f0d5c
*pde = 00000000
Oops: 0000 [#2]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<c04f0d5c>]    Not tainted VLI
EFLAGS: 00010297   (2.6.21-20.el5rt #1)
EIP is at vsnprintf+0x295/0x460
eax: 04da000f   ebx: ffffffff   ecx: 04da000f   edx: fffffffe
esi: e97648d2   edi: f0a98c70   ebp: f0a98c30   esp: f0a98be8
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000001
Process ln (pid: 4861, ti=f0a98000 task=efedc8b0 task.ti=f0a98000)
Stack: f0a98c14 c0444213 efe90d97 f0a98c00 c044422f 00000246 00000336 e97648ca 
       007c1980 00000202 e9764c00 ffffffff ffffffff 00000000 c06d6820 f0a98c70 
       f63412c0 00000336 f0a98c54 c0452506 f0a98c70 c06d6817 f0f07560 f0a98c70 
Call Trace:
 [<c0452506>] audit_log_vformat+0x93/0x147
 [<c04525d5>] audit_log_format+0x1b/0x1d
 [<c04ce2b5>] avc_audit+0xc25/0xc38
 [<c04ce316>] avc_has_perm+0x4e/0x58
 [<c04cf737>] inode_has_perm+0x64/0x6c
 [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7
 [<c04803ae>] permission+0xcc/0xe0
 [<c04812f1>] vfs_permission+0xf/0x11
 [<c048186d>] __link_path_walk+0x143/0xc7a
 [<c04823ef>] link_path_walk+0x4b/0xc0
 [<c0482719>] do_path_lookup+0x191/0x1e2
 [<c0482f06>] __user_walk_fd+0x32/0x44
 [<c047cb54>] vfs_stat_fd+0x1b/0x41
 [<c047cbfc>] vfs_stat+0x11/0x13
 [<c047cc12>] sys_stat64+0x14/0x28
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff798
 =======================
Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c
6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83
fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b 
EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:f0a98be8
BUG: unable to handle kernel paging request at virtual address 04da000f
 printing eip:
c04f0d5c
*pde = 00000000
Oops: 0000 [#3]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<c04f0d5c>]    Not tainted VLI
EFLAGS: 00010297   (2.6.21-20.el5rt #1)
EIP is at vsnprintf+0x295/0x460
eax: 04da000f   ebx: ffffffff   ecx: 04da000f   edx: fffffffe
esi: e97640d5   edi: efb51c70   ebp: efb51c30   esp: efb51be8
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000001
Process mkdir (pid: 4859, ti=efb51000 task=f13f97b0 task.ti=efb51000)
Stack: efb51c14 c0444213 f0f07897 efb51c00 c044422f 00000246 00000333 e97640cd 
       007c1980 00000202 e9764400 ffffffff ffffffff 00000000 c06d6820 efb51c70 
       f62d6a80 00000333 efb51c54 c0452506 efb51c70 c06d6817 f0786aa0 efb51c70 
Call Trace:
 [<c0452506>] audit_log_vformat+0x93/0x147
 [<c04525d5>] audit_log_format+0x1b/0x1d
 [<c04ce2b5>] avc_audit+0xc25/0xc38
 [<c04ce316>] avc_has_perm+0x4e/0x58
 [<c04cf737>] inode_has_perm+0x64/0x6c
 [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7
 [<c04803ae>] permission+0xcc/0xe0
 [<c04812f1>] vfs_permission+0xf/0x11
 [<c048186d>] __link_path_walk+0x143/0xc7a
 [<c04823ef>] link_path_walk+0x4b/0xc0
 [<c0482719>] do_path_lookup+0x191/0x1e2
 [<c0482f06>] __user_walk_fd+0x32/0x44
 [<c047cb54>] vfs_stat_fd+0x1b/0x41
 [<c047cbfc>] vfs_stat+0x11/0x13
 [<c047cc12>] sys_stat64+0x14/0x28
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff798
 =======================
Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c
6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83
fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b 
EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:efb51be8
BUG: unable to handle kernel paging request at virtual address 04da000f
 printing eip:
c04f0d5c
*pde = 00000000
Oops: 0000 [#4]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<c04f0d5c>]    Not tainted VLI
EFLAGS: 00010297   (2.6.21-20.el5rt #1)
EIP is at vsnprintf+0x295/0x460
eax: 04da000f   ebx: ffffffff   ecx: 04da000f   edx: fffffffe
esi: f63820d2   edi: efb4dc70   ebp: efb4dc30   esp: efb4dbe8
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000001
Process ln (pid: 4862, ti=efb4d000 task=efa82db0 task.ti=efb4d000)
Stack: efb4dc14 c0444213 f06df857 efb4dc00 c044422f 00000246 00000336 f63820ca 
       007c1980 00000202 f6382400 ffffffff ffffffff 00000000 c06d6820 efb4dc70 
       f168f180 00000336 efb4dc54 c0452506 efb4dc70 c06d6817 f165fec0 efb4dc70 
Call Trace:
 [<c0452506>] audit_log_vformat+0x93/0x147
 [<c04525d5>] audit_log_format+0x1b/0x1d
 [<c04ce2b5>] avc_audit+0xc25/0xc38
 [<c04ce316>] avc_has_perm+0x4e/0x58
 [<c04cf737>] inode_has_perm+0x64/0x6c
 [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7
 [<c04803ae>] permission+0xcc/0xe0
 [<c04812f1>] vfs_permission+0xf/0x11
 [<c048186d>] __link_path_walk+0x143/0xc7a
 [<c04823ef>] link_path_walk+0x4b/0xc0
 [<c0482719>] do_path_lookup+0x191/0x1e2
 [<c0482f06>] __user_walk_fd+0x32/0x44
 [<c047cb54>] vfs_stat_fd+0x1b/0x41
 [<c047cbfc>] vfs_stat+0x11/0x13
 [<c047cc12>] sys_stat64+0x14/0x28
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff798
 =======================
Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c
6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83
fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b 
EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:efb4dbe8
swap_free: Unused swap offset entry 00000ff0
Eeek! page_mapcount(page) went negative! (-1)
  page pfn = 0
  page->flags = 400
  page->count = 1
  page->mapping = 00000000
  vma->vm_ops = generic_file_vm_ops+0x0/0x18
  vma->vm_ops->nopage = filemap_nopage+0x0/0x329
  vma->vm_file->f_op->mmap = generic_file_mmap+0x0/0x3b
------------[ cut here ]------------
kernel BUG at mm/rmap.c:596!
invalid opcode: 0000 [#5]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<c046d080>]    Not tainted VLI
EFLAGS: 00010282   (2.6.21-20.el5rt #1)
EIP is at page_remove_rmap+0xd7/0xf3
eax: 0000003b   ebx: c1000000   ecx: 00000046   edx: 00000000
esi: eff9ce9c   edi: ef7f6908   ebp: efb51a40   esp: efb51a30
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068  preempt:00000001
Process mkdir (pid: 4859, ti=efb51000 task=f13f97b0 task.ti=efb51000)
Stack: c06cf057 00000000 c1000000 00000000 efb51a98 c04679cc 00000000 eff9ce9c 
       efb51ab4 00000000 00000001 403c4000 f557d400 f63b6b80 c1e28a00 00000000 
       fffffff6 c1a63ddc f557d400 40242000 00363e39 403c4000 00000000 efb51ab4 
Call Trace:
 [<c04679cc>] unmap_vmas+0x25c/0x4e9
 [<c046a63e>] exit_mmap+0x8b/0x110
 [<c0425f36>] mmput+0x23/0x7a
 [<c0429a22>] exit_mm+0xef/0xf4
 [<c042adb2>] do_exit+0x1e2/0x75a
 [<c0406669>] die+0x211/0x237
 [<c06323b3>] do_page_fault+0x451/0x522
 [<c06309fc>] error_code+0x7c/0x84
 [<c04f0d5c>] vsnprintf+0x295/0x460
 [<c0452506>] audit_log_vformat+0x93/0x147
 [<c04525d5>] audit_log_format+0x1b/0x1d
 [<c04ce2b5>] avc_audit+0xc25/0xc38
 [<c04ce316>] avc_has_perm+0x4e/0x58
 [<c04cf737>] inode_has_perm+0x64/0x6c
 [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7
 [<c04803ae>] permission+0xcc/0xe0
 [<c04812f1>] vfs_permission+0xf/0x11
 [<c048186d>] __link_path_walk+0x143/0xc7a
 [<c04823ef>] link_path_walk+0x4b/0xc0
 [<c0482719>] do_path_lookup+0x191/0x1e2
 [<c0482f06>] __user_walk_fd+0x32/0x44
 [<c047cb54>] vfs_stat_fd+0x1b/0x41
 [<c047cbfc>] vfs_stat+0x11/0x13
 [<c047cc12>] sys_stat64+0x14/0x28
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff798
 =======================
Code: c0 74 0d 8b 50 08 b8 87 f0 6c c0 e8 06 bc fd ff 8b 46 48 85 c0 74 14 8b 40
30 85 c0 74 0d 8b 50 2c b8 a6 f0 6c c0 e8 eb bb fd ff <0f> 0b eb fe 8b 53 10 89
d8 83 e2 01 f7 da 83 c2 04 e8 d3 8a ff 
EIP: [<c046d080>] page_remove_rmap+0xd7/0xf3 SS:ESP 0068:efb51a30
Fixing recursive fault but reboot is needed!
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000020
 printing eip:
f89055f3
*pde = 00000000
Oops: 0000 [#6]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    0
EIP:    0060:[<f89055f3>]    Not tainted VLI
EFLAGS: 00010246   (2.6.21-20.el5rt #1)
EIP is at clone_endio+0x2e/0xcb [dm_mod]
eax: 00000000   ebx: 00000001   ecx: f8800080   edx: ef6353a8
esi: 00000000   edi: e9791b40   ebp: f7c27da4   esp: f7c27d80
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068  preempt:00000001
Process IRQ-17 (pid: 439, ti=f7c27000 task=f7c56e70 task.ti=f7c27000)
Stack: 14000000 00000050 f8800080 ef6353a8 00000001 00000000 e9791b40 00001000 
       f89055c5 f7c27dcc c0498d07 f796f800 f7c27db8 f88da3e1 00000000 00000000 
       e9791b40 00001000 00000000 f7c27e30 c04e27b4 00000000 c0808500 f58450c0 
Call Trace:
 [<c0498d07>] bio_endio+0x5f/0x67
 [<c04e27b4>] __end_that_request_first+0x220/0x484
 [<c04e2a20>] end_that_request_chunk+0x8/0xa
 [<f888d8e8>] scsi_end_request+0x20/0xaa [scsi_mod]
 [<f888db26>] scsi_io_completion+0x168/0x33f [scsi_mod]
 [<f88645dc>] sd_rw_intr+0x227/0x250 [sd_mod]
 [<f888936b>] scsi_finish_command+0x87/0x8f [scsi_mod]
 [<f888e2fb>] scsi_softirq_done+0xd0/0xd8 [scsi_mod]
 [<c04e4860>] blk_done_softirq+0x51/0x60
 [<c042cba6>] ___do_softirq+0xf3/0x1fa
 [<c042cfbc>] do_softirq_from_hardirq+0x4b/0x65
 [<c0458f3c>] do_irqd+0x202/0x24e
 [<c0439bcf>] kthread+0xb3/0xdc
 [<c0405c03>] kernel_thread_helper+0x7/0x10
 =======================
Code: 57 89 c7 56 89 ce 53 bb 01 00 00 00 83 ec 18 8b 40 3c 83 7f 20 00 89 45 e8
89 c2 8b 00 8b 00 89 45 ec 8b 4a 04 89 4d e4 8b 41 04 <8b> 40 20 89 45 f0 0f 85
87 00 00 00 f6 47 10 01 75 0a 85 f6 b8 
EIP: [<f89055f3>] clone_endio+0x2e/0xcb [dm_mod] SS:ESP 0068:f7c27d80
 at virtual address 04da000f
 printing eip:
c04f0d5c
*pde = 00000000
Oops: 0000 [#7]
PREEMPT SMP 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 acpi_cpufreq
microcode video sbs i2c_ec dock button battery asus_acpi ac lp floppy
snd_hda_intel snd_hda_codec sg snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ata_generic snd_pcm ata_piix
snd_timer snd rtc_cmos soundcore rtc_core tg3 rtc_lib snd_page_alloc i2c_i801
iTCO_wdt iTCO_vendor_support i2c_core pcspkr parport_pc parport serio_raw
dm_snapshot dm_zero dm_mirror dm_mod ahci libata mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
CPU:    1
EIP:    0060:[<c04f0d5c>]    Not tainted VLI
EFLAGS: 00010297   (2.6.21-20.el5rt #1)
EIP is at vsnprintf+0x295/0x460
eax: 04da000f   ebx: ffffffff   ecx: 04da000f   edx: fffffffe
esi: f63420d2   edi: ef77cc70   ebp: ef77cc30   esp: ef77cbe8
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000001
Process mv (pid: 4856, ti=ef77c000 task=f0b16070 task.ti=ef77c000)
Stack: ef77cc14 c0444213 f0f07697 ef77cc00 c044422f 00000246 00000336 f63420ca 
       007c1980 00000202 f6342400 ffffffff ffffffff 00000000 c06d6820 ef77cc70 
       f168f0c0 00000336 ef77cc54 c0452506 ef77cc70 c06d6817 f7ca89a0 ef77cc70 
Call Trace:
 [<c0452506>] audit_log_vformat+0x93/0x147
 [<c04525d5>] audit_log_format+0x1b/0x1d
 [<c04ce2b5>] avc_audit+0xc25/0xc38
 [<c04ce316>] avc_has_perm+0x4e/0x58
 [<c04cf737>] inode_has_perm+0x64/0x6c
 [<c04d2fb1>] selinux_inode_permission+0xa1/0xa7
 [<c04803ae>] permission+0xcc/0xe0
 [<c04812f1>] vfs_permission+0xf/0x11
 [<c048186d>] __link_path_walk+0x143/0xc7a
 [<c04823ef>] link_path_walk+0x4b/0xc0
 [<c0482719>] do_path_lookup+0x191/0x1e2
 [<c0482f06>] __user_walk_fd+0x32/0x44
 [<c047cb54>] vfs_stat_fd+0x1b/0x41
 [<c047cbfc>] vfs_stat+0x11/0x13
 [<c047cc12>] sys_stat64+0x14/0x28
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff798
 =======================
Code: 73 03 c6 06 20 ff 4d e4 46 83 7d e4 00 7f ee e9 b9 00 00 00 8b 0f b8 8a 7c
6d c0 8b 55 e8 81 f9 ff 0f 00 00 0f 46 c8 89 c8 eb 06 <80> 38 00 74 07 40 4a 83
fa ff 75 f4 29 c8 f6 45 ec 10 89 c3 8b 
EIP: [<c04f0d5c>] vsnprintf+0x295/0x460 SS:ESP 0068:ef77cbe8
Comment 7 Tim Burke 2007-06-07 11:48:28 EDT
Konrad is supposed to be buying new DIMMS.
Comment 8 Jeff Burke 2007-06-08 11:14:06 EDT
The engineering services ticket was just resolved. But it doesn't say what the
resolution was/is.  Let me know if teh system can be used for testing, I
willschedule an RT-kernel test.

https://engineering.redhat.com/rt3/Ticket/Display.html?id=12358
Comment 9 Tim Burke 2007-07-26 09:09:42 EDT
Closing this one out.  Reopen if problem reoccurs with fixed memory.

Note You need to log in before you can comment on or make changes to this bug.