Bug 123153

Summary: Kernel panic - system hangs
Product: Red Hat Enterprise Linux 2.1 Reporter: Werner Prodinger <werner>
Component: kernelAssignee: Jason Baron <jbaron>
Status: CLOSED NOTABUG QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.1CC: jbaron, knoel, riel, shillman
Target Milestone: ---   
Target Release: ---   
Hardware: i586   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-04-11 20:40:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Werner Prodinger 2004-05-13 06:49:00 UTC
Redhat ES2.1 hangs on Dell 6650 with no network response and no 
console response with only the following error in /var/log/messages. 
The happens regularly every 30-60 days and typically under heavy 
load - there is a full system backup at 4am to a remote mounted SAN 
drive '/mnt/emcpowera'. The last two hangs were approx. 4am.

May 12 04:02:02 bkvavnweb1 syslogd 1.4.1: restart.
May 12 04:15:08 bkvavnweb1 kernel: journal_write_metadata_buffer: 
ENOMEM at get_unused_buffer_head, trying again.
May 12 04:15:48 bkvavnweb1 last message repeated 5 times
May 12 04:16:53 bkvavnweb1 last message repeated 5 times
May 12 04:17:54 bkvavnweb1 last message repeated 6 times
May 12 04:18:49 bkvavnweb1 last message repeated 7 times
May 12 04:19:03 bkvavnweb1 kernel: Unable to handle kernel paging 
request at virtual address 61696369
May 12 04:19:03 bkvavnweb1 kernel:  printing eip:
May 12 04:19:03 bkvavnweb1 kernel: c01305fc
May 12 04:19:03 bkvavnweb1 kernel: *pde = 00000000
May 12 04:19:03 bkvavnweb1 kernel: Oops: 0002
May 12 04:19:03 bkvavnweb1 kernel: Kernel 2.4.9-e.27smp
May 12 04:19:03 bkvavnweb1 kernel: CPU:    3
May 12 04:19:03 bkvavnweb1 kernel: EIP:    0010:
[filemap_fdatawait+76/192]    Tainted: P
May 12 04:19:03 bkvavnweb1 kernel: EIP:    0010:[<c01305fc>]    
Tainted: P
May 12 04:19:03 bkvavnweb1 kernel: EFLAGS: 00010202
May 12 04:19:03 bkvavnweb1 kernel: EIP is at filemap_fdatawait 
[kernel] 0x4c
May 12 04:19:03 bkvavnweb1 kernel: eax: 61696365   ebx: cbb19ed7   
ecx: f703eb28   edx: 0000006c
May 12 04:19:03 bkvavnweb1 kernel: esi: cbb1a2b8   edi: 00000300   
ebp: f5c09c60   esp: c4befef4
May 12 04:19:03 bkvavnweb1 kernel: ds: 0018   es: 0018   ss: 0018
May 12 04:19:03 bkvavnweb1 kernel: Process kswapd (pid: 10, 
stackpage=c4bef000)
May 12 04:19:03 bkvavnweb1 kernel: Stack: 00000000 cbb1a200 c015b8b9 
cbb1a2b8 00000000 f5c09c00 c5fca580 c4beff60
May 12 04:19:03 bkvavnweb1 kernel:        00000006 c015b02a c4bbb470 
c5fca580 c5fca580 c015bdfb 00000000 00000000
May 12 04:19:03 bkvavnweb1 kernel:        00000006 00000000 c015c295 
0000067c f05a7048 0000067c ffffffff 00000000
May 12 04:19:03 bkvavnweb1 kernel: Call Trace: 
[try_to_sync_unused_inodes+297/528] try_to_sync_unused_inodes 
[kernel] 0x129
May 12 04:19:03 bkvavnweb1 kernel: Call Trace: [<c015b8b9>] 
try_to_sync_unused_inodes [kernel] 0x129
May 12 04:19:03 bkvavnweb1 kernel: [destroy_inode+42/48] 
destroy_inode [kernel] 0x2a
May 12 04:19:03 bkvavnweb1 kernel: [<c015b02a>] destroy_inode 
[kernel] 0x2a
May 12 04:19:03 bkvavnweb1 kernel: [dispose_list+75/96] dispose_list 
[kernel] 0x4b
May 12 04:19:03 bkvavnweb1 kernel: [<c015bdfb>] dispose_list [kernel] 
0x4b
May 12 04:19:03 bkvavnweb1 kernel: [prune_icache+757/784] 
prune_icache [kernel] 0x2f5
May 12 04:19:03 bkvavnweb1 kernel: [<c015c295>] prune_icache [kernel] 
0x2f5
May 12 04:19:03 bkvavnweb1 kernel: [shrink_icache_memory+33/64] 
shrink_icache_memory [kernel] 0x21
May 12 04:19:03 bkvavnweb1 kernel: [<c015c2d1>] shrink_icache_memory 
[kernel] 0x21
May 12 04:19:03 bkvavnweb1 kernel: [do_try_to_free_pages+38/144] 
do_try_to_free_pages [kernel] 0x26
May 12 04:19:03 bkvavnweb1 kernel: [<c013c8c6>] do_try_to_free_pages 
[kernel] 0x26
May 12 04:19:03 bkvavnweb1 kernel: [kswapd+259/432] kswapd [kernel] 
0x103
May 12 04:19:03 bkvavnweb1 kernel: [<c013ca33>] kswapd [kernel] 0x103
May 12 04:19:03 bkvavnweb1 kernel: [kswapd+0/432] kswapd [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [<c013c930>] kswapd [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [_stext+0/80] stext [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [<c0105000>] stext [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [_stext+0/80] stext [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [<c0105000>] stext [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [arch_kernel_thread+38/48] 
arch_kernel_thread [kernel] 0x26
May 12 04:19:03 bkvavnweb1 kernel: [<c0105836>] arch_kernel_thread 
[kernel] 0x26
May 12 04:19:03 bkvavnweb1 kernel: [kswapd+0/432] kswapd [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel: [<c013c930>] kswapd [kernel] 0x0
May 12 04:19:03 bkvavnweb1 kernel:
May 12 04:19:03 bkvavnweb1 kernel:
May 12 04:19:03 bkvavnweb1 kernel: Code: 89 50 04 89 02 c7 43 04 00 
00 00 00 c7 03 00 00 00 00 8b 06
May 12 04:19:03 bkvavnweb1 kernel:  <0>Kernel panic: not continuing
May 12 08:15:36 bkvavnweb1 syslogd 1.4.1: restart.
May 12 08:15:36 bkvavnweb1 syslog: syslogd startup succeeded

Comment 1 Arjan van de Ven 2004-05-13 06:50:57 UTC
> Tainted: P

which modules are in use ?
Also you're running quite an old kernel; some ext3 interactions with
certain binary only kernel modules have since been fixed/worked around.

Comment 2 Werner Prodinger 2004-05-13 06:56:48 UTC
Modules from /etc/modules.conf

[root@bkvavnweb1 etc]# cat modules.conf
alias parport_lowlevel parport_pc
alias scsi_hostadapter aic7xxx
alias scsi_hostadapter1 megaraid
alias eth0 bcm5700
alias eth1 bcm5700
alias scsi_hostadapter2 megaraid
alias usb-controller usb-ohci
alias scsi_hostadapter97 qla2300_6x
options scsi_mod scsi_allow_ghost_devices=1
add probeall power_path emcpmp
add probeall power_path emcpmpc
add probeall power_path emcppn
add probeall power_path emcp
alias power_path emcp
add probeall power_path emcpioc
post-install emcpioc rmmod emcpioc
add below emcp qla2300_6x
[root@bkvavnweb1 etc]#


Comment 3 Arjan van de Ven 2004-05-13 07:09:47 UTC
I recommend that you open a ticket with Red Hat support; they can
escalate things to EMC regarding powerpath that we in engineering
can't via bugzilla. In addition I strongly recommend you to go to the
latest erratum kernel with all the important fixes.

Comment 4 Suzanne Hillman 2005-04-11 20:40:40 UTC
Closing, as presumably this was brought to support as requested.