Redhat ES2.1 hangs on Dell 6650 with no network response and no console response with only the following error in /var/log/messages. The happens regularly every 30-60 days and typically under heavy load - there is a full system backup at 4am to a remote mounted SAN drive '/mnt/emcpowera'. The last two hangs were approx. 4am. May 12 04:02:02 bkvavnweb1 syslogd 1.4.1: restart. May 12 04:15:08 bkvavnweb1 kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. May 12 04:15:48 bkvavnweb1 last message repeated 5 times May 12 04:16:53 bkvavnweb1 last message repeated 5 times May 12 04:17:54 bkvavnweb1 last message repeated 6 times May 12 04:18:49 bkvavnweb1 last message repeated 7 times May 12 04:19:03 bkvavnweb1 kernel: Unable to handle kernel paging request at virtual address 61696369 May 12 04:19:03 bkvavnweb1 kernel: printing eip: May 12 04:19:03 bkvavnweb1 kernel: c01305fc May 12 04:19:03 bkvavnweb1 kernel: *pde = 00000000 May 12 04:19:03 bkvavnweb1 kernel: Oops: 0002 May 12 04:19:03 bkvavnweb1 kernel: Kernel 2.4.9-e.27smp May 12 04:19:03 bkvavnweb1 kernel: CPU: 3 May 12 04:19:03 bkvavnweb1 kernel: EIP: 0010: [filemap_fdatawait+76/192] Tainted: P May 12 04:19:03 bkvavnweb1 kernel: EIP: 0010:[<c01305fc>] Tainted: P May 12 04:19:03 bkvavnweb1 kernel: EFLAGS: 00010202 May 12 04:19:03 bkvavnweb1 kernel: EIP is at filemap_fdatawait [kernel] 0x4c May 12 04:19:03 bkvavnweb1 kernel: eax: 61696365 ebx: cbb19ed7 ecx: f703eb28 edx: 0000006c May 12 04:19:03 bkvavnweb1 kernel: esi: cbb1a2b8 edi: 00000300 ebp: f5c09c60 esp: c4befef4 May 12 04:19:03 bkvavnweb1 kernel: ds: 0018 es: 0018 ss: 0018 May 12 04:19:03 bkvavnweb1 kernel: Process kswapd (pid: 10, stackpage=c4bef000) May 12 04:19:03 bkvavnweb1 kernel: Stack: 00000000 cbb1a200 c015b8b9 cbb1a2b8 00000000 f5c09c00 c5fca580 c4beff60 May 12 04:19:03 bkvavnweb1 kernel: 00000006 c015b02a c4bbb470 c5fca580 c5fca580 c015bdfb 00000000 00000000 May 12 04:19:03 bkvavnweb1 kernel: 00000006 00000000 c015c295 0000067c f05a7048 0000067c ffffffff 00000000 May 12 04:19:03 bkvavnweb1 kernel: Call Trace: [try_to_sync_unused_inodes+297/528] try_to_sync_unused_inodes [kernel] 0x129 May 12 04:19:03 bkvavnweb1 kernel: Call Trace: [<c015b8b9>] try_to_sync_unused_inodes [kernel] 0x129 May 12 04:19:03 bkvavnweb1 kernel: [destroy_inode+42/48] destroy_inode [kernel] 0x2a May 12 04:19:03 bkvavnweb1 kernel: [<c015b02a>] destroy_inode [kernel] 0x2a May 12 04:19:03 bkvavnweb1 kernel: [dispose_list+75/96] dispose_list [kernel] 0x4b May 12 04:19:03 bkvavnweb1 kernel: [<c015bdfb>] dispose_list [kernel] 0x4b May 12 04:19:03 bkvavnweb1 kernel: [prune_icache+757/784] prune_icache [kernel] 0x2f5 May 12 04:19:03 bkvavnweb1 kernel: [<c015c295>] prune_icache [kernel] 0x2f5 May 12 04:19:03 bkvavnweb1 kernel: [shrink_icache_memory+33/64] shrink_icache_memory [kernel] 0x21 May 12 04:19:03 bkvavnweb1 kernel: [<c015c2d1>] shrink_icache_memory [kernel] 0x21 May 12 04:19:03 bkvavnweb1 kernel: [do_try_to_free_pages+38/144] do_try_to_free_pages [kernel] 0x26 May 12 04:19:03 bkvavnweb1 kernel: [<c013c8c6>] do_try_to_free_pages [kernel] 0x26 May 12 04:19:03 bkvavnweb1 kernel: [kswapd+259/432] kswapd [kernel] 0x103 May 12 04:19:03 bkvavnweb1 kernel: [<c013ca33>] kswapd [kernel] 0x103 May 12 04:19:03 bkvavnweb1 kernel: [kswapd+0/432] kswapd [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [<c013c930>] kswapd [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [_stext+0/80] stext [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [<c0105000>] stext [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [_stext+0/80] stext [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [<c0105000>] stext [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [arch_kernel_thread+38/48] arch_kernel_thread [kernel] 0x26 May 12 04:19:03 bkvavnweb1 kernel: [<c0105836>] arch_kernel_thread [kernel] 0x26 May 12 04:19:03 bkvavnweb1 kernel: [kswapd+0/432] kswapd [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: [<c013c930>] kswapd [kernel] 0x0 May 12 04:19:03 bkvavnweb1 kernel: May 12 04:19:03 bkvavnweb1 kernel: May 12 04:19:03 bkvavnweb1 kernel: Code: 89 50 04 89 02 c7 43 04 00 00 00 00 c7 03 00 00 00 00 8b 06 May 12 04:19:03 bkvavnweb1 kernel: <0>Kernel panic: not continuing May 12 08:15:36 bkvavnweb1 syslogd 1.4.1: restart. May 12 08:15:36 bkvavnweb1 syslog: syslogd startup succeeded
> Tainted: P which modules are in use ? Also you're running quite an old kernel; some ext3 interactions with certain binary only kernel modules have since been fixed/worked around.
Modules from /etc/modules.conf [root@bkvavnweb1 etc]# cat modules.conf alias parport_lowlevel parport_pc alias scsi_hostadapter aic7xxx alias scsi_hostadapter1 megaraid alias eth0 bcm5700 alias eth1 bcm5700 alias scsi_hostadapter2 megaraid alias usb-controller usb-ohci alias scsi_hostadapter97 qla2300_6x options scsi_mod scsi_allow_ghost_devices=1 add probeall power_path emcpmp add probeall power_path emcpmpc add probeall power_path emcppn add probeall power_path emcp alias power_path emcp add probeall power_path emcpioc post-install emcpioc rmmod emcpioc add below emcp qla2300_6x [root@bkvavnweb1 etc]#
I recommend that you open a ticket with Red Hat support; they can escalate things to EMC regarding powerpath that we in engineering can't via bugzilla. In addition I strongly recommend you to go to the latest erratum kernel with all the important fixes.
Closing, as presumably this was brought to support as requested.