From Bugzilla Helper: User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) Description of problem: After an upgrade to 2.4.9-31 I an receiving kernel crashes OFTEN! Apr 3 12:18:16 server4 kernel: Unable to handle kernel paging request at virtua l address 04000000 Apr 3 12:18:16 server4 kernel: printing eip: Apr 3 12:18:16 server4 kernel: c0137d59 Apr 3 12:18:16 server4 kernel: *pde = 00000000 Apr 3 12:18:16 server4 kernel: Oops: 0002 Apr 3 12:18:16 server4 kernel: Kernel 2.4.9-31 Apr 3 12:18:16 server4 kernel: CPU: 0 Apr 3 12:18:16 server4 kernel: EIP: 0010:[__remove_from_queues+25/48] Not tainted Apr 3 12:18:16 server4 kernel: EIP: 0010:[<c0137d59>] Not tainted Apr 3 12:18:16 server4 kernel: EFLAGS: 00010246 Apr 3 12:18:16 server4 kernel: EIP is at __remove_from_queues [kernel] 0x19 Apr 3 12:18:16 server4 kernel: eax: 00000000 ebx: c403b800 ecx: c403b800 edx: 04000000 Apr 3 12:18:16 server4 kernel: esi: c403b800 edi: c403b800 ebp: 00000000 esp: c188df68 Apr 3 12:18:16 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 12:18:16 server4 kernel: Process kswapd (pid: 5, stackpage=c188d000) Apr 3 12:18:16 server4 kernel: Stack: c013a900 c403b800 00000003 c4485260 c1123 254 00000000 00000007 00000000 Apr 3 12:18:16 server4 kernel: c1110ab0 00000000 00000007 c012ec94 c1110 ab0 00000080 00000000 00000000 Apr 3 12:18:16 server4 kernel: 00005ce7 00001dd1 00000000 00000000 00000 0c0 c188c241 0008e000 c012f501 Apr 3 12:18:16 server4 kernel: Call Trace: [try_to_free_buffers+112/272] try_to _free_buffers [kernel] 0x70 @ Apr 3 12:18:16 server4 kernel: 00005ce7 00001dd1 00000000 00000000 00000 0c0 c188c241 0008e000 c012f501 Apr 3 12:18:16 server4 kernel: Call Trace: [try_to_free_buffers+112/272] try_to _free_buffers [kernel] 0x70 Apr 3 12:18:16 server4 kernel: Call Trace: [<c013a900>] try_to_free_buffers [ke rnel] 0x70 Apr 3 12:18:16 server4 kernel: [page_launder+1108/2368] page_launder [kernel] 0 x454 Apr 3 12:18:16 server4 kernel: [<c012ec94>] page_launder [kernel] 0x454 Apr 3 12:18:16 server4 kernel: [do_try_to_free_pages+17/80] do_try_to_free_page s [kernel] 0x11 Apr 3 12:18:16 server4 kernel: [<c012f501>] do_try_to_free_pages [kernel] 0x11 Apr 3 12:18:16 server4 kernel: [kswapd+85/240] kswapd [kernel] 0x55 Apr 3 12:18:16 server4 kernel: [<c012f595>] kswapd [kernel] 0x55 Apr 3 12:18:16 server4 kernel: [_stext+0/48] stext [kernel] 0x0 Apr 3 12:18:16 server4 kernel: [<c0105000>] stext [kernel] 0x0 Apr 3 12:18:16 server4 kernel: [kernel_thread+38/48] kernel_thread [kernel] 0x2 6 Apr 3 12:18:16 server4 kernel: [<c0105726>] kernel_thread [kernel] 0x26 Apr 3 12:18:16 server4 kernel: [kswapd+0/240] kswapd [kernel] 0x0 Apr 3 12:18:16 server4 kernel: [<c012f540>] kswapd [kernel] 0x0 Apr 3 12:18:16 server4 kernel: Apr 3 12:18:16 server4 kernel: Apr 3 12:18:16 server4 kernel: Code: 89 02 c7 41 30 00 00 00 00 0f b7 41 0a 50 51 e8 03 ff ff ff ----- this may also be connected (an earlier Oopps) ------- Apr 3 04:06:23 server4 kernel: Slab inode_cache is still in use! Apr 3 04:06:23 server4 kernel: ------------[ cut here ]------------ Apr 3 04:06:23 server4 kernel: kernel BUG at slab.c:933! Apr 3 04:06:23 server4 kernel: invalid operand: 0000 Apr 3 04:06:23 server4 kernel: Kernel 2.4.9-31 Apr 3 04:06:23 server4 kernel: CPU: 0 Apr 3 04:06:23 server4 kernel: EIP: 0010:[__kmem_cache_shrink+70/176] Not tainted Apr 3 04:06:23 server4 kernel: EIP: 0010:[<c012cd66>] Not tainted Apr 3 04:06:23 server4 kernel: EFLAGS: 00010082 Apr 3 04:06:23 server4 kernel: EIP is at __kmem_cache_shrink [kernel] 0x46 Apr 3 04:06:23 server4 kernel: eax: 0000001a ebx: c037b000 ecx: 00000001 edx: 00018eeb Apr 3 04:06:23 server4 kernel: esi: c1c861a0 edi: c188c000 ebp: 0008e000 esp: c188dfa0 Apr 3 04:06:23 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:06:23 server4 kernel: Process kswapd (pid: 5, stackpage=c188d000) Apr 3 04:06:23 server4 kernel: Stack: c022e091 000003a5 00000001 000000c0 00000 0c0 c012f513 c1c861a0 000000c0 Apr 3 04:06:23 server4 kernel: c1c86130 000000c0 000000c0 00000000 c188c 000 00000006 c012f595 000000c0 Apr 3 04:06:23 server4 kernel: 00000000 00010f00 c188bfb8 c0105000 c0105 726 00000000 c012f540 c02c9fd8 Apr 3 04:06:23 server4 kernel: Call Trace: [IRQ0x0f_interrupt+112881/138304] .r odata.str1.1 [kernel] 0x21cc Apr 3 04:06:23 server4 kernel: Call Trace: [<c022e091>] .rodata.str1.1 [kernel] 0x21cc Apr 3 04:06:23 server4 kernel: [do_try_to_free_pages+35/80] do_try_to_free_page s [kernel] 0x23 Apr 3 04:06:23 server4 kernel: [<c012f513>] do_try_to_free_pages [kernel] 0x23 Apr 3 04:06:23 server4 kernel: [kswapd+85/240] kswapd [kernel] 0x55 Apr 3 04:06:23 server4 kernel: [<c012f595>] kswapd [kernel] 0x55 Apr 3 04:06:23 server4 kernel: [_stext+0/48] stext [kernel] 0x0 Apr 3 04:06:23 server4 kernel: [<c0105000>] stext [kernel] 0x0 Apr 3 04:06:23 server4 kernel: [kernel_thread+38/48] kernel_thread [kernel] 0x2 6 Apr 3 04:06:23 server4 kernel: [<c0105726>] kernel_thread [kernel] 0x26 Apr 3 04:06:23 server4 kernel: [kswapd+0/240] kswapd [kernel] 0x0 Apr 3 04:06:23 server4 kernel: [<c012f540>] kswapd [kernel] 0x0 Apr 3 04:06:23 server4 kernel: Apr 3 04:06:23 server4 kernel: Apr 3 04:06:23 server4 kernel: Code: 0f 0b 58 5a 8b 53 04 8b 03 89 50 04 89 02 c7 43 04 00 00 00 Apr 3 04:06:40 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:06:40 server4 kernel: printing eip: Apr 3 04:06:40 server4 kernel: c012d1b1 Apr 3 04:06:40 server4 kernel: *pde = 00000000 Apr 3 04:06:40 server4 kernel: Oops: 0002 Apr 3 04:06:40 server4 kernel: Kernel 2.4.9-31 Apr 3 04:06:40 server4 kernel: CPU: 0 Apr 3 04:06:40 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:06:40 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:06:40 server4 kernel: EFLAGS: 00010046 Apr 3 04:06:40 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:06:40 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:06:40 server4 kernel: esi: 00000286 edi: c0057e40 ebp: c1f78800 esp: d145bdfc Apr 3 04:06:40 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:06:40 server4 kernel: Process updatedb (pid: 19804, stackpage=d145b000 ) Apr 3 04:06:40 server4 kernel: Stack: 00000000 c59d6900 00000000 c0159892 c1c86 1a0 dff95898 dff95898 c0149bd0 Apr 3 04:06:40 server4 kernel: c1c861a0 000000f0 00000202 c1c86130 c59d6 900 c0159c5b 00000000 dff95898 Apr 3 04:06:40 server4 kernel: 000562a9 c1f78800 c0149f02 c1f78800 00056 2a9 dff95898 00000000 00000000 Apr 3 04:06:40 server4 kernel: Call Trace: [ext2_get_page+34/240] ext2_get_page [kernel] 0x22 Apr 3 04:06:40 server4 kernel: Call Trace: [<c0159892>] ext2_get_page [kernel] 0x22 Apr 3 04:06:40 server4 kernel: [get_new_inode+32/384] get_new_inode [kernel] 0x 20 Apr 3 04:06:40 server4 kernel: [<c0149bd0>] get_new_inode [kernel] 0x20 Apr 3 04:06:40 server4 kernel: [ext2_find_entry+123/368] ext2_find_entry [kerne l] 0x7b Apr 3 04:06:40 server4 kernel: [<c0159c5b>] ext2_find_entry [kernel] 0x7b Apr 3 04:06:40 server4 kernel: [iget4+210/224] iget4 [kernel] 0xd2 Apr 3 04:06:40 server4 kernel: [ext2_inode_by_name+25/112] ext2_inode_by_name [ kernel] 0x19 Apr 3 04:06:40 server4 kernel: [<c0159d99>] ext2_inode_by_name [kernel] 0x19 Apr 3 04:06:40 server4 kernel: [ext2_lookup+67/112] ext2_lookup [kernel] 0x43 Apr 3 04:06:40 server4 kernel: [<c015cd53>] ext2_lookup [kernel] 0x43 Apr 3 04:06:40 server4 kernel: [real_lookup+79/192] real_lookup [kernel] 0x4f Apr 3 04:06:40 server4 kernel: [<c013fc7f>] real_lookup [kernel] 0x4f Apr 3 04:06:40 server4 kernel: [path_walk+1462/2064] path_walk [kernel] 0x5b6 Apr 3 04:06:40 server4 kernel: [<c01403a6>] path_walk [kernel] 0x5b6 Apr 3 04:06:40 server4 kernel: [__user_walk+58/96] __user_walk [kernel] 0x3a Apr 3 04:06:40 server4 kernel: [<c0140aca>] __user_walk [kernel] 0x3a Apr 3 04:06:40 server4 kernel: [vfs_lstat+20/80] vfs_lstat [kernel] 0x14 Apr 3 04:06:40 server4 kernel: [<c013d434>] vfs_lstat [kernel] 0x14 Apr 3 04:06:40 server4 kernel: [sys_lstat64+17/48] sys_lstat64 [kernel] 0x11 Apr 3 04:06:40 server4 kernel: [<c013d9b1>] sys_lstat64 [kernel] 0x11 Apr 3 04:06:40 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:06:40 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:06:40 server4 kernel: printing eip: Apr 3 04:06:40 server4 kernel: c012d1b1 Apr 3 04:06:40 server4 kernel: *pde = 00000000 Apr 3 04:06:40 server4 kernel: Oops: 0002 Apr 3 04:06:40 server4 kernel: Kernel 2.4.9-31 Apr 3 04:06:40 server4 kernel: CPU: 0 Apr 3 04:06:40 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:06:40 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:06:40 server4 kernel: EFLAGS: 00010046 Apr 3 04:06:40 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:06:40 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:06:40 server4 kernel: esi: 00000286 edi: c0057c80 ebp: c3b1eda0 esp: c823df0c Apr 3 04:06:40 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:06:40 server4 kernel: Process crond (pid: 19631, stackpage=c823d000) Apr 3 04:06:40 server4 kernel: Stack: d21999a0 cfdcc2e0 08051ca4 00000001 c1c86 1a0 cfbb35a0 bfffe7f8 c0149b42 Apr 3 04:06:40 server4 kernel: c1c861a0 000000f0 c823c000 c013f5b6 c823c 000 c013f682 00000286 00000001 Apr 3 04:06:40 server4 kernel: ffffffe9 d5d15ea0 db240900 00000f44 c819d 000 c013efb1 000000bc 000000bc Apr 3 04:06:40server4 kernel: Call Trace: [get_empty_inode+18/128] get_empty_i node [kernel] 0x12 Apr 3 04:06:40 server4 kernel: Call Trace: [<c0149b42>] get_empty_inode [kernel ] 0x12 Apr 3 04:06:40 server4 kernel: [get_pipe_inode+6/160] get_pipe_inode [kernel] 0 x6 Apr 3 04:06:40 server4 kernel: [<c013f5b6>] get_pipe_inode [kernel] 0x6 Apr 3 04:06:40 server4 kernel: [do_pipe+50/608] do_pipe [kernel] 0x32 Apr 3 04:06:40 server4 kernel: [<c013f682>] do_pipe [kernel] 0x32 Apr 3 04:06:40 server4 kernel: [pipe_read+497/544] pipe_read [kernel] 0x1f1 Apr 3 04:06:40 server4 kernel: [<c013efb1>] pipe_read [kernel] 0x1f1 Apr 3 04:06:40 server4 kernel: [sys_read+235/256] sys_read [kernel] 0xeb Apr 3 04:06:40 server4 kernel: [<c013681b>] sys_read [kernel] 0xeb Apr 3 04:06:40 server4 kernel: [sys_pipe+19/96] sys_pipe [kernel] 0x13 Apr 3 04:06:40 server4 kernel: [<c010b353>] sys_pipe [kernel] 0x13 Apr 3 04:06:40 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:06:40 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:06:40 server4 kernel: printing eip: Apr 3 04:06:40 server4 kernel: c012d1b1 Apr 3 04:06:40 server4 kernel: *pde = 00000000 Apr 3 04:06:40 server4 kernel: Oops: 0002 Apr 3 04:06:40 server4 kernel: Kernel 2.4.9-31 Apr 3 04:06:40 server4 kernel: CPU: 0 Apr 3 04:06:40 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:06:40 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:06:40 server4 kernel: EFLAGS: 00010046 Apr 3 04:06:40 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:06:40 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:06:40 server4 kernel: esi: 00000286 edi: c0057ac0 ebp: c1bf2000 esp: c6737c0c Apr 3 04:06:40 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:06:40 server4 kernel: Process run-parts (pid: 19808, stackpage=c673700 Apr 3 04:06:40 server4 kernel: Stack: 00000000 c1b9de40 00000000 c0159892 c1c86 1a0 dff8aa68 dff8aa68 c0149bd0 Apr 3 04:06:40 server4 kernel: c1c861a0 000000f0 c020773b 00000010 c1b9d e40 c0159c5b 00000000 dff8aa68 Apr 3 04:06:40 server4 kernel: 0001902e c1bf2000 c0149f02 c1bf2000 00019 02e dff8aa68 00000000 00000000 Apr 3 04:06:40 server4 kernel: Call Trace: [ext2_get_page+34/240] ext2_get_page [kernel] 0x22 Apr 3 04:06:40 server4 kernel: Call Trace: [<c0159892>] ext2_get_page [kernel] 0x22 Apr 3 04:06:40 server4 kernel: [get_new_inode+32/384] get_new_inode [kernel] 0x 20 Apr 3 04:06:40 server4 kernel: [<c0149bd0>] get_new_inode [kernel] 0x20 Apr 3 04:06:40 server4 kernel: [clear_user+43/64] clear_user [kernel] 0x2b Apr 3 04:06:40 server4 kernel: [<c020773b>] clear_user [kernel] 0x2b Apr 3 04:06:40 server4 kernel: [ext2_find_entry+123/368] ext2_find_entry [kerne l] 0x7b Apr 3 04:06:40 server4 kernel: [<c0159c5b>] ext2_find_entry [kernel] 0x7b Apr 3 04:06:40 server4 kernel: [iget4+210/224] iget4 [kernel] 0xd2 Apr 3 04:06:40 server4 kernel: [<c0149f02>] iget4 [kernel] 0xd2 Apr 3 04:06:40 server4 kernel: [ext2_inode_by_name+25/112] ext2_inode_by_name [ kernel] 0x19 Apr 3 04:06:40 server4 kernel: [<c0159d99>] ext2_inode_by_name [kernel] 0x19 Apr 3 04:06:40 server4 kernel: [ext2_lookup+67/112] ext2_lookup [kernel] 0x43 Apr 3 04:06:40 server4 kernel: [<c015cd53>] ext2_lookup [kernel] 0x43 Apr 3 04:06:40 server4 kernel: [real_lookup+79/192] real_lookup [kernel] 0x4f Apr 3 04:06:40 server4 kernel: [<c013fc7f>] real_lookup [kernel] 0x4f Apr 3 04:06:40 server4 kernel: [path_walk+1462/2064] path_walk [kernel] 0x5b6 Apr 3 04:06:40 server4 kernel: [<c01403a6>] path_walk [kernel] 0x5b6 Apr 3 04:06:40 server4 kernel: [open_exec+37/192] open_exec [kernel] 0x25 Apr 3 04:06:40 server4 kernel: [<c013e015>] open_exec [kernel] 0x25 Apr 3 04:06:40 server4 kernel: [copy_strings_kernel+39/48] copy_strings_kernel [kernel] 0x27 Apr 3 04:06:40 server4 kernel: [<c013de27>] copy_strings_kernel [kernel] 0x27 Apr 3 04:06:40 server4 kernel: [load_script+366/432] load_script [kernel] 0x16e Apr 3 04:06:40 server4 kernel: [<c015028e>] load_script [kernel] 0x16e Apr 3 04:06:40 server4 kernel: [update_atime+74/80] update_atime [kernel] 0x4a Apr 3 04:06:40 server4 kernel: [<c014a3ca>] update_atime [kernel] 0x4a Apr 3 04:06:40 server4 kernel: [do_generic_file_read+1300/1312] do_generic_file _read [kernel] 0x514 Apr 3 04:06:40 server4 kernel: [load_script+0/432] load_script [kernel] 0x0 Apr 3 04:06:40 server4 kernel: [<c0150120>] load_script [kernel] 0x0 Apr 3 04:06:40 server4 kernel: [search_binary_handler+113/384] search_binary_ha ndler [kernel] 0x71 Apr 3 04:06:40 server4 kernel: [<c013e881>] search_binary_handler [kernel] 0x71 Apr 3 04:06:40 server4 kernel: [do_execve+380/480] do_execve [kernel] 0x17c Apr 3 04:06:40 server4 kernel: [<c013eb0c>] do_execve [kernel] 0x17c Apr 3 04:06:40 server4 kernel: [getname+94/160] getname [kernel] 0x5e Apr 3 04:06:40 server4 kernel: [<c013f9ce>] getname [kernel] 0x5e Apr 3 04:06:40 server4 kernel: [sys_execve+48/96] sys_execve [kernel] 0x30 Apr 3 04:06:40 server4 kernel: [<c0105b70>] sys_execve [kernel] 0x30 Apr 3 04:06:40 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:06:40 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:06:40 server4 kernel: printing eip: Apr 3 04:06:40 server4 kernel: c012d1b1 Apr 3 04:06:40 server4 kernel: *pde = 00000000 Apr 3 04:06:40 server4 kernel: Oops: 0002 Apr 3 04:06:40 server4 kernel: Kernel 2.4.9-31 Apr 3 04:06:40 server4 kernel: CPU: 0 Apr 3 04:06:40 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:06:40 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:06:40 server4 kernel: EFLAGS: 00010046 Apr 3 04:06:40 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:06:40 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:06:40 server4 kernel: esi: 00000282 edi: c0057900 ebp: bffff4b0 esp: d86f5ebc Apr 3 04:06:40 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:06:40 server4 kernel: Process esd (pid: 1416, stackpage=d86f5000) Apr 3 04:06:40 server4 kernel: Stack: 00000000 d85936c0 c013038f 000000f0 c1c86 1a0 d85936a4 d86f5ef8 c0149b42 Apr 3 04:06:40 server4 kernel: c1c861a0 000000f0 00000005 c01bbd36 00000 005 c01bc96c 0000c0e8 ffffffe8 Apr 3 04:06:40 server4 kernel: c1044010 c02b2f04 00000216 ffffffff 00006 074 cd0e8008 00000000 cd0e8008 Apr 3 04:06:40 server4 kernel: Call Trace: [__alloc_pages+15/160] __alloc_pages [kernel] 0xf Apr 3 04:06:40 server4 kernel: Call Trace: [<c013038f>] __alloc_pages [kernel] 0xf Apr 3 04:06:40 server4 kernel: [get_empty_inode+18/128] get_empty_inode [kernel ] 0x12 Apr 3 04:06:40 server4 kernel: [<c0149b42>] get_empty_inode [kernel] 0x12 Apr 3 04:06:40 server4 kernel: [sock_alloc+6/176] sock_alloc [kernel] 0x6 Apr 3 04:06:40 server4 kernel: [<c01bbd36>] sock_alloc [kernel] 0x6 Apr 3 04:06:40 server4 kernel: [sys_accept+60/240] sys_accept [kernel] 0x3c Apr 3 04:06:40 server4 kernel: [<c01bc96c>] sys_accept [kernel] 0x3c Apr 3 04:06:40 server4 kernel: [poll_freewait+68/80] poll_freewait [kernel] 0x4 4 Apr 3 04:06:40 server4 kernel: [<c0144324>] poll_freewait [kernel] 0x44 Apr 3 04:06:40 server4 kernel: [do_select+550/576] do_select [kernel] 0x226 Apr 3 04:06:40 server4 kernel: [<c01446c6>] do_select [kernel] 0x226 Apr 3 04:06:40 server4 kernel: [sys_select+1138/1152] sys_select [kernel] 0x472 Apr 3 04:06:40 server4 kernel: [<c0144b82>] sys_select [kernel] 0x472 Apr 3 04:06:40 server4 kernel: [sys_socketcall+176/512] sys_socketcall [kernel] 0xb0 Apr 3 04:06:40 server4 kernel: [<c01bd3e0>] sys_socketcall [kernel] 0xb0 Apr 3 04:06:40 server4 kernel: [do_IRQ+156/176] do_IRQ [kernel] 0x9c Apr 3 04:06:40 server4 kernel: [<c010846c>] do_IRQ [kernel] 0x9c Apr 3 04:06:40 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Apr 3 04:06:40 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:15:03 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:15:03 server4 kernel: printing eip: Apr 3 04:15:03 server4 kernel: c012d1b1 Apr 3 04:15:03 server4 kernel: *pde = 00000000 Apr 3 04:15:03 server4 kernel: Oops: 0002 Apr 3 04:15:03 server4 kernel: Kernel 2.4.9-31 Apr 3 04:15:03 server4 kernel: CPU: 0 Apr 3 04:15:03 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:15:03 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:15:03 server4 kernel: EFLAGS: 00210046 Apr 3 04:15:03 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:15:03 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:15:03 server4 kernel: esi: 00200282 edi: c0057740 ebp: c03611c0 esp: c4305f08 Apr 3 04:15:03 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:15:03 server4 kernel: Process superquadrics (pid: 19810, stackpage=c43 05000) Apr 3 04:15:03 server4 kernel: Stack: c01145ea d88785c0 d826a260 404b70d0 c1c86 1a0 00000004 ffffff9f c0149b42 Apr 3 04:15:03 server4 kernel: c1c861a0 000000f0 00000001 c01bbd36 00000 001 c01bc675 d826a6a0 d826a6a0 Apr 3 04:15:03 server4 kernel: d88785c0 00001000 00000000 40018000 00001 000 c1c87d50 00000001 40393c08 Apr 3 04:15:03 server4 kernel: Call Trace: [do_page_fault+378/1168] do_page_fau lt [kernel] 0x17a Apr 3 04:15:03 server4 kernel: Call Trace: [<c01145ea>] do_page_fault [kernel] 0x17a Apr 3 04:15:03 server4 kernel: [get_empty_inode+18/128] get_empty_inode [kernel ] 0x12 Apr 3 04:15:03 server4 kernel: [<c0149b42>] get_empty_inode [kernel] 0x12 Apr 3 04:15:03 server4 kernel: [sock_alloc+6/176] sock_alloc [kernel] 0x6 Apr 3 04:15:03 server4 kernel: [<c01bbd36>] sock_alloc [kernel] 0x6 Apr 3 04:15:03 server4 kernel: [sock_create+165/256] sock_create [kernel] 0xa5 Apr 3 04:15:03 server4 kernel: [<c01bc675>] sock_create [kernel] 0xa5 Apr 3 04:15:03 server4 kernel: [sys_socket+27/80] sys_socket [kernel] 0x1b Apr 3 04:15:03 server4 kernel: [<c01bc6eb>] sys_socket [kernel] 0x1b Apr 3 04:15:03 server4 kernel: [sys_socketcall+98/512] sys_socketcall [kernel] 0x62 Apr 3 04:15:03 server4 kernel: [<c01bd392>] sys_socketcall [kernel] 0x62 Apr 3 04:15:03 server4 kernel: [do_page_fault+0/1168] do_page_fault [kernel] 0x 0 Apr 3 04:15:03 server4 kernel: [<c0114470>] do_page_fault [kernel] 0x0 Apr 3 04:15:03 server4 kernel: [error_code+56/64] error_code [kernel] 0x38 Apr 3 04:15:03 server4 kernel: [<c0107038>] error_code [kernel] 0x38 Apr 3 04:15:03 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:15:03 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:15:03 server4 kernel: Apr 3 04:15:03 server4 kernel: Apr 3 04:15:03 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:15:41 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:15:41 server4 kernel: printing eip: Apr 3 04:15:41 server4 kernel: c012d1b1 Apr 3 04:15:41 server4 kernel: *pde = 00000000 Apr 3 04:15:41 server4 kernel: Oops: 0002 Apr 3 04:15:41 server4 kernel: Kernel 2.4.9-31 Apr 3 04:15:41 server4 kernel: CPU: 0 Apr 3 04:15:41 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:15:41 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:15:41 server4 kernel: EFLAGS: 00010046 Apr 3 04:15:41 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:15:41 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:15:41 server4 kernel: esi: 00000286 edi: c0057580 ebp: 0000000b esp: d3a2be30 Apr 3 04:15:41 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:15:41 server4 kernel: Process top (pid: 19121, stackpage=d3a2b000) Apr 3 04:15:41 server4 kernel: Stack: 39383130 00000031 d3a2bef0 00000008 c1c86 1a0 c188a000 c1c83400 c0149b42 Apr 3 04:15:41 server4 kernel: c1c861a0 000000f0 c0233696 c01532a5 c0233 696 cb6d3e04 c023369a c02b4280 Apr 3 04:15:41 server4 kernel: c0153566 c1c83400 c188a000 0000000b c188a 000 ffffffea c5662c80 c01480f9 Apr 3 04:15:41 server4 kernel: Call Trace: [get_empty_inode+18/128] get_empty_i node [kernel] 0x12 Apr 3 04:15:41 server4 kernel: Call Trace: [<c0149b42>] get_empty_inode [kernel ] 0x12 Apr 3 04:15:41 server4 kernel: [IRQ0x0f_interrupt+134902/138304] .rodata.str1.1 [kernel] 0x77d1 Apr 3 04:15:41 server4 kernel: [<c0233696>] .rodata.str1.1 [kernel] 0x77d1 Apr 3 04:15:41 server4 kernel: [proc_pid_make_inode+21/176] proc_pid_make_inode [kernel] 0x15 Apr 3 04:15:41 server4 kernel: [<c01532a5>] proc_pid_make_inode [kernel] 0x15 Apr 3 04:15:41 server4 kernel: [IRQ0x0f_interrupt+134902/138304] .rodata.str1.1 [kernel] 0x77d1 Apr 3 04:15:41 server4 kernel: [<c0233696>] .rodata.str1.1 [kernel] 0x77d1 Apr 3 04:15:41 server4 kernel: [IRQ0x0f_interrupt+134906/138304] .rodata.str1.1 [kernel] 0x77d5 Apr 3 04:15:41 server4 kernel: [<c023369a>] .rodata.str1.1 [kernel] 0x77d5 Apr 3 04:15:41 server4 kernel: [proc_base_lookup+134/528] proc_base_lookup [ker nel] 0x86 Apr 3 04:15:41 server4 kernel: [<c0153566>] proc_base_lookup [kernel] 0x86 Apr 3 04:15:41 server4 kernel: [d_alloc+25/368] d_alloc [kernel] 0x19 Apr 3 04:15:41 server4 kernel: [<c01480f9>] d_alloc [kernel] 0x19 Apr 3 04:15:41 server4 kernel: [real_lookup+79/192] real_lookup [kernel] 0x4f Apr 3 04:15:41 server4 kernel: [<c013fc7f>] real_lookup [kernel] 0x4f Apr 3 04:15:41 server4 kernel: [path_walk+1462/2064] path_walk [kernel] 0x5b6 Apr 3 04:15:41 server4 kernel: [<c01403a6>] path_walk [kernel] 0x5b6 Apr 3 04:15:41 server4 kernel: [dput+28/368] dput [kernel] 0x1c Apr 3 04:15:41 server4 kernel: [<c014796c>] dput [kernel] 0x1c Apr 3 04:15:41 server4 kernel: [open_namei+140/1488] open_namei [kernel] 0x8c Apr 3 04:15:41 server4 kernel: [<c0140c6c>] open_namei [kernel] 0x8c Apr 3 04:15:41 server4 kernel: [filp_open+54/96] filp_open [kernel] 0x36 Apr 3 04:15:41 server4 kernel: [<c0135fb6>] filp_open [kernel] 0x36 Apr 3 04:15:41 server4 kernel: [getname+94/160] getname [kernel] 0x5e Apr 3 04:15:41 server4 kernel: [<c013f9ce>] getname [kernel] 0x5e Apr 3 04:15:41 server4 kernel: [sys_open+54/176] sys_open [kernel] 0x36 Apr 3 04:15:41 server4 kernel: [<c01362a6>] sys_open [kernel] 0x36 Apr 3 04:15:41 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:15:41 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:15:41 server4 kernel: Apr 3 04:15:41 server4 kernel: Apr 3 04:15:41 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:16:29 server4 kernel: <1>Unable to handle kernel NULL pointer derefer ence at virtual address 00000004 Apr 3 04:16:29 server4 kernel: printing eip: Apr 3 04:16:29 server4 kernel: c012d1b1 Apr 3 04:16:29 server4 kernel: *pde = 00000000 Apr 3 04:16:29 server4 kernel: Oops: 0002 Apr 3 04:16:29 server4 kernel: Kernel 2.4.9-31 Apr 3 04:16:29 server4 kernel: CPU: 0 Apr 3 04:16:29 server4 kernel: EIP: 0010:[kmem_cache_alloc+113/192] Not t ainted Apr 3 04:16:29 server4 kernel: EIP: 0010:[<c012d1b1>] Not tainted Apr 3 04:16:29 server4 kernel: EFLAGS: 00010046 Apr 3 04:16:29 server4 kernel: EIP is at kmem_cache_alloc [kernel] 0x71 Apr 3 04:16:29 server4 kernel: eax: 00000000 ebx: c1c861a0 ecx: c0057000 edx: c1c861b0 Apr 3 04:16:29 server4 kernel: esi: 00000286 edi: c00573c0 ebp: c1c83400 esp: d6cafdc0 Apr 3 04:16:29 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:16:29 server4 kernel: Process cpumemusage_app (pid: 1471, stackpage=d6 caf000) Apr 3 04:16:29 server4 kernel: Stack: 56555453 5a595857 33323100 37363534 c1c86 1a0 dffaee18 dffaee18 c0149bd0 Apr 3 04:16:29 server4 kernel: c1c861a0 000000f0 6e6d6c6b 7271706f 76757 473 7a797877 00000000 dffaee18 Apr 3 04:16:29 server4 kernel: 00001007 c1c83400 c0149f02 c1c83400 00001 007 dffaee18 00000000 00000000 Apr 3 04:16:29 server4 kernel: Call Trace: [get_new_inode+32/384] get_new_inode [kernel] 0x20 Apr 3 04:16:29 server4 kernel: Call Trace: [<c0149bd0>] get_new_inode [kernel] 0x20 Apr 3 04:16:29 server4 kernel: [iget4+210/224] iget4 [kernel] 0xd2 Apr 3 04:16:29 server4 kernel: [<c0149f02>] iget4 [kernel] 0xd2 Apr 3 04:16:29 server4 kernel: [proc_get_inode+64/256] proc_get_inode [kernel] 0x40 Apr 3 04:16:29 server4 kernel: [<c01523a0>] proc_get_inode [kernel] 0x40 Apr 3 04:16:29 server4 kernel: [proc_lookup+162/192] proc_lookup [kernel] 0xa2 Apr 3 04:16:29 server4 kernel: [<c0153f72>] proc_lookup [kernel] 0xa2 Apr 3 04:16:29 server4 kernel: [proc_root_lookup+42/80] proc_root_lookup [kerne l] 0x2a Apr 3 04:16:29 server4 kernel: [<c015253a>] proc_root_lookup [kernel] 0x2a Apr 3 04:16:29 server4 kernel: [real_lookup+79/192] real_lookup [kernel] 0x4f Apr 3 04:16:29 server4 kernel: [<c013fc7f>] real_lookup [kernel] 0x4f Apr 3 04:16:29 server4 kernel: [path_walk+1462/2064] path_walk [kernel] 0x5b6 Apr 3 04:16:29 server4 kernel: [<c01403a6>] path_walk [kernel] 0x5b6 Apr 3 04:16:29 server4 kernel: [schedule+612/960] schedule [kernel] 0x264 Apr 3 04:16:29 server4 kernel: [<c01150f4>] schedule [kernel] 0x264 Apr 3 04:16:29 server4 kernel: [open_namei+140/1488] open_namei [kernel] 0x8c Apr 3 04:16:29 server4 kernel: [<c0140c6c>] open_namei [kernel] 0x8c Apr 3 04:16:29 server4 kernel: [filp_open+54/96] filp_open [kernel] 0x36 Apr 3 04:16:29 server4 kernel: [<c0135fb6>] filp_open [kernel] 0x36 Apr 3 04:16:29 server4 kernel: [getname+94/160] getname [kernel] 0x5e Apr 3 04:16:29 server4 kernel: [<c013f9ce>] getname [kernel] 0x5e Apr 3 04:16:29 server4 kernel: [sys_open+54/176] sys_open [kernel] 0x36 Apr 3 04:16:29 server4 kernel: [<c01362a6>] sys_open [kernel] 0x36 Apr 3 04:16:29 server4 kernel: [system_call+51/56] system_call [kernel] 0x33 Apr 3 04:16:29 server4 kernel: [<c0106f3b>] system_call [kernel] 0x33 Apr 3 04:16:29 server4 kernel: Apr 3 04:16:29 server4 kernel: Apr 3 04:16:29 server4 kernel: Code: 89 50 04 89 02 8d 53 08 c7 41 04 00 00 00 00 c7 01 00 00 00 Apr 3 04:42:40 server4 kernel: Slab inode_cache is still in use! Apr 3 04:42:40 server4 kernel: ------------[ cut here ]------------ Apr 3 04:42:40 server4 kernel: kernel BUG at slab.c:933! Apr 3 04:42:40 server4 kernel: invalid operand: 0000 Apr 3 04:42:40 server4 kernel: Kernel 2.4.9-31 Apr 3 04:42:40 server4 kernel: CPU: 0 Apr 3 04:42:40 server4 kernel: EIP: 0010:[__kmem_cache_shrink+70/176] Not tainted Apr 3 04:42:40 server4 kernel: EIP: 0010:[<c012cd66>] Not tainted Apr 3 04:42:40 server4 kernel: EFLAGS: 00010082 Apr 3 04:42:40 server4 kernel: EIP is at __kmem_cache_shrink [kernel] 0x46 Apr 3 04:42:40 server4 kernel: eax: 0000001a ebx: c037b000 ecx: 00000001 edx: 0001b8d0 Apr 3 04:42:40 server4 kernel: esi: c1c861a0 edi: c080e000 ebp: 000000d2 esp: c080fe04 Apr 3 04:42:40 server4 kernel: ds: 0018 es: 0018 ss: 0018 Apr 3 04:42:40 server4 kernel: Process rhn_check (pid: 19813, stackpage=c080f00 0) Apr 3 04:42:40 server4 kernel: Stack: c022e091 000003a5 0000b970 000000d2 00000 000 c012f513 c1c861a0 000000d2 Apr 3 04:42:40 server4 kernel: c1c86130 000000d2 000000d2 00000001 c080e 000 00000001 c012f688 000000d2 Apr 3 04:42:40 server4 kernel: 00000001 c080e000 c01302de 000000d2 00000 001 000000d2 d1b28ae0 00000000 Apr 3 04:42:40 server4 kernel: Call Trace: [IRQ0x0f_interrupt+112881/138304] .r odata.str1.1 [kernel] 0x21cc Apr 3 04:42:40 server4 kernel: Call Trace: [<c022e091>] .rodata.str1.1 [kernel] 0x21cc Apr 3 04:42:40 server4 kernel: [do_try_to_free_pages+35/80] do_try_to_free_page s [kernel] 0x23 Apr 3 04:42:40 server4 kernel: [<c012f513>] do_try_to_free_pages [kernel] 0x23 Apr 3 04:42:40 server4 kernel: [try_to_free_pages+40/64] try_to_free_pages [ker nel] 0x28 Apr 3 04:42:40 server4 kernel: [<c012f688>] try_to_free_pages [kernel] 0x28 Apr 3 04:42:40 server4 kernel: [_wrapped_alloc_pages+446/608] _wrapped_alloc_pa ges [kernel] 0x1be @ Apr 3 04:42:40 server4 kernel: [<c01302de>] _wrapped_alloc_pages [kernel] 0x1be Apr 3 04:42:40 server4 kernel: [__alloc_pages+15/160] __alloc_pages [kernel] 0x f Apr 3 04:42:40 server4 kernel: [<c013038f>] __alloc_pages [kernel] 0xf Apr 3 04:42:40 server4 kernel: [do_anonymous_page+58/224] do_anonymous_page [ke rnel] 0x3a Apr 3 04:42:40 server4 kernel: [<c0125c9a>] do_anonymous_page [kernel] 0x3a Apr 3 04:42:40 server4 kernel: [do_no_page+50/272] do_no_page [kernel] 0x32 Apr 3 04:42:40 server4 kernel: [<c0125d72>] do_no_page [kernel] 0x32 Apr 3 04:42:40 server4 kernel: [handle_mm_fault+101/224] handle_mm_fault [kerne l] 0x65 Apr 3 04:42:40 server4 kernel: [<c0125eb5>] handle_mm_fault [kernel] 0x65 Apr 3 04:42:40 server4 kernel: [ip_local_deliver+298/448] ip_local_deliver [ker nel] 0x12a Apr 3 04:42:40 server4 kernel: [<c01d1a4a>] ip_local_deliver [kernel] 0x12a Apr 3 04:42:40 server4 kernel: [do_page_fault+0/1168] do_page_fault [kernel] 0x 0 Apr 3 04:42:40 server4 kernel: [<c0114470>] do_page_fault [kernel] 0x0 Apr 3 04:42:40 server4 kernel: [do_page_fault+378/1168] do_page_fault [kernel] 0x17a Apr 3 04:42:40 server4 kernel: [<c01145ea>] do_page_fault [kernel] 0x17a Apr 3 04:42:40 server4 kernel: [ip_rcv+803/912] ip_rcv [kernel] 0x323 Apr 3 04:42:40 server4 kernel: [<c01d1e03>] ip_rcv [kernel] 0x323 Apr 3 04:42:40 server4 kernel: [do_munmap+100/608] do_munmap [kernel] 0x64 Apr 3 04:42:40 server4 kernel: [<c0126dc4>] do_munmap [kernel] 0x64 Apr 3 04:42:40 server4 kernel: [net_rx_action+426/624] net_rx_action [kernel] 0 x1aa Apr 3 04:42:40 server4 kernel: [<c01c30fa>] net_rx_action [kernel] 0x1aa Apr 3 04:42:40 server4 kernel: [do_brk+180/352] do_brk [kernel] 0xb4 Apr 3 04:42:40 server4 kernel: [<c01270c4>] do_brk [kernel] 0xb4 Apr 3 04:42:40 server4 kernel: [sys_brk+169/224] sys_brk [kernel] 0xa9 Apr 3 04:42:40 server4 kernel: [<c01261f9>] sys_brk [kernel] 0xa9 Apr 3 04:42:40 server4 kernel: [do_page_fault+0/1168] do_page_fault [kernel] 0x 0 Apr 3 04:42:40 server4 kernel: [<c0114470>] do_page_fault [kernel] 0x0 Apr 3 04:42:40 server4 kernel: [error_code+56/64] error_code [kernel] 0x38 Apr 3 04:42:40 server4 kernel: [<c0107038>] error_code [kernel] 0x38 Apr 3 04:42:40 server4 kernel: Apr 3 04:42:40 server4 kernel: @ Apr 3 04:42:40 server4 kernel: Code: 0f 0b 58 5a 8b 53 04 8b 03 89 50 04 89 02 Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Run system for a few hours under load 2. A heavy NFS load seems to do it 3. Actual Results: System Locked Up... no response even to ctl-alt-del Expected Results: uh... a running system I will supply any info needed... just Email. Additional info: Uname -a =Linux server4 2.4.9-31 #1 Tue Feb 26 07:11:02 EST 2002 i686 unknown [root@server4 /proc]# more cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 3 model name : AMD Athlon(tm) Processor stepping : 0 cpu MHz : 655.856 cache size : 64 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat p se36 mmx fxsr syscall mmxext 3dnowext 3dnow bogomips : 1307.44
ok ... got this kinda figured out.... has to do with "heavy" disk IO load and having both disks on the same IDE channel. Once I seperated the disks from (hda,hdb) to (hda,hdd) the problem is eliminated. So its hardware interaction... between the IDE channel and DMA is my guess.
This ends up being a hardware error.... memory !! Thank you.