86487 – "ps" hangs in proc_info_read->alloc_inode

Bug 86487 - "ps" hangs in proc_info_read->alloc_inode

Summary: "ps" hangs in proc_info_read->alloc_inode

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Linux
Classification:	Retired
Component:	kernel
Sub Component:
Version:	8.0
Hardware:	athlon
OS:	Linux
Priority:	medium
Severity:	high
Target Milestone:	---
Assignee:	Arjan van de Ven
QA Contact:	Brian Brock
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2003-03-24 04:05 UTC by Konstantin Olchanski
Modified:	2007-04-18 16:52 UTC (History)
CC List:	0 users
Fixed In Version:
Clone Of:
Environment:
Last Closed:	2004-09-30 15:40:41 UTC
Embargoed:

Attachments	(Terms of Use)

Description Konstantin Olchanski 2003-03-24 04:05:36 UTC

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030301

Description of problem:
"ps", "top" and others hang on reads from /proc/PID/stat. The hung processes
cannot be killed and the system eventually becomes unusable and has to be reset.

This hung happens rarely, less than 1 per month per 20 systems. I have seen it
with kernels older than 2.4.18-19.8.0, never with vanilla 2.4.20 and (never?)
with redhat-7.x systems. I will be updating the currently hung system to
kernel-smp-2.4.18-27.8.0.athlon.rpm and will report if the hang reappears.

I am attaching the output of "strace ps -ef" and "alt-sysrq-t". Note that "ps"
is hung on a read() from "/proc/30731/stat" and the kernel stack trace shows it
waiting on a lock inside sys_read()->proc_info_read()->alloc_inide(). Please let
me know if you need any additional information from the hung systems.

K.O. olchansk


Version-Release number of selected component (if applicable):
kernel-smp-2.4.18-19.8.0smp

How reproducible:
Sometimes

Steps to Reproduce:
1. blah...
2.
3.
    

Additional info:

Comment 1 Konstantin Olchanski 2003-03-24 04:06:36 UTC

Output from "strace ps -ef":

stat64("/proc/29709", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/29709/stat", O_RDONLY)      = 7
read(7, "29709 (onx.exe) S 13281 29709 13"..., 511) = 206
close(7)                                = 0
open("/proc/29709/statm", O_RDONLY)     = 7
read(7, "5414 133 69 0 0 133 64\n", 511) = 23
close(7)                                = 0
open("/proc/29709/status", O_RDONLY)    = 7
read(7, "Name:\tonx.exe\nState:\tS (sleeping"..., 511) = 466
close(7)                                = 0
open("/proc/29709/cmdline", O_RDONLY)   = 7
read(7, "/OSC/OnX/v12r0/Linux-i686/onx.ex"..., 2047) = 34
close(7)                                = 0
open("/proc/29709/environ", O_RDONLY)   = -1 EACCES (Permission denied)
stat64("/proc/29769", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/29769/stat", O_RDONLY)      = 7
read(7, "29769 (kdesktop_lock) S 13237 13"..., 511) = 202
close(7)                                = 0
open("/proc/29769/statm", O_RDONLY)     = 7
read(7, "660 240 240 0 0 240 0\n", 511) = 22
close(7)                                = 0
open("/proc/29769/status", O_RDONLY)    = 7
read(7, "Name:\tkdesktop_lock\nState:\tS (sl"..., 511) = 472
close(7)                                = 0
open("/proc/29769/cmdline", O_RDONLY)   = 7
read(7, "/usr/bin/kdesktop_lock\0--forcelo"..., 2047) = 35
close(7)                                = 0
open("/proc/29769/environ", O_RDONLY)   = -1 EACCES (Permission denied)
stat64("/proc/29770", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/29770/stat", O_RDONLY)      = 7
read(7, "29770 (kbanner.kss) S 29769 1318"..., 511) = 208
close(7)                                = 0
open("/proc/29770/statm", O_RDONLY)     = 7
read(7, "709 322 249 4 0 318 73\n", 511) = 23
close(7)                                = 0
open("/proc/29770/status", O_RDONLY)    = 7
read(7, "Name:\tkbanner.kss\nState:\tS (slee"..., 511) = 470
close(7)                                = 0
open("/proc/29770/cmdline", O_RDONLY)   = 7
read(7, "kbanner.kss\0-window-id\0004194308\0", 2047) = 31
close(7)                                = 0
open("/proc/29770/environ", O_RDONLY)   = -1 EACCES (Permission denied)
stat64("/proc/20402", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/20402/stat", O_RDONLY)      = 7
read(7, "20402 (mozilla-bin) S 10267 1318"..., 511) = 197
close(7)                                = 0
open("/proc/20402/statm", O_RDONLY)     = 7
read(7, "5790 4135 3120 6 0 4129 1935\n", 511) = 29
close(7)                                = 0
open("/proc/20402/status", O_RDONLY)    = 7
read(7, "Name:\tmozilla-bin\nState:\tS (slee"..., 511) = 471
close(7)                                = 0
open("/proc/20402/cmdline", O_RDONLY)   = 7
read(7, "/usr/lib/mozilla-1.0.1/mozilla-b"..., 2047) = 35
close(7)                                = 0
open("/proc/20402/environ", O_RDONLY)   = -1 EACCES (Permission denied)
stat64("/proc/30731", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/30731/stat", O_RDONLY)      = 7
read(7, 
 <unfinished ...>

Comment 2 Konstantin Olchanski 2003-03-24 04:07:20 UTC

Output from alt-sysrq-t:

Mar 23 19:22:35 dork kernel: SysRq : Show State
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel:                          free                     
  sibling
Mar 23 19:22:35 dork kernel:   task             PC    stack   pid father child
younger older
Mar 23 19:22:35 dork kernel: init          S F7FEAE00     0     1      0 18266 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xc36b7eec))
Mar 23 19:22:35 dork kernel: t [kernel] 0xad (0xf17afedc))
Mar 23 19:22:35 dork kernel: [<c01339cb>] handle_mm_fault [kernel] 0x10b
(0xf17afee0))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xf17aff14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xf17aff74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf17aff94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf17affc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S C68E1500     0 13443  13260       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xcccb1edc))
Mar 23 19:22:35 dork kernel: [<c01339cb>] handle_mm_fault [kernel] 0x10b
(0xcccb1ee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xcccb1efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xcccb1f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xcccb1f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xcccb1f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xcccb1fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S C705FD80     4 13444  13257       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xcb865edc))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xcb865ee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xcb865efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xcb865f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xcb865f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xcb865f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xcb865fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S C7D8F0C0     0 13445  13262       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xe9d71edc))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xe9d71ee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xe9d71efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xe9d71f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xe9d71f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xe9d71f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe9d71fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S F5B6E500     4 13453  13248       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xf0335edc))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xf0335ee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xf0335efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xf0335f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xf0335f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf0335f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf0335fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S F7764780     4 13457  13250       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xd9c39edc))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xd9c39ee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xd9c39efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xd9c39f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xd9c39f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd9c39f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd9c39fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: kdeinit       S E46913C0     4 14308  13186 14310 
 10258 13267 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xe9dcbeec))
Mar 23 19:22:35 dork kernel: [<c018bf55>] tty_poll [kernel] 0x85 (0xe9dcbf0c))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xe9dcbf24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xe9dcbf60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe9dcbfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S C03AC880     0 14310  14308       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c01905c9>] change_termios [kernel]
0x1a9 (0xda8b9ec8))
Mar 23 19:22:35 dork kernel: [<c012b0cd>] schedule_timeout [kernel] 0xad
(0xda8b9edc))
Mar 23 19:22:35 dork kernel: [<c01339cb>] handle_mm_fault [kernel] 0x10b
(0xda8b9ee0))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xda8b9f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xda8b9f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xda8b9f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xda8b9fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S C03AC880     0 10258  13186 10267 
       14308 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xe4b9bf0c))
Mar 23 19:22:35 dork kernel: [<c0155917>] pipe_poll [kernel] 0x37 (0xe4b9bf10))
Mar 23 19:22:35 dork kernel: [<c015d21b>] do_pollfd [kernel] 0x5b (0xe4b9bf28))
Mar 23 19:22:35 dork kernel: [<c015d326>] do_pollfd [kernel] 0x166 (0xe4b9bf44))
Mar 23 19:22:35 dork kernel: [<c015d4e3>] sys_poll [kernel] 0x163 (0xe4b9bf78))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe4b9bfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S C03AD240     0 10267  10258 15990 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xc51a9f0c))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xc51a9f2c))
Mar 23 19:22:35 dork kernel: [<c015d326>] do_pollfd [kernel] 0x166 (0xc51a9f44))
Mar 23 19:22:35 dork kernel: [<c015d4e3>] sys_poll [kernel] 0x163 (0xc51a9f78))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xc51a9fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S C03AD240  2384 10268  10267       
 10269       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xee2bdf0c))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xee2bdf2c))
Mar 23 19:22:35 dork kernel: [<c015d326>] do_pollfd [kernel] 0x166 (0xee2bdf44))
Mar 23 19:22:35 dork kernel: [<c015d4e3>] sys_poll [kernel] 0x163 (0xee2bdf78))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xee2bdfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S F5AD5C00  2384 10269  10267       
 10271 10268 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c01082db>] sys_rt_sigsuspend [kernel]
0xeb (0xeec67f98))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xeec67fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S F5AD5C00  2384 10271  10267       
 20402 10269 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xd8d83f64))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xd8d83f84))
Mar 23 19:22:35 dork kernel: [<c012b1b1>] sys_nanosleep [kernel] 0xd1 (0xd8d83f9c))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd8d83fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: onx.exe       S C03AD240     0 29709  13281       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xd4b07eec))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xd4b07f0c))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xd4b07f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xd4b07f60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd4b07fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: kdesktop_lock S C03AC880     0 29769  13237 29770 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xccd4beec))
Mar 23 19:22:35 dork kernel: [<c024fa6e>] unix_poll [kernel] 0x2e (0xccd4bf00))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xccd4bf24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xccd4bf60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xccd4bfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: kbanner.kss   S C03AC880     0 29770  29769       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xd9b73eec))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xd9b73f0c))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xd9b73f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xd9b73f60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd9b73fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S C03AC880    48 20402  10267       
 15990 10271 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c01082db>] sys_rt_sigsuspend [kernel]
0xeb (0xe7cb7f98))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe7cb7fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mofia         D F5DF1D80  3276 30731      1       
 31123 13275 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0136d99>] __lock_page [kernel] 0x99
(0xe5263958))
Mar 23 19:22:35 dork kernel: [<c0136dec>] lock_page [kernel] 0x1c (0xe5263980))
Mar 23 19:22:35 dork kernel: [<c013864e>] filemap_nopage [kernel] 0x11e
(0xe5263988))
Mar 23 19:22:35 dork kernel: [<c01336b0>] do_no_page [kernel] 0x80 (0xe52639b8))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xe52639f8))
Mar 23 19:22:35 dork kernel: [<c01324bd>] get_user_pages [kernel] 0xdd (0xe5263a24))
Mar 23 19:22:35 dork kernel: [<c016e7e0>] elf_core_dump [kernel] 0x950 (0xe5263a54))
Mar 23 19:22:35 dork kernel: [<c0154fb0>] do_coredump [kernel] 0x180 (0xe5263ed4))
Mar 23 19:22:35 dork kernel: [<c012bc14>] dequeue_signal [kernel] 0x64 (0xe5263efc))
Mar 23 19:22:35 dork kernel: [<c01091d1>] do_signal [kernel] 0x251 (0xe5263f14))
Mar 23 19:22:35 dork kernel: [<c0126b54>] bh_action [kernel] 0x54 (0xe5263f64))
Mar 23 19:22:35 dork kernel: [<c01269f7>] tasklet_hi_action [kernel] 0x67
(0xe5263f6c))
Mar 23 19:22:35 dork kernel: [<c012adbe>] update_process_times [kernel] 0x3e
(0xe5263f8c))
Mar 23 19:22:35 dork kernel: [<c011b6f0>] do_page_fault [kernel] 0x0 (0xe5263fb8))
Mar 23 19:22:35 dork kernel: [<c0109498>] signal_return [kernel] 0x14 (0xe5263fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mofia         D C8791680  3508 31123      1       
  5555 30731 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0136d99>] __lock_page [kernel] 0x99
(0xf033d958))
Mar 23 19:22:35 dork kernel: [<c0136dec>] lock_page [kernel] 0x1c (0xf033d980))
Mar 23 19:22:35 dork kernel: [<c013864e>] filemap_nopage [kernel] 0x11e
(0xf033d988))
Mar 23 19:22:35 dork kernel: [<c01336b0>] do_no_page [kernel] 0x80 (0xf033d9b8))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xf033d9f8))
Mar 23 19:22:35 dork kernel: [<c01324bd>] get_user_pages [kernel] 0xdd (0xf033da24))
Mar 23 19:22:35 dork kernel: [<c016e7e0>] elf_core_dump [kernel] 0x950 (0xf033da54))
Mar 23 19:22:35 dork kernel: [<c0154fb0>] do_coredump [kernel] 0x180 (0xf033ded4))
Mar 23 19:22:35 dork kernel: [<c012bc14>] dequeue_signal [kernel] 0x64 (0xf033defc))
Mar 23 19:22:35 dork kernel: [<c01091d1>] do_signal [kernel] 0x251 (0xf033df14))
Mar 23 19:22:35 dork kernel: [<c018ae1d>] tty_write [kernel] 0xfd (0xf033df74))
Mar 23 19:22:35 dork kernel: [<c018f420>] write_chan [kernel] 0x0 (0xf033df78))
Mar 23 19:22:35 dork kernel: [<c014a429>] sys_write [kernel] 0x109 (0xf033df94))
Mar 23 19:22:35 dork kernel: [<c011b6f0>] do_page_fault [kernel] 0x0 (0xf033dfb8))
Mar 23 19:22:35 dork kernel: [<c0109498>] signal_return [kernel] 0x14 (0xf033dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AD240    16 31206   1011 31210 
 17566       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xd73c9e54))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xd73c9e60))
Mar 23 19:22:35 dork kernel: [<c024f35f>] unix_stream_data_wait [kernel] 0xcf
(0xd73c9e8c))
Mar 23 19:22:35 dork kernel: [<c024f7a1>] unix_stream_recvmsg [kernel] 0x3f1
(0xd73c9ec4))
Mar 23 19:22:35 dork kernel: [<c01f8548>] sock_recvmsg [kernel] 0x58 (0xd73c9f08))
Mar 23 19:22:35 dork kernel: [<c01490da>] dentry_open [kernel] 0x15a (0xd73c9f3c))
Mar 23 19:22:35 dork kernel: [<c01f8687>] sock_read [kernel] 0xa7 (0xd73c9f50))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd73c9f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd73c9fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S F5B6E200    16 31210  31206 31213 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xc67e7eec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xc67e7ef4))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xc67e7f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xc67e7f60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xc67e7fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: tcsh          S F5B6EBC0     0 31213  31210  5117 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xd9f0bedc))
Mar 23 19:22:35 dork kernel: [<c01339cb>] handle_mm_fault [kernel] 0x10b
(0xd9f0bee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xd9f0befc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xd9f0bf14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xd9f0bf74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd9f0bf94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd9f0bfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: emacs         S C03AC880  2384   814  31213       
  5071       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xca221eec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xca221ef4))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xca221f0c))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xca221f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xca221f60))
Mar 23 19:22:35 dork kernel: [<c0108588>] restore_sigcontext [kernel] 0x128
(0xca221f7c))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xca221fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: emacs         S C03AC880  2384  5071  31213       
  5117   814 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xc8027eec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xc8027ef4))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xc8027f0c))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xc8027f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xc8027f60))
Mar 23 19:22:35 dork kernel: [<c0108588>] restore_sigcontext [kernel] 0x128
(0xc8027f7c))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xc8027fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: emacs         S C03AD240  2388  5117  31213       
        5071 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xdeb8feec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xdeb8fef4))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xdeb8ff0c))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xdeb8ff24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xdeb8ff60))
Mar 23 19:22:35 dork kernel: [<c0108588>] restore_sigcontext [kernel] 0x128
(0xdeb8ff7c))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdeb8ffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: top           D C03AC880     0  5504      1       
  5519  5655 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xc8f5de44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xc8f5de58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xc8f5de7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xc8f5de88))
Mar 23 19:22:35 dork kernel: [<c0163257>] new_inode [kernel] 0x17 (0xc8f5dea4))
Mar 23 19:22:35 dork kernel: [<c01702d8>] proc_pid_make_inode [kernel] 0x88
(0xc8f5deac))
Mar 23 19:22:35 dork kernel: [<c0170621>] proc_base_lookup [kernel] 0x111
(0xc8f5dec0))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xc8f5def8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xc8f5df0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xc8f5df3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xc8f5df6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xc8f5df94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xc8f5dfa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xc8f5dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AC880     4  5519      1       
  9965  5504 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xe883de44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xe883de58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xe883de7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xe883de88))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xe883deb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xe883decc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xe883def8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xe883df0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xe883df3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xe883df6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xe883df94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xe883dfa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe883dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: top           D C03AC880     0  5555      1       
  5598 31123 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xce17fe44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xce17fe58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xce17fe7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xce17fe88))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xce17feb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xce17fecc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xce17fef8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xce17ff0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xce17ff3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xce17ff6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xce17ff94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xce17ffa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xce17ffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AD240     4  5598      1       
  5655  5555 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xc656de44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xc656de58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xc656de7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xc656de88))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xc656deb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xc656decc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xc656def8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xc656df0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xc656df3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xc656df6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xc656df94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xc656dfa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xc656dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AC880     0  5655      1       
  5504  5598 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xf7bafe44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xf7bafe58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xf7bafe7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xf7bafe88))
Mar 23 19:22:35 dork kernel: [<c012c705>] wake_up_parent [kernel] 0x25 (0xf7bafefc))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xf7baff3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xf7baff6c))
Mar 23 19:22:35 dork kernel: [<c011d0c6>] schedule [kernel] 0x186 (0xf7baff7c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf7baff94))
Mar 23 19:22:35 dork kernel: [<c010dc31>] syscall_trace [kernel] 0x51 (0xf7baffac))
Mar 23 19:22:35 dork kernel: [<c01094cb>] tracesys [kernel] 0x1f (0xf7baffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AC880  2384  9965      1       
 17515  5519 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xcc323e44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xcc323e58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xcc323e7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xcc323e88))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xcc323eb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xcc323ecc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xcc323ef8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xcc323f0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xcc323f3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xcc323f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xcc323f94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xcc323fa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xcc323fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: crond         S F5DF1480    16 13845   1129 13849 
  9899       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xf0fcdf38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xf0fcdf6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf0fcdf94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf0fcdfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: triumf-up2dat S F5DF1780     0 13849  13845 13864 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0124f37>] sys_wait4 [kernel] 0x117
(0xd63fbf80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd63fbfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash2         S C36CC500     0 13863  13849 13869 
 13864       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xf75bdf38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xf75bdf6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf75bdf94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf75bdfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: triumf-up2dat S F5DF1180     0 13864  13849 13867 
       13863 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xf0423f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xf0423f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf0423f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf0423fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: cat           S F5B6E080     0 13867  13864       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xda7f9f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xda7f9f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xda7f9f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xda7f9fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash2         S F5B6E2C0     0 13869  13863 13873 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0124f37>] sys_wait4 [kernel] 0x117
(0xd6a87f80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd6a87fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AD240     0 13870  13869       
 13871       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xd47c1e44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xd47c1e58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xd47c1e7c))
Mar 23 19:22:35 dork kernel: [<c0141273>] __alloc_pages [kernel] 0x93 (0xd47c1e98))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xd47c1eb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xd47c1ecc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xd47c1ef8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xd47c1f0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xd47c1f3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xd47c1f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd47c1f94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xd47c1fa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd47c1fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: grep          S CA1E33C0     0 13871  13869       
 13872 13870 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xcbebff38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xcbebff6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xcbebff94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xcbebffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: wc            S C03AD240    64 13872  13869       
 13873 13871 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xd250df38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xd250df6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd250df94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd250dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: cut           S C03AD240     0 13873  13869       
       13872 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xdd15df38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xdd15df6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xdd15df94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdd15dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: crond         S D8B8B200     0  9899   1129  9903 
       13845 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xdbd3ff38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xdbd3ff6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xdbd3ff94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdbd3ffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: triumf-up2dat S D8B8BB00     0  9903   9899  9917 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0124f37>] sys_wait4 [kernel] 0x117
(0xd020ff80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd020ffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash2         S D8B8B980     4  9915   9903  9922 
  9917       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xd0595f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xd0595f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd0595f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd0595fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: triumf-up2dat S DB6E5480     0  9917   9903  9920 
        9915 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xe463df38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xe463df6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xe463df94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe463dfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: cat           S CA1E39C0     0  9920   9917       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xd6f33f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xd6f33f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd6f33f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd6f33fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash2         S D8B8BC80    64  9922   9915  9926 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0124f37>] sys_wait4 [kernel] 0x117
(0xd5ebff80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd5ebffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AC880    24  9923   9922       
  9924       (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xdc973e44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xdc973e58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xdc973e7c))
Mar 23 19:22:35 dork kernel: [<c0141273>] __alloc_pages [kernel] 0x93 (0xdc973e98))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xdc973eb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xdc973ecc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xdc973ef8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xdc973f0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xdc973f3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xdc973f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xdc973f94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xdc973fa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdc973fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: grep          S C36CCC80    64  9924   9922       
  9925  9923 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xd8ab3f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xd8ab3f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd8ab3f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd8ab3fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: wc            S C36CC200    64  9925   9922       
  9926  9924 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xe6703f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xe6703f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xe6703f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe6703fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: cut           S C03AD240     4  9926   9922       
        9925 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0155358>] pipe_wait [kernel] 0x88
(0xe6121f38))
Mar 23 19:22:35 dork kernel: [<c0155450>] pipe_read [kernel] 0xc0 (0xe6121f6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xe6121f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe6121fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mozilla-bin   S C03AD240     0 15990  10267       
       20402 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xdf155f64))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xdf155f84))
Mar 23 19:22:35 dork kernel: [<c012b1b1>] sys_nanosleep [kernel] 0xd1 (0xdf155f9c))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdf155fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: top           D C03AC880     4 17515      1       
 17565  9965 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xcd3dfe44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xcd3dfe58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xcd3dfe7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xcd3dfe88))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xcd3dfeb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xcd3dfecc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xcd3dfef8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xcd3dff0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xcd3dff3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xcd3dff6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xcd3dff94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xcd3dffa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xcd3dffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AC880     0 17565      1       
 18266 17515 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xdb22fe44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xdb22fe58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xdb22fe7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xdb22fe88))
Mar 23 19:22:35 dork kernel: [<c012c705>] wake_up_parent [kernel] 0x25 (0xdb22fefc))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xdb22ff3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xdb22ff6c))
Mar 23 19:22:35 dork kernel: [<c011d0c6>] schedule [kernel] 0x186 (0xdb22ff7c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xdb22ff94))
Mar 23 19:22:35 dork kernel: [<c010dc31>] syscall_trace [kernel] 0x51 (0xdb22ffac))
Mar 23 19:22:35 dork kernel: [<c01094cb>] tracesys [kernel] 0x1f (0xdb22ffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AC880    28 17566   1011 17568 
 18007 31206 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xd845fe54))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xd845fe60))
Mar 23 19:22:35 dork kernel: [<c024f35f>] unix_stream_data_wait [kernel] 0xcf
(0xd845fe8c))
Mar 23 19:22:35 dork kernel: [<c024f7a1>] unix_stream_recvmsg [kernel] 0x3f1
(0xd845fec4))
Mar 23 19:22:35 dork kernel: [<c01f8548>] sock_recvmsg [kernel] 0x58 (0xd845ff08))
Mar 23 19:22:35 dork kernel: [<c01490da>] dentry_open [kernel] 0x15a (0xd845ff3c))
Mar 23 19:22:35 dork kernel: [<c01f8687>] sock_read [kernel] 0xa7 (0xd845ff50))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd845ff94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd845ffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AC880     0 17568  17566 17570 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xdd1cdeec))
Mar 23 19:22:35 dork kernel: [<c024fa6e>] unix_poll [kernel] 0x2e (0xdd1cdf00))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xdd1cdf24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xdd1cdf60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdd1cdfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash          S C03AD240    24 17570  17568       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c01905c9>] change_termios [kernel]
0x1a9 (0xe0ecbec8))
Mar 23 19:22:35 dork kernel: [<c012b0cd>] schedule_timeout [kernel] 0xad
(0xe0ecbedc))
Mar 23 19:22:35 dork kernel: [<c01339cb>] handle_mm_fault [kernel] 0x10b
(0xe0ecbee0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xe0ecbefc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xe0ecbf14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xe0ecbf74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xe0ecbf94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe0ecbfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AD240    64 18007   1011 18011 
 18109 17566 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xed583e54))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xed583e60))
Mar 23 19:22:35 dork kernel: [<c024f35f>] unix_stream_data_wait [kernel] 0xcf
(0xed583e8c))
Mar 23 19:22:35 dork kernel: [<c024f7a1>] unix_stream_recvmsg [kernel] 0x3f1
(0xed583ec4))
Mar 23 19:22:35 dork kernel: [<c01f8548>] sock_recvmsg [kernel] 0x58 (0xed583f08))
Mar 23 19:22:35 dork kernel: [<c01490da>] dentry_open [kernel] 0x15a (0xed583f3c))
Mar 23 19:22:35 dork kernel: [<c01f8687>] sock_read [kernel] 0xa7 (0xed583f50))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xed583f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xed583fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AC880     0 18011  18007 18015 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xf32ffeec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xf32ffef4))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xf32fff24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xf32fff60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf32fffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash          S C03AD240     0 18015  18011 18067 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c01269f7>] tasklet_hi_action [kernel]
0x67 (0xe89a1f6c))
Mar 23 19:22:35 dork kernel: [<c0124f37>] sys_wait4 [kernel] 0x117 (0xe89a1f80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xe89a1fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D C03AC880     0 18067  18015       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xf5aefe44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xf5aefe58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xf5aefe7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xf5aefe88))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xf5aefeb8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xf5aefecc))
Mar 23 19:22:35 dork kernel: [<c0160af0>] dput [kernel] 0x30 (0xf5aefef8))
Mar 23 19:22:35 dork kernel: [<c0156de6>] link_path_walk [kernel] 0x656
(0xf5aeff0c))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xf5aeff3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xf5aeff6c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf5aeff94))
Mar 23 19:22:35 dork kernel: [<c01495c2>] sys_open [kernel] 0xa2 (0xf5aeffa8))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf5aeffc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AD240    64 18109   1011 18114 
 18177 18007 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xd0109e54))
Mar 23 19:22:35 dork kernel: [<c0133947>] handle_mm_fault [kernel] 0x87
(0xd0109e60))
Mar 23 19:22:35 dork kernel: [<c024f35f>] unix_stream_data_wait [kernel] 0xcf
(0xd0109e8c))
Mar 23 19:22:35 dork kernel: [<c024f7a1>] unix_stream_recvmsg [kernel] 0x3f1
(0xd0109ec4))
Mar 23 19:22:35 dork kernel: [<c01f8548>] sock_recvmsg [kernel] 0x58 (0xd0109f08))
Mar 23 19:22:35 dork kernel: [<c01490da>] dentry_open [kernel] 0x15a (0xd0109f3c))
Mar 23 19:22:35 dork kernel: [<c01f8687>] sock_read [kernel] 0xa7 (0xd0109f50))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd0109f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd0109fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AD240     0 18114  18109 18119 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xcc321eec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xcc321ef4))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xcc321f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xcc321f60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xcc321fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash          S F7764B40    64 18119  18114 18171 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0124f37>] sys_wait4 [kernel] 0x117
(0xf1f31f80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf1f31fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: strace        S F7764480    64 18171  18119 18172 
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c0124f37>] sys_wait4 [kernel] 0x117
(0xf17cbf80))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf17cbfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: ps            D 66656463     0 18172  18171       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c025ecdc>] rwsem_down_failed_common
[kernel] 0x5c (0xd8d0be44))
Mar 23 19:22:35 dork kernel: [<c025eb39>] rwsem_down_read_failed [kernel] 0x29
(0xd8d0be58))
Mar 23 19:22:35 dork kernel: [<c0172deb>] .text.lock.array [kernel] 0x51
(0xd8d0be7c))
Mar 23 19:22:35 dork kernel: [<c01623bc>] alloc_inode [kernel] 0xbc (0xd8d0be88))
Mar 23 19:22:35 dork kernel: [<c012c705>] wake_up_parent [kernel] 0x25 (0xd8d0befc))
Mar 23 19:22:35 dork kernel: [<c014901e>] dentry_open [kernel] 0x9e (0xd8d0bf3c))
Mar 23 19:22:35 dork kernel: [<c016f947>] proc_info_read [kernel] 0x77 (0xd8d0bf6c))
Mar 23 19:22:35 dork kernel: [<c011d0c6>] schedule [kernel] 0x186 (0xd8d0bf7c))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xd8d0bf94))
Mar 23 19:22:35 dork kernel: [<c010dc31>] syscall_trace [kernel] 0x51 (0xd8d0bfac))
Mar 23 19:22:35 dork kernel: [<c01094cb>] tracesys [kernel] 0x1f (0xd8d0bfc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: sshd          S C03AC880    64 18177   1011 18189 
       18109 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xd8d01eec))
Mar 23 19:22:35 dork kernel: [<c021f9c6>] tcp_poll [kernel] 0x36 (0xd8d01ef4))
Mar 23 19:22:35 dork kernel: [<c015cb9d>] do_select [kernel] 0x12d (0xd8d01f24))
Mar 23 19:22:35 dork kernel: [<c015d03e>] sys_select [kernel] 0x34e (0xd8d01f60))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xd8d01fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: bash          S D44450C0    28 18189  18177       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b0cd>] schedule_timeout [kernel]
0xad (0xf03d5edc))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xf03d5efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xf03d5f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xf03d5f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf03d5f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf03d5fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: mingetty      S 00004C0F     0 18266      1       
       17565 (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c01e6a99>] vgacon_cursor [kernel]
0xf9 (0xf40f7ec4))
Mar 23 19:22:35 dork kernel: [<c012b0cd>] schedule_timeout [kernel] 0xad
(0xf40f7edc))
Mar 23 19:22:35 dork kernel: [<c019905b>] set_cursor [kernel] 0x7b (0xf40f7ee0))
Mar 23 19:22:35 dork kernel: [<c019cdb0>] con_flush_chars [kernel] 0x60
(0xf40f7ef0))
Mar 23 19:22:35 dork kernel: [<c018f582>] write_chan [kernel] 0x162 (0xf40f7efc))
Mar 23 19:22:35 dork kernel: [<c018f048>] read_chan [kernel] 0x298 (0xf40f7f14))
Mar 23 19:22:35 dork kernel: [<c018acf3>] tty_read [kernel] 0x113 (0xf40f7f74))
Mar 23 19:22:35 dork kernel: [<c014a263>] sys_read [kernel] 0xa3 (0xf40f7f94))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xf40f7fc0))
Mar 23 19:22:35 dork kernel: 
Mar 23 19:22:35 dork kernel: portmap       S C03AD240     0 18288    729       
             (NOTLB)
Mar 23 19:22:35 dork kernel: Call Trace: [<c012b07d>] schedule_timeout [kernel]
0x5d (0xdf641f0c))
Mar 23 19:22:35 dork kernel: [<c012b010>] process_timeout [kernel] 0x0 (0xdf641f2c))
Mar 23 19:22:35 dork kernel: [<c015d326>] do_pollfd [kernel] 0x166 (0xdf641f44))
Mar 23 19:22:35 dork kernel: [<c015d4e3>] sys_poll [kernel] 0x163 (0xdf641f78))
Mar 23 19:22:35 dork kernel: [<c010945f>] system_call [kernel] 0x33 (0xdf641fc0))

Comment 3 Konstantin Olchanski 2003-04-11 04:19:00 UTC

Okey, I see the same problem on a redhat 7.3 system with the 2.4.18-24.7.xsmp
kernel. The symptoms are slightly different- userland still hangs on reads from
/proc/PID/stat, but the alt-sysqr-t output now show them hanging in
sys_read->proc_info_read->__alloc_pages->.text.lock.array: (full trace available
on request)

Apr 10 21:12:20 lin06 kernel: top           D CAB10680     4 13219  13218      
  13220       (NOTLB)
Apr 10 21:12:20 lin06 kernel: Call Trace: [<c014dbe0>] cached_lookup [kernel]
0x10 (0xdfd43ec8))
Apr 10 21:12:20 lin06 kernel: [<c023b4c5>] rwsem_down_read_failed [kernel] 0x135
(0xdfd43ee4))
Apr 10 21:12:20 lin06 kernel: [<c0167197>] .text.lock.array [kernel] 0x73
(0xdfd43f10))
Apr 10 21:12:20 lin06 kernel: [<c013b39b>] __alloc_pages [kernel] 0x6b (0xdfd43f3c))
Apr 10 21:12:20 lin06 kernel: [<c016443c>] proc_info_read [kernel] 0x4c
(0xdfd43f58))
Apr 10 21:12:20 lin06 kernel: [<c01434a6>] sys_read [kernel] 0x96 (0xdfd43f7c))
Apr 10 21:12:20 lin06 kernel: [<c0142e37>] sys_open [kernel] 0x57 (0xdfd43fac))
Apr 10 21:12:20 lin06 kernel: [<c0108ca3>] system_call [kernel] 0x33 (0xdfd43fc0))

K.O.

Comment 4 Bugzilla owner 2004-09-30 15:40:41 UTC

Thanks for the bug report. However, Red Hat no longer maintains this version of
the product. Please upgrade to the latest version and open a new bug if the problem
persists.

The Fedora Legacy project (http://fedoralegacy.org/) maintains some older releases, 
and if you believe this bug is interesting to them, please report the problem in
the bug tracker at: http://bugzilla.fedora.us/

Note You need to log in before you can comment on or make changes to this bug.