Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 84738 - Kernel "lockup" or "busy wait" in do_get_write_access"
Summary: Kernel "lockup" or "busy wait" in do_get_write_access"
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: kernel
Version: 8.0
Hardware: i686
OS: Linux
high
high
Target Milestone: ---
Assignee: Arjan van de Ven
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-02-20 23:13 UTC by Bill
Modified: 2005-10-31 22:00 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2004-09-30 15:40:33 UTC

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Bill 2003-02-20 23:13:37 UTC 
Description of problem:

Kernel will enter a busy wait state will perform the file system ext3 
journaling.  We can recreate this bug at will.  The symptoms are different 
depending on whether you are on a 1 or 2 cpu system and on how many processes 
enter the "busy wait" state.  

On a single cpu system you can ping or get a sysReq 't' from the console but 
all other access is not available.  

If you are on a multi-processor system and you happen to lock up only one cpu 
you can still gain access to system functions.. (ls, ps, console, etc.)  
However, you can not kill any of the "suspect" processes (even kill -9 is 
ignored.) as the processes seem to be in a busy loop in the kernal code.

See SysReq output in Additional info field.

We are seeing this very frequently because we are using named pipes (inducing 
file journaling) in real time mode under high performance conditions.  But in 
searching your database this seems very similar to bugs 79257 & 79573 and the 
results they experience and explain are consistant with our testing.  I suspect 
that our testing is just much more likely to expose the bug / symptom...

Version-Release number of selected component (if applicable):

8.0 / 7.3 Linux

How reproducible:

at will / every time we run the test

Steps to reproduce

1.multiple named pipes (opened under ext3 file system)
2.set processes reading / writing to these pipes to real time mode (either FIFO 
or Round Robin)
3.randomly read/write to the pipes under very high frequency (order of hundreds 
of transactions per second on each pipe)
NOTE this is HW independent and we can reproduce at will in 7.3 and 8.0
        
Actual results:

processes performing journaling enter a busy loop taking up all available CPU.  



Expected results:

Exit busy wait state in kernel normally

Additional info:


Here is the sysreq 't" output.  On multiple cpu systems we can observe a top 
running with every process below that is in the "do_get_write_access" call 
utilizing all of the CPU.  Note that when this occurs you can still ping the 
system and sysReq will work even on a single CPU system...

SysRq : Show State

                         free                        sibling
  task             PC    stack   pid father child younger older
init          R 00000000  3292     1      0  1471               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

migration_CPU S C024514A  6068     2      1             3       (L-TLB)
Call Trace: [<c011a199>] migration_thread [kernel] 0x119 
[<c0105000>] stext [kernel] 0x0 
[<c0105000>] stext [kernel] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c011a080>] migration_thread [kernel] 0x0 

keventd       R F7EEFF84  5400     3      1             4     2 (L-TLB)
Call Trace: [<c0118f79>] __wake_up [kernel] 0x49 
[<c0129fb1>] context_thread [kernel] 0x131 
[<c0129e80>] context_thread [kernel] 0x0 
[<c0105000>] stext [kernel] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c0129e80>] context_thread [kernel] 0x0 

kapmd         R 00000000  5388     4      1             5     3 (L-TLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01145f3>] apm_mainloop [kernel] 0x83 
[<c0114f46>] apm [kernel] 0x286 
[<c0105000>] stext [kernel] 0x0 
[<c0160018>] sys_quotactl [kernel] 0x88 
[<c0105000>] stext [kernel] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c0114cc0>] apm [kernel] 0x0 

ksoftirqd_CPU R FFFFFFFB  5532     5      1             6     4 (L-TLB)
Call Trace: [<c012148f>] ksoftirqd [kernel] 0xaf 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c01213e0>] ksoftirqd [kernel] 0x0 

kswapd        R 00000000  5356     6      1             7     5 (L-TLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c013920e>] wakeup_memwaiters [kernel] 0xce 
[<c0138fec>] kswapd [kernel] 0x2bc 
[<c0105000>] stext [kernel] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c0138d30>] kswapd [kernel] 0x0 

bdflush       S 00000000  6368     7      1             8     6 (L-TLB)
Call Trace: [<c011911b>] interruptible_sleep_on [kernel] 0x4b 
[<c014694e>] bdflush [kernel] 0xee 
[<c0105000>] stext [kernel] 0x0 
[<c0105000>] stext [kernel] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c0146860>] bdflush [kernel] 0x0 

kupdated      R 00000282  5408     8      1             9     7 (L-TLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c014670c>] sync_old_buffers [kernel] 0x2c 
[<c01469f4>] kupdate [kernel] 0xa4 
[<c0105000>] stext [kernel] 0x0 
[<c0105000>] stext [kernel] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c0146950>] kupdate [kernel] 0x0 

mdrecoveryd   S 081D3CCE  6328     9      1            13     8 (L-TLB)
Call Trace: [<c01d1bb7>] md_thread [kernel] 0x117 
[<c0107286>] kernel_thread [kernel] 0x26 
[<c01d1aa0>] md_thread [kernel] 0x0 

kjournald     R 00000000  4808    13      1           113     9 (L-TLB)
Call Trace: [<c011911b>] interruptible_sleep_on [kernel] 0x4b 
[<f881286a>] kjournald [jbd] 0x15a 
[<f88126f0>] commit_timeout [jbd] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<f8812710>] kjournald [jbd] 0x0 

kjournald     S C011C78B  5104   113      1           114    13 (L-TLB)
Call Trace: [<c011c78b>] call_console_drivers [kernel] 0xeb 
[<c011911b>] interruptible_sleep_on [kernel] 0x4b 
[<f881286a>] kjournald [jbd] 0x15a 
[<f88126f0>] commit_timeout [jbd] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<f8812710>] kjournald [jbd] 0x0 

kjournald     S C011C78B   352   114      1           115   113 (L-TLB)
Call Trace: [<c011c78b>] call_console_drivers [kernel] 0xeb 
[<c011911b>] interruptible_sleep_on [kernel] 0x4b 
[<f881286a>] kjournald [jbd] 0x15a 
[<f88126f0>] commit_timeout [jbd] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<f8812710>] kjournald [jbd] 0x0 

kjournald     R 00000286    68   115      1           116   114 (L-TLB)
Call Trace: [<c0142f46>] __wait_on_buffer [kernel] 0x76 
[<f881002f>] journal_commit_transaction [jbd] 0x9df 
[<c0118ecc>] schedule [kernel] 0x37c 
[<f8812846>] kjournald [jbd] 0x136 
[<f88126f0>] commit_timeout [jbd] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<f8812710>] kjournald [jbd] 0x0 

kjournald     S 00000000  2400   116      1           543   115 (L-TLB)
Call Trace: [<c011911b>] interruptible_sleep_on [kernel] 0x4b 
[<f881286a>] kjournald [jbd] 0x15a 
[<f88126f0>] commit_timeout [jbd] 0x0 
[<c0107286>] kernel_thread [kernel] 0x26 
[<f8812710>] kjournald [jbd] 0x0 

syslogd       R 00000000  1376   543      1           548   116 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c0108ad1>] do_signal [kernel] 0x231 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

klogd         R F686DD40  1376   548      1           568   543 (NOTLB)
Call Trace: [<c011c25d>] do_syslog [kernel] 0xdd 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01206b2>] sys_time [kernel] 0x12 
[<c0108c6b>] system_call [kernel] 0x33 

portmap       S 00000001     8   568      1           596   548 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c0152849>] do_pollfd [kernel] 0x59 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c014171a>] filp_close [kernel] 0x9a 
[<c0108c6b>] system_call [kernel] 0x33 

rpc.statd     S F68578E0  1376   596      1           742   568 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c0157d8f>] clear_inode [kernel] 0xf 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

sshd          S F686DF40    68   742      1           775   596 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

xinetd        S F5CAADA0     0   775      1  1513     812   742 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

snmpdm        S F57F4540    68   812      1           833   775 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

gpm           R 00000000  5252   833      1           851   812 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

crond         R 00000000    68   851      1           921   833 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0108072>] sys_sigreturn [kernel] 0xd2 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

procmgr       R 00000000    68   921      1  1675    1078   851 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c0152849>] do_pollfd [kernel] 0x59 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

MasterClerk   R 00000000  2400   922    921           923       (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

ProcWatch     R 00000282    68   923    921           924   922 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0165e86>] proc_pid_read_maps [kernel] 0x226 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0141d0f>] sys_read [kernel] 0x10f 
[<c0108c6b>] system_call [kernel] 0x33 

SysState      R 00000000    68   924    921           925   923 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

appOam        R 00000000    68   925    921           926   924 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

cdrGen        R 00000000    68   926    921           927   925 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

faultmon      D F6854800    68   927    921           928   926 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f8812e1a>] log_wait_commit_Rsmp_e7e6dd91 [jbd] 0x5a 
[<f880e849>] journal_stop_Rsmp_d72faea2 [jbd] 0x199 
[<f880e8aa>] journal_force_commit_Rsmp_88c2e786 [jbd] 0x4a 
[<f8827462>] ext3_force_commit [ext3] 0x42 
[<f881de17>] ext3_sync_file [ext3] 0x87 
[<f8820b50>] ext3_writepage [ext3] 0x0 
[<c0143532>] sys_fsync [kernel] 0x62 
[<c0108c6b>] system_call [kernel] 0x33 

idb.autozap   R 00000000    68   928    921           929   927 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

idbsvc        R 00000000     8   929    921           930   928 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

idbsvc        R 00000000    68   930    921           931   929 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

inetsync      R 00000000     0   931    921           932   930 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C01334D0   964   932    921  1002     933   931 (NOTLB)
Call Trace: [<c01334d0>] sys_mprotect [kernel] 0x120 
[<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000    68   933    921  1003     934   932 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

netFailClerk  R 00000000     8   934    921           935   933 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

pltOam        R 00000000    68   935    921           936   934 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

statclerk     R 00000000    68   936    921           937   935 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01dd2bb>] net_rx_action [kernel] 0x1eb 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120f0b>] do_softirq [kernel] 0x6b 
[<c0108c6b>] system_call [kernel] 0x33 

vxiMibSnmpAgt S 00000000  2400   937    921           938   936 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

vxiSnmpTrapAg R 00000000     8   938    921           941   937 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

Congest       R 00000000     0   941    921           944   938 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c010e423>] sys_ipc [kernel] 0x1b3 
[<c0108c6b>] system_call [kernel] 0x33 

TCAPAgent     R 00000000     8   944    921           945   941 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<f8816d50>] .rodata.str1.1 [jbd] 0x30 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c0158bc1>] update_atime [kernel] 0x51 
[<c014b72c>] pipe_read [kernel] 0x21c 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

bsmAgent      R 00000000    68   945    921   999     946   944 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

controller    S 00000000  2400   946    921   966     947   945 (NOTLB)
Call Trace: [<c010e652>] sys_pause [kernel] 0x12 
[<c0108c6b>] system_call [kernel] 0x33 

cpHelper      R 00000000  2400   947    921           948   946 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0141d0f>] sys_read [kernel] 0x10f 
[<c0108c6b>] system_call [kernel] 0x33 

iuaAgent      R 00000000     0   948    921           949   947 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<f8816d50>] .rodata.str1.1 [jbd] 0x30 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c014b94a>] pipe_write [kernel] 0x20a 
[<c0141db6>] sys_write [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

priAgent      R 00000000     8   949    921           950   948 (NOTLB)
Call Trace: [<c0107925>] __down_interruptible [kernel] 0x75 
[<c0107a03>] __down_failed_interruptible [kernel] 0x7 
[<c014c0e3>] .text.lock.pipe [kernel] 0x19 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

sctpDaemon    R 00000000  2400   950    921           951   949 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c0152849>] do_pollfd [kernel] 0x59 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

sipAgent      R 00000000     0   951    921           952   950 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<c010829a>] setup_sigcontext [kernel] 0xda 
[<f8816d50>] .rodata.str1.1 [jbd] 0x30 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c0158bc1>] update_atime [kernel] 0x51 
[<c014b72c>] pipe_read [kernel] 0x21c 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

talonAdmin    R 00000000     0   952    921           953   951 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0108c6b>] system_call [kernel] 0x33 

tqtool        R 00000000     8   953    921          1613   952 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<c010829a>] setup_sigcontext [kernel] 0xda 
[<f8816d50>] .rodata.str1.1 [jbd] 0x30 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c0158bc1>] update_atime [kernel] 0x51 
[<c014b72c>] pipe_read [kernel] 0x21c 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

controller    R 00000000     4   966    946   972               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0118ecc>] schedule [kernel] 0x37c 
[<c0108c6b>] system_call [kernel] 0x33 

controller    R 00000000     8   967    966           969       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

controller    S 00000000     0   969    966           970   967 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

controller    R 00000000    68   970    966  1281     971   969 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

controller    S 00000000     8   971    966           972   970 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c011a3b9>] ll_copy_to_user [kernel] 0x49 
[<c01daaa6>] wait_for_packet [kernel] 0xe6 
[<c01dabcf>] skb_recv_datagram [kernel] 0xcf 
[<c020fb69>] udp_recvmsg [kernel] 0x59 
[<c021c40f>] fib_lookup [kernel] 0xff 
[<c0216359>] inet_recvmsg [kernel] 0x39 
[<c01d54c1>] sock_recvmsg [kernel] 0x31 
[<c01d6481>] sys_recvfrom [kernel] 0xa1 
[<c0125e3e>] ignored_signal [kernel] 0x2e 
[<c01260e2>] send_sig_info [kernel] 0x82 
[<c0126406>] kill_something_info [kernel] 0x156 
[<c02118e4>] arp_rcv [kernel] 0x94 
[<c0126c99>] sys_kill [kernel] 0x49 
[<c01d6bcb>] sys_socketcall [kernel] 0x17b 
[<c0108c6b>] system_call [kernel] 0x33 

controller    S 00000000  2400   972    966                 971 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

bsmAgent      R 00000000  5612   999    945  1000               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120f0b>] do_softirq [kernel] 0x6b 
[<c0108c6b>] system_call [kernel] 0x33 

bsmAgent      S 00000000  5448  1000    999                     (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  5612  1002    932  1140               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120070>] sys_wait4 [kernel] 0x3e0 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  5284  1003    933  1142               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120070>] sys_wait4 [kernel] 0x3e0 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  2404  1004   1002          1008       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  5456  1005   1003          1007       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  1376  1007   1003          1009  1005 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C0107F75  2404  1008   1002          1010  1004 (NOTLB)
Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C0107F75     0  1009   1003          1031  1007 (NOTLB)
Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C0107F75  2404  1010   1002          1030  1008 (NOTLB)
Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  5376  1030   1002          1032  1010 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  2400  1031   1003          1033  1009 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0126805>] sys_rt_sigprocmask [kernel] 0x165 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1032   1002          1034  1030 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1033   1003          1035  1031 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1034   1002          1036  1032 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1035   1003          1037  1033 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  5284  1036   1002          1064  1034 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  5212  1037   1003          1065  1035 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C022ED72     4  1064   1002          1067  1036 (NOTLB)
Call Trace: [<c022ed72>] vsnprintf [kernel] 0x2a2 
[<c0125444>] schedule_timeout [kernel] 0x14 
[<c01fbdd7>] wait_for_connect [kernel] 0x117 
[<c01fbf53>] tcp_accept [kernel] 0xb3 
[<c021611b>] inet_accept [kernel] 0x2b 
[<c01d60b3>] sys_accept [kernel] 0x63 
[<c0118f79>] __wake_up [kernel] 0x49 
[<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c01d6b00>] sys_socketcall [kernel] 0xb0 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C022ED72     8  1065   1003          1068  1037 (NOTLB)
Call Trace: [<c022ed72>] vsnprintf [kernel] 0x2a2 
[<c0125444>] schedule_timeout [kernel] 0x14 
[<c01fbdd7>] wait_for_connect [kernel] 0x117 
[<c01fbf53>] tcp_accept [kernel] 0xb3 
[<c021611b>] inet_accept [kernel] 0x2b 
[<c01d60b3>] sys_accept [kernel] 0x63 
[<c0118f79>] __wake_up [kernel] 0x49 
[<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c01d6b00>] sys_socketcall [kernel] 0xb0 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000     0  1067   1002          1069  1064 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000     0  1068   1003          1070  1065 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1069   1002          1074  1067 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S F54E8360  6012  1070   1003          1075  1068 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R FFFFFFFF  5668  1074   1002          1077  1069 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0108072>] sys_sigreturn [kernel] 0xd2 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  5600  1075   1003          1076  1070 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0108072>] sys_sigreturn [kernel] 0xd2 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  4532  1076   1003          1083  1075 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          R 00000000  4532  1077   1002          1082  1074 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

xfs           S 00000000  4872  1078      1          1100   921 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

java          S F192FD20  5656  1082   1002          1136  1077 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c0139dd2>] __alloc_pages [kernel] 0x72 
[<c01fbdd7>] wait_for_connect [kernel] 0x117 
[<c01fbf53>] tcp_accept [kernel] 0xb3 
[<c021611b>] inet_accept [kernel] 0x2b 
[<c01d60b3>] sys_accept [kernel] 0x63 
[<c011780d>] do_page_fault [kernel] 0x12d 
[<c0118f79>] __wake_up [kernel] 0x49 
[<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c01d6b00>] sys_socketcall [kernel] 0xb0 
[<c01176e0>] do_page_fault [kernel] 0x0 
[<c0108d5c>] error_code [kernel] 0x34 
[<c0108c6b>] system_call [kernel] 0x33 

java          S C022ED72  5612  1083   1003          1137  1076 (NOTLB)
Call Trace: [<c022ed72>] vsnprintf [kernel] 0x2a2 
[<c0125444>] schedule_timeout [kernel] 0x14 
[<c01fbdd7>] wait_for_connect [kernel] 0x117 
[<c01fbf53>] tcp_accept [kernel] 0xb3 
[<c021611b>] inet_accept [kernel] 0x2b 
[<c01d60b3>] sys_accept [kernel] 0x63 
[<c011780d>] do_page_fault [kernel] 0x12d 
[<c0118f79>] __wake_up [kernel] 0x49 
[<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c01d6b00>] sys_socketcall [kernel] 0xb0 
[<c01176e0>] do_page_fault [kernel] 0x0 
[<c0108d5c>] error_code [kernel] 0x34 
[<c0108c6b>] system_call [kernel] 0x33 

atd           S 00001000  5252  1100      1          1107  1078 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

mingetty      S C1E130E0  4820  1107      1          1108  1100 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01875b9>] set_cursor [kernel] 0x69 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012e715>] sys_munmap [kernel] 0x35 
[<c0108c6b>] system_call [kernel] 0x33 

mingetty      S F5661D40     0  1108      1          1109  1107 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012e715>] sys_munmap [kernel] 0x35 
[<c0108c6b>] system_call [kernel] 0x33 

mingetty      S F5661E80  2060  1109      1          1110  1108 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012e715>] sys_munmap [kernel] 0x35 
[<c0108c6b>] system_call [kernel] 0x33 

mingetty      S F786D620   764  1110      1          1111  1109 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012e715>] sys_munmap [kernel] 0x35 
[<c0108c6b>] system_call [kernel] 0x33 

mingetty      S F786D8A0     0  1111      1          1112  1110 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012e715>] sys_munmap [kernel] 0x35 
[<c0108c6b>] system_call [kernel] 0x33 

mingetty      S F5661CA0     4  1112      1          1113  1111 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c012e6cf>] do_munmap [kernel] 0x28f 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012e715>] sys_munmap [kernel] 0x35 
[<c0108c6b>] system_call [kernel] 0x33 

kdm           S F1AC3B20  4504  1113      1  1131    1471  1112 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

X             S 00000000     8  1130   1113          1131       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

kdm           S 00000000    96  1131   1113  1171          1130 (NOTLB)
Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c 
[<c014b5c4>] pipe_read [kernel] 0xb4 
[<c0141c96>] sys_read [kernel] 0x96 
[<c0108c6b>] system_call [kernel] 0x33 

java          S FFFFFFFF  4576  1136   1002          1140  1082 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0108072>] sys_sigreturn [kernel] 0xd2 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S FFFFFFFF  4576  1137   1003          1142  1083 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c0108072>] sys_sigreturn [kernel] 0xd2 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1140   1002                1136 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

java          S 00000000  6012  1142   1003                1137 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01255fb>] sys_nanosleep [kernel] 0x11b 
[<c0108c6b>] system_call [kernel] 0x33 

kdm_greet     R 00000000  4452  1171   1131                     (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c01d5891>] sock_ioctl [kernel] 0x51 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1190    970  1191    1260       (NOTLB)
Call Trace: [<c010e652>] sys_pause [kernel] 0x12 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000  5332  1191   1190  1223               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120070>] sys_wait4 [kernel] 0x3e0 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000  4768  1192   1191          1193       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000012  5172  1193   1191          1194  1192 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000  5320  1194   1191          1195  1193 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000  6056  1195   1191          1196  1194 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000  2404  1196   1191          1197  1195 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EDE62360  6096  1197   1191          1198  1196 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000  5280  1198   1191          1199  1197 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c010a54e>] handle_IRQ_event [kernel] 0x5e 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000  5220  1199   1191          1200  1198 (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000  5060  1200   1191          1209  1199 (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EDE4BF3C  5600  1209   1191          1211  1200 (NOTLB)
Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c 
[<c014b5c4>] pipe_read [kernel] 0xb4 
[<c0141c96>] sys_read [kernel] 0x96 
[<c010a79f>] do_IRQ [kernel] 0xdf 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 000004B6  5308  1211   1191          1212  1209 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000  4632  1212   1191          1223  1211 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01fa380>] tcp_data_wait [kernel] 0x100 
[<c01fab9f>] tcp_recvmsg [kernel] 0x51f 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c0216359>] inet_recvmsg [kernel] 0x39 
[<c01d54c1>] sock_recvmsg [kernel] 0x31 
[<c0142c06>] fput [kernel] 0x16 
[<c01d6481>] sys_recvfrom [kernel] 0xa1 
[<c0125e3e>] ignored_signal [kernel] 0x2e 
[<c01f1025>] ip_local_deliver [kernel] 0xf5 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d8db9>] __kfree_skb [kernel] 0x129 
[<c01dcf92>] net_tx_action [kernel] 0x62 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d64fd>] sys_recv [kernel] 0x1d 
[<c01d6ba4>] sys_socketcall [kernel] 0x154 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000  4772  1223   1191                1212 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1260    970  1261    1281  1190 (NOTLB)
Call Trace: [<c010e652>] sys_pause [kernel] 0x12 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000     0  1261   1260  1278               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120070>] sys_wait4 [kernel] 0x3e0 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000     0  1262   1261          1263       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000012     0  1263   1261          1264  1262 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000012     0  1264   1261          1265  1263 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1265   1261          1266  1264 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1266   1261          1267  1265 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EC276360     0  1267   1261          1268  1266 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000     8  1268   1261          1269  1267 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S F5999CD4     0  1269   1261          1270  1268 (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1270   1261          1272  1269 (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EC6F5F3C  2400  1272   1261          1273  1270 (NOTLB)
Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c 
[<c014b5c4>] pipe_read [kernel] 0xb4 
[<c0141c96>] sys_read [kernel] 0x96 
[<c012d625>] sys_brk [kernel] 0xc5 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000418     0  1273   1261          1274  1272 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R C01D8DB9     0  1274   1261          1275  1273 (NOTLB)
Call Trace: [<c01d8db9>] __kfree_skb [kernel] 0x129 
[<c0125444>] schedule_timeout [kernel] 0x14 
[<c01fa380>] tcp_data_wait [kernel] 0x100 
[<c01fab9f>] tcp_recvmsg [kernel] 0x51f 
[<c0216359>] inet_recvmsg [kernel] 0x39 
[<c01d54c1>] sock_recvmsg [kernel] 0x31 
[<c01f1025>] ip_local_deliver [kernel] 0xf5 
[<c01d6481>] sys_recvfrom [kernel] 0xa1 
[<c0120f0b>] do_softirq [kernel] 0x6b 
[<c010a79f>] do_IRQ [kernel] 0xdf 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d64fd>] sys_recv [kernel] 0x1d 
[<c01d6ba4>] sys_socketcall [kernel] 0x154 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S C0107F75     0  1275   1261          1276  1274 (NOTLB)
Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000012     0  1276   1261          1277  1275 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R C01D8DB9     0  1277   1261          1278  1276 (NOTLB)
Call Trace: [<c01d8db9>] __kfree_skb [kernel] 0x129 
[<c0125444>] schedule_timeout [kernel] 0x14 
[<c01fa380>] tcp_data_wait [kernel] 0x100 
[<c01fab9f>] tcp_recvmsg [kernel] 0x51f 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c0216359>] inet_recvmsg [kernel] 0x39 
[<c01d54c1>] sock_recvmsg [kernel] 0x31 
[<c0142c02>] fput [kernel] 0x12 
[<c01d6481>] sys_recvfrom [kernel] 0xa1 
[<c0118b24>] scheduler_tick [kernel] 0x334 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d8db9>] __kfree_skb [kernel] 0x129 
[<c01dcf92>] net_tx_action [kernel] 0x62 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d64fd>] sys_recv [kernel] 0x1d 
[<c01d6ba4>] sys_socketcall [kernel] 0x154 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S C0107F75     0  1278   1261                1277 (NOTLB)
Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 
[<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1281    970  1282          1260 (NOTLB)
Call Trace: [<c010e652>] sys_pause [kernel] 0x12 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000     0  1282   1281  1296               (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01528f6>] do_pollfd [kernel] 0x106 
[<c015292e>] do_pollfd [kernel] 0x13e 
[<c0152ba2>] sys_poll [kernel] 0x252 
[<c0120070>] sys_wait4 [kernel] 0x3e0 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000     0  1283   1282          1284       (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000012     0  1284   1282          1285  1283 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1285   1282          1286  1284 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1286   1282          1287  1285 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EC5FA360     0  1287   1282          1288  1286 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EC5F8360     0  1288   1282          1289  1287 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R 00000000     0  1289   1282          1290  1288 (NOTLB)
Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c 
[<c0125420>] process_timeout [kernel] 0x0 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S E7823594     0  1290   1282          1291  1289 (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1291   1282          1293  1290 (NOTLB)
Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb 
[<c010e3d0>] sys_ipc [kernel] 0x160 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S EC5C9F3C  2400  1293   1282          1294  1291 (NOTLB)
Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c 
[<c014b5c4>] pipe_read [kernel] 0xb4 
[<c0141c96>] sys_read [kernel] 0x96 
[<c010a79f>] do_IRQ [kernel] 0xdf 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 000004B6     0  1294   1282          1295  1293 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    R F04D46D8     0  1295   1282          1296  1294 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01da628>] memcpy_toiovec [kernel] 0x38 
[<c01fa380>] tcp_data_wait [kernel] 0x100 
[<c01fab9f>] tcp_recvmsg [kernel] 0x51f 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c0216359>] inet_recvmsg [kernel] 0x39 
[<c01d54c1>] sock_recvmsg [kernel] 0x31 
[<c0142c02>] fput [kernel] 0x12 
[<c01d6481>] sys_recvfrom [kernel] 0xa1 
[<c01181f9>] wake_up_process [kernel] 0x9 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d8db9>] __kfree_skb [kernel] 0x129 
[<c01dcf92>] net_tx_action [kernel] 0x62 
[<c01d8c2c>] kfree_skbmem [kernel] 0xc 
[<c01d64fd>] sys_recv [kernel] 0x1d 
[<c01d6ba4>] sys_socketcall [kernel] 0x154 
[<c0108c6b>] system_call [kernel] 0x33 

taliClient    S 00000000     0  1296   1282                1295 (NOTLB)
Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b 
[<c0108c6b>] system_call [kernel] 0x33 

agetty        R 40015000  4776  1471      1                1113 (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c01212db>] bh_action [kernel] 0x4b 
[<c0141c96>] sys_read [kernel] 0x96 
[<c010a79f>] do_IRQ [kernel] 0xdf 
[<c0108c6b>] system_call [kernel] 0x33 

in.telnetd    R F36AA640     0  1513    775  1514               (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c01d58df>] sock_poll [kernel] 0x1f 
[<c01522ff>] do_select [kernel] 0x1ff 
[<c01526a9>] sys_select [kernel] 0x339 
[<c010a79f>] do_IRQ [kernel] 0xdf 
[<c0108c6b>] system_call [kernel] 0x33 

login         S 00000000     0  1514   1513  1515               (NOTLB)
Call Trace: [<c0120048>] sys_wait4 [kernel] 0x3b8 
[<c0108c6b>] system_call [kernel] 0x33 

ksh           S 642F6C61     4  1515   1514                     (NOTLB)
Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 
[<c017ecba>] read_chan [kernel] 0x37a 
[<c017ab87>] tty_read [kernel] 0xd7 
[<c017c802>] tty_ioctl [kernel] 0x372 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01517d4>] sys_ioctl [kernel] 0x294 
[<c0108c6b>] system_call [kernel] 0x33 

Ss7Agent      D 00000000  2400  1613    921          1614   953 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<c010829a>] setup_sigcontext [kernel] 0xda 
[<c01086a9>] setup_rt_frame [kernel] 0x1b9 
[<f8816d50>] .rodata.str1.1 [jbd] 0x30 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c0158bc1>] update_atime [kernel] 0x51 
[<c014b72c>] pipe_read [kernel] 0x21c 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

SwitchAgent   D 00000000     0  1614    921          1675  1613 (NOTLB)
Call Trace: [<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<f8816d50>] .rodata.str1.1 [jbd] 0x30 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c0158bc1>] update_atime [kernel] 0x51 
[<c014b72c>] pipe_read [kernel] 0x21c 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33 

CP            R current      0  1675    921                1614 (NOTLB)
Call Trace: [<c0118efb>] schedule [kernel] 0x3ab 
[<c011920b>] sleep_on [kernel] 0x4b 
[<f880db2b>] do_get_write_access [jbd] 0x2db 
[<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 
[<f8822851>] ext3_reserve_inode_write [ext3] 0x31 
[<f8816d20>] .rodata.str1.1 [jbd] 0x0 
[<f8814137>] __jbd_kmalloc [jbd] 0x27 
[<f880d11e>] get_transaction [jbd] 0xbe 
[<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 
[<f88229a8>] ext3_dirty_inode [ext3] 0x98 
[<c015723e>] __mark_inode_dirty [kernel] 0x2e 
[<c0158bc1>] update_atime [kernel] 0x51 
[<c014b72c>] pipe_read [kernel] 0x21c 
[<c0141c96>] sys_read [kernel] 0x96 
[<c01207bb>] sys_gettimeofday [kernel] 0x1b 
[<c0108c6b>] system_call [kernel] 0x33
Comment 1 Arjan van de Ven 2003-02-21 14:56:58 UTC 
is it possible to test the rawhide kernel with this ?
It has some ext3 vs realtime fixes
Comment 2 Stephen Tweedie 2003-02-24 12:26:09 UTC 
Current rawhide contains a fix that addresses exactly these symptoms ---
scheduling storms or realtime lockups in do_get_write_access.  Please reopen if
you can reproduce with a current rawhide kernel.
Comment 3 Bill 2003-02-24 19:22:12 UTC 
This is insanely disappointing!!!  I find a bug we can recreated at will that 
hangs the kernal in 7.3 and 8.0 (both widely used) and the bug report is closed 
with the statement basically saying "get the next release".  This is why linux 
will never be widely used i.e. you will never have a reliable product because 
bug fixes and new code are continuously put in, instead of offering fixes / 
patches to existing releases!
Comment 4 Bugzilla owner 2004-09-30 15:40:33 UTC 
Thanks for the bug report. However, Red Hat no longer maintains this version of
the product. Please upgrade to the latest version and open a new bug if the problem
persists.

The Fedora Legacy project (http://fedoralegacy.org/) maintains some older releases, 
and if you believe this bug is interesting to them, please report the problem in
the bug tracker at: http://bugzilla.fedora.us/
Note You need to log in before you can comment on or make changes to this bug.