Bug 84738
| Summary: | Kernel "lockup" or "busy wait" in do_get_write_access" | ||
|---|---|---|---|
| Product: | [Retired] Red Hat Linux | Reporter: | Bill <bill.hall> |
| Component: | kernel | Assignee: | Arjan van de Ven <arjanv> |
| Status: | CLOSED WONTFIX | QA Contact: | Brian Brock <bbrock> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 8.0 | CC: | dmaley, mitr, sct |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | i686 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2004-09-30 15:40:33 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
is it possible to test the rawhide kernel with this ? It has some ext3 vs realtime fixes Current rawhide contains a fix that addresses exactly these symptoms --- scheduling storms or realtime lockups in do_get_write_access. Please reopen if you can reproduce with a current rawhide kernel. This is insanely disappointing!!! I find a bug we can recreated at will that hangs the kernal in 7.3 and 8.0 (both widely used) and the bug report is closed with the statement basically saying "get the next release". This is why linux will never be widely used i.e. you will never have a reliable product because bug fixes and new code are continuously put in, instead of offering fixes / patches to existing releases! Thanks for the bug report. However, Red Hat no longer maintains this version of the product. Please upgrade to the latest version and open a new bug if the problem persists. The Fedora Legacy project (http://fedoralegacy.org/) maintains some older releases, and if you believe this bug is interesting to them, please report the problem in the bug tracker at: http://bugzilla.fedora.us/ |
Description of problem: Kernel will enter a busy wait state will perform the file system ext3 journaling. We can recreate this bug at will. The symptoms are different depending on whether you are on a 1 or 2 cpu system and on how many processes enter the "busy wait" state. On a single cpu system you can ping or get a sysReq 't' from the console but all other access is not available. If you are on a multi-processor system and you happen to lock up only one cpu you can still gain access to system functions.. (ls, ps, console, etc.) However, you can not kill any of the "suspect" processes (even kill -9 is ignored.) as the processes seem to be in a busy loop in the kernal code. See SysReq output in Additional info field. We are seeing this very frequently because we are using named pipes (inducing file journaling) in real time mode under high performance conditions. But in searching your database this seems very similar to bugs 79257 & 79573 and the results they experience and explain are consistant with our testing. I suspect that our testing is just much more likely to expose the bug / symptom... Version-Release number of selected component (if applicable): 8.0 / 7.3 Linux How reproducible: at will / every time we run the test Steps to reproduce 1.multiple named pipes (opened under ext3 file system) 2.set processes reading / writing to these pipes to real time mode (either FIFO or Round Robin) 3.randomly read/write to the pipes under very high frequency (order of hundreds of transactions per second on each pipe) NOTE this is HW independent and we can reproduce at will in 7.3 and 8.0 Actual results: processes performing journaling enter a busy loop taking up all available CPU. Expected results: Exit busy wait state in kernel normally Additional info: Here is the sysreq 't" output. On multiple cpu systems we can observe a top running with every process below that is in the "do_get_write_access" call utilizing all of the CPU. Note that when this occurs you can still ping the system and sysReq will work even on a single CPU system... SysRq : Show State free sibling task PC stack pid father child younger older init R 00000000 3292 1 0 1471 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 migration_CPU S C024514A 6068 2 1 3 (L-TLB) Call Trace: [<c011a199>] migration_thread [kernel] 0x119 [<c0105000>] stext [kernel] 0x0 [<c0105000>] stext [kernel] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<c011a080>] migration_thread [kernel] 0x0 keventd R F7EEFF84 5400 3 1 4 2 (L-TLB) Call Trace: [<c0118f79>] __wake_up [kernel] 0x49 [<c0129fb1>] context_thread [kernel] 0x131 [<c0129e80>] context_thread [kernel] 0x0 [<c0105000>] stext [kernel] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<c0129e80>] context_thread [kernel] 0x0 kapmd R 00000000 5388 4 1 5 3 (L-TLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01145f3>] apm_mainloop [kernel] 0x83 [<c0114f46>] apm [kernel] 0x286 [<c0105000>] stext [kernel] 0x0 [<c0160018>] sys_quotactl [kernel] 0x88 [<c0105000>] stext [kernel] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<c0114cc0>] apm [kernel] 0x0 ksoftirqd_CPU R FFFFFFFB 5532 5 1 6 4 (L-TLB) Call Trace: [<c012148f>] ksoftirqd [kernel] 0xaf [<c0107286>] kernel_thread [kernel] 0x26 [<c01213e0>] ksoftirqd [kernel] 0x0 kswapd R 00000000 5356 6 1 7 5 (L-TLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c013920e>] wakeup_memwaiters [kernel] 0xce [<c0138fec>] kswapd [kernel] 0x2bc [<c0105000>] stext [kernel] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<c0138d30>] kswapd [kernel] 0x0 bdflush S 00000000 6368 7 1 8 6 (L-TLB) Call Trace: [<c011911b>] interruptible_sleep_on [kernel] 0x4b [<c014694e>] bdflush [kernel] 0xee [<c0105000>] stext [kernel] 0x0 [<c0105000>] stext [kernel] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<c0146860>] bdflush [kernel] 0x0 kupdated R 00000282 5408 8 1 9 7 (L-TLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c014670c>] sync_old_buffers [kernel] 0x2c [<c01469f4>] kupdate [kernel] 0xa4 [<c0105000>] stext [kernel] 0x0 [<c0105000>] stext [kernel] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<c0146950>] kupdate [kernel] 0x0 mdrecoveryd S 081D3CCE 6328 9 1 13 8 (L-TLB) Call Trace: [<c01d1bb7>] md_thread [kernel] 0x117 [<c0107286>] kernel_thread [kernel] 0x26 [<c01d1aa0>] md_thread [kernel] 0x0 kjournald R 00000000 4808 13 1 113 9 (L-TLB) Call Trace: [<c011911b>] interruptible_sleep_on [kernel] 0x4b [<f881286a>] kjournald [jbd] 0x15a [<f88126f0>] commit_timeout [jbd] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<f8812710>] kjournald [jbd] 0x0 kjournald S C011C78B 5104 113 1 114 13 (L-TLB) Call Trace: [<c011c78b>] call_console_drivers [kernel] 0xeb [<c011911b>] interruptible_sleep_on [kernel] 0x4b [<f881286a>] kjournald [jbd] 0x15a [<f88126f0>] commit_timeout [jbd] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<f8812710>] kjournald [jbd] 0x0 kjournald S C011C78B 352 114 1 115 113 (L-TLB) Call Trace: [<c011c78b>] call_console_drivers [kernel] 0xeb [<c011911b>] interruptible_sleep_on [kernel] 0x4b [<f881286a>] kjournald [jbd] 0x15a [<f88126f0>] commit_timeout [jbd] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<f8812710>] kjournald [jbd] 0x0 kjournald R 00000286 68 115 1 116 114 (L-TLB) Call Trace: [<c0142f46>] __wait_on_buffer [kernel] 0x76 [<f881002f>] journal_commit_transaction [jbd] 0x9df [<c0118ecc>] schedule [kernel] 0x37c [<f8812846>] kjournald [jbd] 0x136 [<f88126f0>] commit_timeout [jbd] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<f8812710>] kjournald [jbd] 0x0 kjournald S 00000000 2400 116 1 543 115 (L-TLB) Call Trace: [<c011911b>] interruptible_sleep_on [kernel] 0x4b [<f881286a>] kjournald [jbd] 0x15a [<f88126f0>] commit_timeout [jbd] 0x0 [<c0107286>] kernel_thread [kernel] 0x26 [<f8812710>] kjournald [jbd] 0x0 syslogd R 00000000 1376 543 1 548 116 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c0108ad1>] do_signal [kernel] 0x231 [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 klogd R F686DD40 1376 548 1 568 543 (NOTLB) Call Trace: [<c011c25d>] do_syslog [kernel] 0xdd [<c0141c96>] sys_read [kernel] 0x96 [<c01206b2>] sys_time [kernel] 0x12 [<c0108c6b>] system_call [kernel] 0x33 portmap S 00000001 8 568 1 596 548 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c0152849>] do_pollfd [kernel] 0x59 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c014171a>] filp_close [kernel] 0x9a [<c0108c6b>] system_call [kernel] 0x33 rpc.statd S F68578E0 1376 596 1 742 568 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c0157d8f>] clear_inode [kernel] 0xf [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 sshd S F686DF40 68 742 1 775 596 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 xinetd S F5CAADA0 0 775 1 1513 812 742 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 snmpdm S F57F4540 68 812 1 833 775 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 gpm R 00000000 5252 833 1 851 812 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 crond R 00000000 68 851 1 921 833 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0108072>] sys_sigreturn [kernel] 0xd2 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 procmgr R 00000000 68 921 1 1675 1078 851 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c0152849>] do_pollfd [kernel] 0x59 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 MasterClerk R 00000000 2400 922 921 923 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 ProcWatch R 00000282 68 923 921 924 922 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0165e86>] proc_pid_read_maps [kernel] 0x226 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0141d0f>] sys_read [kernel] 0x10f [<c0108c6b>] system_call [kernel] 0x33 SysState R 00000000 68 924 921 925 923 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 appOam R 00000000 68 925 921 926 924 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 cdrGen R 00000000 68 926 921 927 925 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 faultmon D F6854800 68 927 921 928 926 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f8812e1a>] log_wait_commit_Rsmp_e7e6dd91 [jbd] 0x5a [<f880e849>] journal_stop_Rsmp_d72faea2 [jbd] 0x199 [<f880e8aa>] journal_force_commit_Rsmp_88c2e786 [jbd] 0x4a [<f8827462>] ext3_force_commit [ext3] 0x42 [<f881de17>] ext3_sync_file [ext3] 0x87 [<f8820b50>] ext3_writepage [ext3] 0x0 [<c0143532>] sys_fsync [kernel] 0x62 [<c0108c6b>] system_call [kernel] 0x33 idb.autozap R 00000000 68 928 921 929 927 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 idbsvc R 00000000 8 929 921 930 928 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c012e6cf>] do_munmap [kernel] 0x28f [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 idbsvc R 00000000 68 930 921 931 929 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c012e6cf>] do_munmap [kernel] 0x28f [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 inetsync R 00000000 0 931 921 932 930 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 java S C01334D0 964 932 921 1002 933 931 (NOTLB) Call Trace: [<c01334d0>] sys_mprotect [kernel] 0x120 [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 68 933 921 1003 934 932 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 netFailClerk R 00000000 8 934 921 935 933 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 pltOam R 00000000 68 935 921 936 934 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 statclerk R 00000000 68 936 921 937 935 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01dd2bb>] net_rx_action [kernel] 0x1eb [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120f0b>] do_softirq [kernel] 0x6b [<c0108c6b>] system_call [kernel] 0x33 vxiMibSnmpAgt S 00000000 2400 937 921 938 936 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 vxiSnmpTrapAg R 00000000 8 938 921 941 937 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 Congest R 00000000 0 941 921 944 938 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c010e423>] sys_ipc [kernel] 0x1b3 [<c0108c6b>] system_call [kernel] 0x33 TCAPAgent R 00000000 8 944 921 945 941 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<f8816d50>] .rodata.str1.1 [jbd] 0x30 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c0158bc1>] update_atime [kernel] 0x51 [<c014b72c>] pipe_read [kernel] 0x21c [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 bsmAgent R 00000000 68 945 921 999 946 944 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 controller S 00000000 2400 946 921 966 947 945 (NOTLB) Call Trace: [<c010e652>] sys_pause [kernel] 0x12 [<c0108c6b>] system_call [kernel] 0x33 cpHelper R 00000000 2400 947 921 948 946 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0141d0f>] sys_read [kernel] 0x10f [<c0108c6b>] system_call [kernel] 0x33 iuaAgent R 00000000 0 948 921 949 947 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<f8816d50>] .rodata.str1.1 [jbd] 0x30 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c014b94a>] pipe_write [kernel] 0x20a [<c0141db6>] sys_write [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 priAgent R 00000000 8 949 921 950 948 (NOTLB) Call Trace: [<c0107925>] __down_interruptible [kernel] 0x75 [<c0107a03>] __down_failed_interruptible [kernel] 0x7 [<c014c0e3>] .text.lock.pipe [kernel] 0x19 [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 sctpDaemon R 00000000 2400 950 921 951 949 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c0152849>] do_pollfd [kernel] 0x59 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 sipAgent R 00000000 0 951 921 952 950 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<c010829a>] setup_sigcontext [kernel] 0xda [<f8816d50>] .rodata.str1.1 [jbd] 0x30 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c0158bc1>] update_atime [kernel] 0x51 [<c014b72c>] pipe_read [kernel] 0x21c [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 talonAdmin R 00000000 0 952 921 953 951 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0108c6b>] system_call [kernel] 0x33 tqtool R 00000000 8 953 921 1613 952 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<c010829a>] setup_sigcontext [kernel] 0xda [<f8816d50>] .rodata.str1.1 [jbd] 0x30 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c0158bc1>] update_atime [kernel] 0x51 [<c014b72c>] pipe_read [kernel] 0x21c [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 controller R 00000000 4 966 946 972 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0118ecc>] schedule [kernel] 0x37c [<c0108c6b>] system_call [kernel] 0x33 controller R 00000000 8 967 966 969 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 controller S 00000000 0 969 966 970 967 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 controller R 00000000 68 970 966 1281 971 969 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 controller S 00000000 8 971 966 972 970 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c011a3b9>] ll_copy_to_user [kernel] 0x49 [<c01daaa6>] wait_for_packet [kernel] 0xe6 [<c01dabcf>] skb_recv_datagram [kernel] 0xcf [<c020fb69>] udp_recvmsg [kernel] 0x59 [<c021c40f>] fib_lookup [kernel] 0xff [<c0216359>] inet_recvmsg [kernel] 0x39 [<c01d54c1>] sock_recvmsg [kernel] 0x31 [<c01d6481>] sys_recvfrom [kernel] 0xa1 [<c0125e3e>] ignored_signal [kernel] 0x2e [<c01260e2>] send_sig_info [kernel] 0x82 [<c0126406>] kill_something_info [kernel] 0x156 [<c02118e4>] arp_rcv [kernel] 0x94 [<c0126c99>] sys_kill [kernel] 0x49 [<c01d6bcb>] sys_socketcall [kernel] 0x17b [<c0108c6b>] system_call [kernel] 0x33 controller S 00000000 2400 972 966 971 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 bsmAgent R 00000000 5612 999 945 1000 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120f0b>] do_softirq [kernel] 0x6b [<c0108c6b>] system_call [kernel] 0x33 bsmAgent S 00000000 5448 1000 999 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 5612 1002 932 1140 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120070>] sys_wait4 [kernel] 0x3e0 [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 5284 1003 933 1142 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120070>] sys_wait4 [kernel] 0x3e0 [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 2404 1004 1002 1008 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 5456 1005 1003 1007 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 1376 1007 1003 1009 1005 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S C0107F75 2404 1008 1002 1010 1004 (NOTLB) Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S C0107F75 0 1009 1003 1031 1007 (NOTLB) Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S C0107F75 2404 1010 1002 1030 1008 (NOTLB) Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 5376 1030 1002 1032 1010 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 2400 1031 1003 1033 1009 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0126805>] sys_rt_sigprocmask [kernel] 0x165 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1032 1002 1034 1030 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1033 1003 1035 1031 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1034 1002 1036 1032 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1035 1003 1037 1033 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 5284 1036 1002 1064 1034 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 5212 1037 1003 1065 1035 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S C022ED72 4 1064 1002 1067 1036 (NOTLB) Call Trace: [<c022ed72>] vsnprintf [kernel] 0x2a2 [<c0125444>] schedule_timeout [kernel] 0x14 [<c01fbdd7>] wait_for_connect [kernel] 0x117 [<c01fbf53>] tcp_accept [kernel] 0xb3 [<c021611b>] inet_accept [kernel] 0x2b [<c01d60b3>] sys_accept [kernel] 0x63 [<c0118f79>] __wake_up [kernel] 0x49 [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c01d6b00>] sys_socketcall [kernel] 0xb0 [<c0108c6b>] system_call [kernel] 0x33 java S C022ED72 8 1065 1003 1068 1037 (NOTLB) Call Trace: [<c022ed72>] vsnprintf [kernel] 0x2a2 [<c0125444>] schedule_timeout [kernel] 0x14 [<c01fbdd7>] wait_for_connect [kernel] 0x117 [<c01fbf53>] tcp_accept [kernel] 0xb3 [<c021611b>] inet_accept [kernel] 0x2b [<c01d60b3>] sys_accept [kernel] 0x63 [<c0118f79>] __wake_up [kernel] 0x49 [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c01d6b00>] sys_socketcall [kernel] 0xb0 [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 0 1067 1002 1069 1064 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 0 1068 1003 1070 1065 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1069 1002 1074 1067 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java S F54E8360 6012 1070 1003 1075 1068 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 java R FFFFFFFF 5668 1074 1002 1077 1069 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0108072>] sys_sigreturn [kernel] 0xd2 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 5600 1075 1003 1076 1070 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0108072>] sys_sigreturn [kernel] 0xd2 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 4532 1076 1003 1083 1075 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java R 00000000 4532 1077 1002 1082 1074 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 xfs S 00000000 4872 1078 1 1100 921 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 java S F192FD20 5656 1082 1002 1136 1077 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c0139dd2>] __alloc_pages [kernel] 0x72 [<c01fbdd7>] wait_for_connect [kernel] 0x117 [<c01fbf53>] tcp_accept [kernel] 0xb3 [<c021611b>] inet_accept [kernel] 0x2b [<c01d60b3>] sys_accept [kernel] 0x63 [<c011780d>] do_page_fault [kernel] 0x12d [<c0118f79>] __wake_up [kernel] 0x49 [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c01d6b00>] sys_socketcall [kernel] 0xb0 [<c01176e0>] do_page_fault [kernel] 0x0 [<c0108d5c>] error_code [kernel] 0x34 [<c0108c6b>] system_call [kernel] 0x33 java S C022ED72 5612 1083 1003 1137 1076 (NOTLB) Call Trace: [<c022ed72>] vsnprintf [kernel] 0x2a2 [<c0125444>] schedule_timeout [kernel] 0x14 [<c01fbdd7>] wait_for_connect [kernel] 0x117 [<c01fbf53>] tcp_accept [kernel] 0xb3 [<c021611b>] inet_accept [kernel] 0x2b [<c01d60b3>] sys_accept [kernel] 0x63 [<c011780d>] do_page_fault [kernel] 0x12d [<c0118f79>] __wake_up [kernel] 0x49 [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c01d6b00>] sys_socketcall [kernel] 0xb0 [<c01176e0>] do_page_fault [kernel] 0x0 [<c0108d5c>] error_code [kernel] 0x34 [<c0108c6b>] system_call [kernel] 0x33 atd S 00001000 5252 1100 1 1107 1078 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 mingetty S C1E130E0 4820 1107 1 1108 1100 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01875b9>] set_cursor [kernel] 0x69 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c012e6cf>] do_munmap [kernel] 0x28f [<c0141c96>] sys_read [kernel] 0x96 [<c012e715>] sys_munmap [kernel] 0x35 [<c0108c6b>] system_call [kernel] 0x33 mingetty S F5661D40 0 1108 1 1109 1107 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c012e6cf>] do_munmap [kernel] 0x28f [<c0141c96>] sys_read [kernel] 0x96 [<c012e715>] sys_munmap [kernel] 0x35 [<c0108c6b>] system_call [kernel] 0x33 mingetty S F5661E80 2060 1109 1 1110 1108 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c012e6cf>] do_munmap [kernel] 0x28f [<c0141c96>] sys_read [kernel] 0x96 [<c012e715>] sys_munmap [kernel] 0x35 [<c0108c6b>] system_call [kernel] 0x33 mingetty S F786D620 764 1110 1 1111 1109 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c012e6cf>] do_munmap [kernel] 0x28f [<c0141c96>] sys_read [kernel] 0x96 [<c012e715>] sys_munmap [kernel] 0x35 [<c0108c6b>] system_call [kernel] 0x33 mingetty S F786D8A0 0 1111 1 1112 1110 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c012e6cf>] do_munmap [kernel] 0x28f [<c0141c96>] sys_read [kernel] 0x96 [<c012e715>] sys_munmap [kernel] 0x35 [<c0108c6b>] system_call [kernel] 0x33 mingetty S F5661CA0 4 1112 1 1113 1111 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c012e6cf>] do_munmap [kernel] 0x28f [<c0141c96>] sys_read [kernel] 0x96 [<c012e715>] sys_munmap [kernel] 0x35 [<c0108c6b>] system_call [kernel] 0x33 kdm S F1AC3B20 4504 1113 1 1131 1471 1112 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 X S 00000000 8 1130 1113 1131 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 kdm S 00000000 96 1131 1113 1171 1130 (NOTLB) Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c [<c014b5c4>] pipe_read [kernel] 0xb4 [<c0141c96>] sys_read [kernel] 0x96 [<c0108c6b>] system_call [kernel] 0x33 java S FFFFFFFF 4576 1136 1002 1140 1082 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0108072>] sys_sigreturn [kernel] 0xd2 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java S FFFFFFFF 4576 1137 1003 1142 1083 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c0108072>] sys_sigreturn [kernel] 0xd2 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1140 1002 1136 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 java S 00000000 6012 1142 1003 1137 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01255fb>] sys_nanosleep [kernel] 0x11b [<c0108c6b>] system_call [kernel] 0x33 kdm_greet R 00000000 4452 1171 1131 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c01d5891>] sock_ioctl [kernel] 0x51 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1190 970 1191 1260 (NOTLB) Call Trace: [<c010e652>] sys_pause [kernel] 0x12 [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 5332 1191 1190 1223 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120070>] sys_wait4 [kernel] 0x3e0 [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 4768 1192 1191 1193 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000012 5172 1193 1191 1194 1192 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 5320 1194 1191 1195 1193 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 6056 1195 1191 1196 1194 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 2404 1196 1191 1197 1195 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S EDE62360 6096 1197 1191 1198 1196 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 5280 1198 1191 1199 1197 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c010a54e>] handle_IRQ_event [kernel] 0x5e [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 5220 1199 1191 1200 1198 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 5060 1200 1191 1209 1199 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 taliClient S EDE4BF3C 5600 1209 1191 1211 1200 (NOTLB) Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c [<c014b5c4>] pipe_read [kernel] 0xb4 [<c0141c96>] sys_read [kernel] 0x96 [<c010a79f>] do_IRQ [kernel] 0xdf [<c0108c6b>] system_call [kernel] 0x33 taliClient S 000004B6 5308 1211 1191 1212 1209 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 4632 1212 1191 1223 1211 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01fa380>] tcp_data_wait [kernel] 0x100 [<c01fab9f>] tcp_recvmsg [kernel] 0x51f [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c0216359>] inet_recvmsg [kernel] 0x39 [<c01d54c1>] sock_recvmsg [kernel] 0x31 [<c0142c06>] fput [kernel] 0x16 [<c01d6481>] sys_recvfrom [kernel] 0xa1 [<c0125e3e>] ignored_signal [kernel] 0x2e [<c01f1025>] ip_local_deliver [kernel] 0xf5 [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d8db9>] __kfree_skb [kernel] 0x129 [<c01dcf92>] net_tx_action [kernel] 0x62 [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d64fd>] sys_recv [kernel] 0x1d [<c01d6ba4>] sys_socketcall [kernel] 0x154 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 4772 1223 1191 1212 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1260 970 1261 1281 1190 (NOTLB) Call Trace: [<c010e652>] sys_pause [kernel] 0x12 [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 0 1261 1260 1278 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120070>] sys_wait4 [kernel] 0x3e0 [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 0 1262 1261 1263 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000012 0 1263 1261 1264 1262 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000012 0 1264 1261 1265 1263 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1265 1261 1266 1264 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1266 1261 1267 1265 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S EC276360 0 1267 1261 1268 1266 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 8 1268 1261 1269 1267 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 taliClient S F5999CD4 0 1269 1261 1270 1268 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1270 1261 1272 1269 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 taliClient S EC6F5F3C 2400 1272 1261 1273 1270 (NOTLB) Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c [<c014b5c4>] pipe_read [kernel] 0xb4 [<c0141c96>] sys_read [kernel] 0x96 [<c012d625>] sys_brk [kernel] 0xc5 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000418 0 1273 1261 1274 1272 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R C01D8DB9 0 1274 1261 1275 1273 (NOTLB) Call Trace: [<c01d8db9>] __kfree_skb [kernel] 0x129 [<c0125444>] schedule_timeout [kernel] 0x14 [<c01fa380>] tcp_data_wait [kernel] 0x100 [<c01fab9f>] tcp_recvmsg [kernel] 0x51f [<c0216359>] inet_recvmsg [kernel] 0x39 [<c01d54c1>] sock_recvmsg [kernel] 0x31 [<c01f1025>] ip_local_deliver [kernel] 0xf5 [<c01d6481>] sys_recvfrom [kernel] 0xa1 [<c0120f0b>] do_softirq [kernel] 0x6b [<c010a79f>] do_IRQ [kernel] 0xdf [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d64fd>] sys_recv [kernel] 0x1d [<c01d6ba4>] sys_socketcall [kernel] 0x154 [<c0108c6b>] system_call [kernel] 0x33 taliClient S C0107F75 0 1275 1261 1276 1274 (NOTLB) Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000012 0 1276 1261 1277 1275 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R C01D8DB9 0 1277 1261 1278 1276 (NOTLB) Call Trace: [<c01d8db9>] __kfree_skb [kernel] 0x129 [<c0125444>] schedule_timeout [kernel] 0x14 [<c01fa380>] tcp_data_wait [kernel] 0x100 [<c01fab9f>] tcp_recvmsg [kernel] 0x51f [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c0216359>] inet_recvmsg [kernel] 0x39 [<c01d54c1>] sock_recvmsg [kernel] 0x31 [<c0142c02>] fput [kernel] 0x12 [<c01d6481>] sys_recvfrom [kernel] 0xa1 [<c0118b24>] scheduler_tick [kernel] 0x334 [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d8db9>] __kfree_skb [kernel] 0x129 [<c01dcf92>] net_tx_action [kernel] 0x62 [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d64fd>] sys_recv [kernel] 0x1d [<c01d6ba4>] sys_socketcall [kernel] 0x154 [<c0108c6b>] system_call [kernel] 0x33 taliClient S C0107F75 0 1278 1261 1277 (NOTLB) Call Trace: [<c0107f75>] restore_sigcontext [kernel] 0x115 [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1281 970 1282 1260 (NOTLB) Call Trace: [<c010e652>] sys_pause [kernel] 0x12 [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 0 1282 1281 1296 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01528f6>] do_pollfd [kernel] 0x106 [<c015292e>] do_pollfd [kernel] 0x13e [<c0152ba2>] sys_poll [kernel] 0x252 [<c0120070>] sys_wait4 [kernel] 0x3e0 [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 0 1283 1282 1284 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000012 0 1284 1282 1285 1283 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1285 1282 1286 1284 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1286 1282 1287 1285 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S EC5FA360 0 1287 1282 1288 1286 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient S EC5F8360 0 1288 1282 1289 1287 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R 00000000 0 1289 1282 1290 1288 (NOTLB) Call Trace: [<c01254ac>] schedule_timeout [kernel] 0x7c [<c0125420>] process_timeout [kernel] 0x0 [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c0108c6b>] system_call [kernel] 0x33 taliClient S E7823594 0 1290 1282 1291 1289 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1291 1282 1293 1290 (NOTLB) Call Trace: [<c017649b>] sys_msgrcv [kernel] 0x2bb [<c010e3d0>] sys_ipc [kernel] 0x160 [<c0108c6b>] system_call [kernel] 0x33 taliClient S EC5C9F3C 2400 1293 1282 1294 1291 (NOTLB) Call Trace: [<c014b4dc>] pipe_wait [kernel] 0x7c [<c014b5c4>] pipe_read [kernel] 0xb4 [<c0141c96>] sys_read [kernel] 0x96 [<c010a79f>] do_IRQ [kernel] 0xdf [<c0108c6b>] system_call [kernel] 0x33 taliClient S 000004B6 0 1294 1282 1295 1293 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 taliClient R F04D46D8 0 1295 1282 1296 1294 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01da628>] memcpy_toiovec [kernel] 0x38 [<c01fa380>] tcp_data_wait [kernel] 0x100 [<c01fab9f>] tcp_recvmsg [kernel] 0x51f [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c0216359>] inet_recvmsg [kernel] 0x39 [<c01d54c1>] sock_recvmsg [kernel] 0x31 [<c0142c02>] fput [kernel] 0x12 [<c01d6481>] sys_recvfrom [kernel] 0xa1 [<c01181f9>] wake_up_process [kernel] 0x9 [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d8db9>] __kfree_skb [kernel] 0x129 [<c01dcf92>] net_tx_action [kernel] 0x62 [<c01d8c2c>] kfree_skbmem [kernel] 0xc [<c01d64fd>] sys_recv [kernel] 0x1d [<c01d6ba4>] sys_socketcall [kernel] 0x154 [<c0108c6b>] system_call [kernel] 0x33 taliClient S 00000000 0 1296 1282 1295 (NOTLB) Call Trace: [<c0107d1b>] sys_rt_sigsuspend [kernel] 0x10b [<c0108c6b>] system_call [kernel] 0x33 agetty R 40015000 4776 1471 1 1113 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c01212db>] bh_action [kernel] 0x4b [<c0141c96>] sys_read [kernel] 0x96 [<c010a79f>] do_IRQ [kernel] 0xdf [<c0108c6b>] system_call [kernel] 0x33 in.telnetd R F36AA640 0 1513 775 1514 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c01d58df>] sock_poll [kernel] 0x1f [<c01522ff>] do_select [kernel] 0x1ff [<c01526a9>] sys_select [kernel] 0x339 [<c010a79f>] do_IRQ [kernel] 0xdf [<c0108c6b>] system_call [kernel] 0x33 login S 00000000 0 1514 1513 1515 (NOTLB) Call Trace: [<c0120048>] sys_wait4 [kernel] 0x3b8 [<c0108c6b>] system_call [kernel] 0x33 ksh S 642F6C61 4 1515 1514 (NOTLB) Call Trace: [<c0125444>] schedule_timeout [kernel] 0x14 [<c017ecba>] read_chan [kernel] 0x37a [<c017ab87>] tty_read [kernel] 0xd7 [<c017c802>] tty_ioctl [kernel] 0x372 [<c0141c96>] sys_read [kernel] 0x96 [<c01517d4>] sys_ioctl [kernel] 0x294 [<c0108c6b>] system_call [kernel] 0x33 Ss7Agent D 00000000 2400 1613 921 1614 953 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<c010829a>] setup_sigcontext [kernel] 0xda [<c01086a9>] setup_rt_frame [kernel] 0x1b9 [<f8816d50>] .rodata.str1.1 [jbd] 0x30 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c0158bc1>] update_atime [kernel] 0x51 [<c014b72c>] pipe_read [kernel] 0x21c [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 SwitchAgent D 00000000 0 1614 921 1675 1613 (NOTLB) Call Trace: [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<f8816d50>] .rodata.str1.1 [jbd] 0x30 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c0158bc1>] update_atime [kernel] 0x51 [<c014b72c>] pipe_read [kernel] 0x21c [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33 CP R current 0 1675 921 1614 (NOTLB) Call Trace: [<c0118efb>] schedule [kernel] 0x3ab [<c011920b>] sleep_on [kernel] 0x4b [<f880db2b>] do_get_write_access [jbd] 0x2db [<f880de17>] journal_get_write_access_Rsmp_71c823fa [jbd] 0x37 [<f8822851>] ext3_reserve_inode_write [ext3] 0x31 [<f8816d20>] .rodata.str1.1 [jbd] 0x0 [<f8814137>] __jbd_kmalloc [jbd] 0x27 [<f880d11e>] get_transaction [jbd] 0xbe [<f88228e8>] ext3_mark_inode_dirty [ext3] 0x18 [<f88229a8>] ext3_dirty_inode [ext3] 0x98 [<c015723e>] __mark_inode_dirty [kernel] 0x2e [<c0158bc1>] update_atime [kernel] 0x51 [<c014b72c>] pipe_read [kernel] 0x21c [<c0141c96>] sys_read [kernel] 0x96 [<c01207bb>] sys_gettimeofday [kernel] 0x1b [<c0108c6b>] system_call [kernel] 0x33