Attachment 144819 Details for Bug 217356 – system lockdump

system lockdump

b03-lockdump-4 (text/plain), 107.88 KB, created by Russell Cattelan on 2007-01-04 17:07:49 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Russell Cattelan

Created: 2007-01-04 17:07:49 UTC

Size: 107.88 KB

patch

obsolete

>Script started on Wed 03 Jan 2007 01:59:37 PM CST
>
>cattelan has logged on :0 from local.
>cattelan has logged on pts/1 from localhost.
>cattelan has logged on tty1 from local.
>xenon[1:59pm]-=>

>xenon[1:59pm]-=>telnet bear-03-ts

>Trying 10.15.86.228...

>Connected to bear-03-ts.lab.msp.redhat.com (10.15.86.228).

>Escape character is '^]'.

>
>Fedora Core release 6 (Rawhide)
>Kernel 2.6.20-rc3+kdb on an i686
>
>bear-03.lab.msp.redhat.com login: [143148.720000] SysRq : Show State
>[143148.720000] 
>[143148.720000]                          free                        sibling
>[143148.728000]   task             PC    stack   pid father child younger older
>[143148.736000] init          S C392FB30     0     1      0     2               (NOTLB)
>[143148.744000]        c392fb44 00000086 00000002 c392fb30 c392fb2c 00000000 00000000 c392fb04 
>[143148.752000]        c037c997 00000000 0000000a c392d5b0 bbc08f00 00008230 00000000 c392d6d4 
>[143148.760000]        c3821780 00000000 0220e9e2 00000000 00000286 c392fb54 c392fb44 c012c2e5 
>[143148.768000] Call Trace:
>[143148.772000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143148.776000]  [<c0175c30>] do_select+0x466/0x4ce
>[143148.784000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143148.788000]  [<c01764e1>] sys_select+0x99/0x16c
>[143148.792000]  [<c0102eac>] syscall_call+0x7/0xb
>[143148.796000]  =======================
>[143148.800000] migration/0   S C3937F80     0     2      1             3       (L-TLB)
>[143148.808000]        c3937f94 00000046 00000002 c3937f80 c3937f7c 00000000 000081e5 c010393c 
>[143148.816000]        00000092 f0459f18 00000001 c392c030 a7a37000 000081e5 00000000 c392c154 
>[143148.824000]        c3821780 00000000 021faefb 00000000 c3937f78 00000282 f0459f50 c3937f94 
>[143148.832000] Call Trace:
>[143148.836000]  [<c0120c0d>] migration_thread+0x164/0x1eb
>[143148.840000]  [<c0134edb>] kthread+0xb3/0xdc
>[143148.844000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143148.852000]  =======================
>[143148.856000] ksoftirqd/0   S F6708734     0     3      1             4     2 (L-TLB)
>[143148.864000]        c393bfb8 00000046 00000246 f6708734 f6708730 00000000 c3822800 00000004 
>[143148.872000]        c393bf68 c016bb28 0000000a c39395f0 0c7f0d00 00006c4b 00000000 c3939714 
>[143148.880000]        c3821780 00000000 c012836b 00000001 c0504b28 c393bfac c0127f82 00000000 
>[143148.888000] Call Trace:
>[143148.892000]  [<c0128285>] ksoftirqd+0x42/0xc5
>[143148.896000]  [<c0134edb>] kthread+0xb3/0xdc
>[143148.900000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143148.904000]  =======================
>[143148.908000] watchdog/0    S C393DFA4     0     4      1             5     3 (L-TLB)
>[143148.916000]        c393dfb8 00000046 00000002 c393dfa4 c393dfa0 00000000 00000000 00000000 
>[143148.924000]        c014be5a c393df80 00000001 c3939090 3ee1e500 00008231 00000000 c39391b4 
>[143148.932000]        c3821780 00000000 0220ec08 00000000 c3939090 c392ff18 c014be5a c38217d4 
>[143148.940000] Call Trace:
>[143148.944000]  [<c014bea1>] watchdog+0x47/0x56
>[143148.948000]  [<c0134edb>] kthread+0xb3/0xdc
>[143148.952000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143148.956000]  =======================
>[143148.960000] migration/1   S C393FF80     0     5      1             6     4 (L-TLB)
>[143148.968000]        c393ff94 00000046 00000002 c393ff80 c393ff7c 00000000 000081e5 c010393c 
>[143148.976000]        00000092 f0459f18 00000001 c3938b30 a7e07900 000081e5 00000000 c3938c54 
>[143148.988000]        c3829780 00000001 021faefb 00000000 c393ff78 00000282 f0459f50 c393ff94 
>[143148.996000] Call Trace:
>[143148.996000]  [<c0120c0d>] migration_thread+0x164/0x1eb
>[143149.004000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.008000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.012000]  =======================
>[143149.016000] ksoftirqd/1   S C3945FA4     0     6      1             7     5 (L-TLB)
>[143149.024000]        c3945fb8 00000046 00000002 c3945fa4 c3945fa0 00000000 c382a800 00000004 
>[143149.032000]        c3945f68 c016bb28 0000000a c39385d0 20c52500 00003f29 00000000 c39386f4 
>[143149.040000]        c3829780 00000001 0107c53f 00000000 c0504b28 c3945fac c0127f82 00000001 
>[143149.048000] Call Trace:
>[143149.052000]  [<c0128285>] ksoftirqd+0x42/0xc5
>[143149.056000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.060000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.064000]  =======================
>[143149.068000] watchdog/1    S C3947FA4     0     7      1             8     6 (L-TLB)
>[143149.076000]        c3947fb8 00000046 00000002 c3947fa4 c3947fa0 00000000 00000000 00000292 
>[143149.084000]        c3938530 f7f3da18 00000001 c3938070 4b47b900 00008231 00000000 c3938194 
>[143149.092000]        c3829780 00000001 0220ec3c 00000000 0000007b 0000007b 000000d8 c38297d4 
>[143149.104000] Call Trace:
>[143149.104000]  [<c014bea1>] watchdog+0x47/0x56
>[143149.108000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.112000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.120000]  =======================
>[143149.124000] migration/2   S C394BF80     0     8      1             9     7 (L-TLB)
>[143149.132000]        c394bf94 00000046 00000002 c394bf80 c394bf7c 00000000 00000600 f5c500f0 
>[143149.140000]        c394bf94 c011d9cf 00000008 c3949630 d627b400 0000822f 00000000 c3949754 
>[143149.148000]        c3831780 00000002 0220e602 00000000 00000001 c3831c4c c38297d4 c3831c4c 
>[143149.156000] Call Trace:
>[143149.160000]  [<c0120c0d>] migration_thread+0x164/0x1eb
>[143149.164000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.168000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.172000]  =======================
>[143149.176000] ksoftirqd/2   S C396FFA4     0     9      1            10     8 (L-TLB)
>[143149.184000]        c396ffb8 00000046 00000002 c396ffa4 c396ffa0 00000000 c3832800 00000004 
>[143149.192000]        c396ff68 c016bb28 0000000a c39490d0 b18aed00 00006c2d 00000000 c39491f4 
>[143149.200000]        c3831780 00000002 01c4969d 00000000 c0504b28 c396ffac c0127f82 00000002 
>[143149.208000] Call Trace:
>[143149.212000]  [<c0128285>] ksoftirqd+0x42/0xc5
>[143149.216000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.220000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.228000]  =======================
>[143149.232000] watchdog/2    S C3971FA4     0    10      1            11     9 (L-TLB)
>[143149.240000]        c3971fb8 00000046 00000002 c3971fa4 c3971fa0 00000000 c3971f7c 00000292 
>[143149.248000]        c3949030 00000292 00000007 c3948b70 790e7900 00008231 00000000 c3948c94 
>[143149.256000]        c3831780 00000002 0220ecfc 00000000 c011e955 c3971fc0 00000001 c3831c4c 
>[143149.264000] Call Trace:
>[143149.268000]  [<c014bea1>] watchdog+0x47/0x56
>[143149.272000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.276000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.280000]  =======================
>[143149.284000] migration/3   S 032D6980     0    11      1            12    10 (L-TLB)
>[143149.292000]        c3975f94 00000046 0000033d 032d6980 c3975f7c a7a37000 000081e5 c010393c 
>[143149.300000]        00000092 f0459f18 00000005 c3973670 a7a37000 000081e5 00000000 c3973794 
>[143149.308000]        c3839780 00000003 f0459f54 00000282 c3975f78 00000282 f0459f50 c3975f94 
>[143149.316000] Call Trace:
>[143149.320000]  [<c0120c0d>] migration_thread+0x164/0x1eb
>[143149.324000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.328000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.336000]  =======================
>[143149.336000] ksoftirqd/3   S F7E1BFA4     0    12      1            13    11 (L-TLB)
>[143149.344000]        f7e1bfb8 00000046 00000002 f7e1bfa4 f7e1bfa0 00000000 f7e1bf68 c017c7ad 
>[143149.356000]        00000000 c383a800 00000009 c3973110 56854100 00003dd2 003d0900 c3973234 
>[143149.364000]        c3839780 00000003 01022783 00000000 c0504b28 f7e1bfac c0127f82 00000003 
>[143149.372000] Call Trace:
>[143149.372000]  [<c0128285>] ksoftirqd+0x42/0xc5
>[143149.380000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.384000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.388000]  =======================
>[143149.392000] watchdog/3    S F7E1DFA4     0    13      1            14    12 (L-TLB)
>[143149.400000]        f7e1dfb8 00000046 00000002 f7e1dfa4 f7e1dfa0 00000000 00000000 00000000 
>[143149.408000]        c014be5a f7e1df80 00000003 c3972bb0 7fbb7500 00008231 00000000 c3972cd4 
>[143149.416000]        c3839780 00000003 0220ed18 00000000 c3972bb0 c392fed8 c014be5a c3839c4c 
>[143149.424000] Call Trace:
>[143149.428000]  [<c014bea1>] watchdog+0x47/0x56
>[143149.432000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.436000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.440000]  =======================
>[143149.444000] events/0      S F7E23F70     0    14      1            15    13 (L-TLB)
>[143149.452000]        f7e23f84 00000046 00000002 f7e23f70 f7e23f6c 00000000 c3979f00 00000082 
>[143149.460000]        f7e23f38 00000082 0000000a f7e216b0 0215ee00 00008231 00000000 f7e217d4 
>[143149.468000]        c3821780 00000000 0220eb09 00000000 c3979ee8 00000246 f7e23f70 00000246 
>[143149.476000] Call Trace:
>[143149.480000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.484000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.488000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.496000]  =======================
>[143149.500000] events/1      S F7E43F70     0    15      1            16    14 (L-TLB)
>[143149.508000]        f7e43f84 00000046 00000002 f7e43f70 f7e43f6c 00000000 c3979e74 00000082 
>[143149.516000]        f7e43f38 00000082 0000000a f7e21150 3cf99d00 00008231 00000000 f7e21274 
>[143149.524000]        c3829780 00000001 0220ec00 00000000 c3979e5c 00000246 f7e43f70 00000246 
>[143149.532000] Call Trace:
>[143149.536000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.540000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.544000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.548000]  =======================
>[143149.552000] events/2      S F7E45F70     0    16      1            17    15 (L-TLB)
>[143149.560000]        f7e45f84 00000046 00000002 f7e45f70 f7e45f6c 00000000 c3979de8 00000082 
>[143149.568000]        f7e45f38 00000082 0000000a f7e20bf0 3c428200 00008231 00000000 f7e20d14 
>[143149.576000]        c3831780 00000002 0220ebfd 00000000 c3979dd0 00000246 f7e45f70 00000246 
>[143149.584000] Call Trace:
>[143149.588000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.592000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.596000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.604000]  =======================
>[143149.608000] events/3      S F7E47F70     0    17      1            18    16 (L-TLB)
>[143149.616000]        f7e47f84 00000046 00000002 f7e47f70 f7e47f6c 00000000 c3979d5c 00000082 
>[143149.624000]        f7e47f38 00000082 0000000a f7e20690 77263100 00008231 00000000 f7e207b4 
>[143149.632000]        c3839780 00000003 0220ecf4 00000000 c3979d44 00000246 f7e47f70 00000246 
>[143149.640000] Call Trace:
>[143149.644000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.648000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.652000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.656000]  =======================
>[143149.660000] khelper       S 0000007B     0    18      1            19    17 (L-TLB)
>[143149.668000]        f7e49f84 00000046 00000000 0000007b 0000007b 00000003 c3979c44 00000082 
>[143149.676000]        f7e49f38 00000082 0000000a f7e20130 22d75400 0000002f 00000000 f7e20254 
>[143149.684000]        c3831780 00000002 c3979c24 00000246 c3979c2c 00000246 f7e49f70 00000246 
>[143149.692000] Call Trace:
>[143149.696000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.700000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.704000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.712000]  =======================
>[143149.716000] kthread       S F7E4DF70     0    19      1    69     474    18 (L-TLB)
>[143149.724000]        f7e4df84 00000046 00000002 f7e4df70 f7e4df6c 00000000 f7e31640 00000082 
>[143149.732000]        f7e4df38 00000082 0000000a f7e4b6f0 525f1200 0000003c 00000000 f7e4b814 
>[143149.740000]        c3821780 00000000 ffffd80d 00000000 f7e31628 00000246 f7e4df70 00000246 
>[143149.748000] Call Trace:
>[143149.752000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.756000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.760000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.764000]  =======================
>[143149.768000] kblockd/0     S C02692C6     0    69     19            70       (L-TLB)
>[143149.776000]        f7c13f84 00000046 f7c13f20 c02692c6 f7c8dcac 00000003 f7e7e758 00000082 
>[143149.784000]        f7c13f38 00000082 0000000a f7c11970 a911a600 000081e5 00000000 f7c11a94 
>[143149.792000]        c3821780 00000000 f7e7e738 00000246 f7e7e740 00000246 f7c13f70 00000246 
>[143149.800000] Call Trace:
>[143149.804000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.808000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.812000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.820000]  =======================
>[143149.824000] kblockd/1     S C02692C6     0    70     19            71    69 (L-TLB)
>[143149.828000]        f7c15f84 00000046 f7c15f20 c02692c6 f7c8dcac 00000003 f7e7e6cc 00000082 
>[143149.840000]        f7c15f38 00000082 0000000a f7c11410 1a805e00 00006c4a 00000000 f7c11534 
>[143149.848000]        c3829780 00000001 f7e7e6ac 00000246 f7e7e6b4 00000246 f7c15f70 00000246 
>[143149.856000] Call Trace:
>[143149.856000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.864000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.868000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.872000]  =======================
>[143149.876000] kblockd/2     S C02692C6     0    71     19            72    70 (L-TLB)
>[143149.884000]        f7c17f84 00000046 f7c17f20 c02692c6 f7c8dcac 00000003 f7e7e640 00000082 
>[143149.892000]        f7c17f38 00000082 0000000a f7c10eb0 f993fa00 00007fb6 00000000 f7c10fd4 
>[143149.900000]        c3831780 00000002 f7e7e620 00000246 f7e7e628 00000246 f7c17f70 00000246 
>[143149.908000] Call Trace:
>[143149.912000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.916000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.920000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.928000]  =======================
>[143149.928000] kblockd/3     S C02692C6     0    72     19            73    71 (L-TLB)
>[143149.936000]        f7c19f84 00000046 f7c19f20 c02692c6 f7c8dcac 00000003 f7e7e5b4 00000082 
>[143149.948000]        f7c19f38 00000082 0000000a f7c10950 9b149a00 00007c70 00000000 f7c10a74 
>[143149.956000]        c3839780 00000003 f7e7e594 00000246 f7e7e59c 00000246 f7c19f70 00000246 
>[143149.964000] Call Trace:
>[143149.964000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143149.972000]  [<c0134edb>] kthread+0xb3/0xdc
>[143149.976000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143149.980000]  =======================
>[143149.984000] kacpid        S C392D5B0     0    73     19           184    72 (L-TLB)
>[143149.992000]        f7c1df84 00000046 c3839780 c392d5b0 f7c103f0 f7c1df44 c0101fce c392d770 
>[143150.000000]        f7c105b0 c3836a80 00000001 f7c103f0 0baa3073 00000000 000005ee f7c10514 
>[143150.008000]        c3839780 00000003 c39eae34 00000246 f7e7e13c 00000246 f7c1df70 c38397d4 
>[143150.016000] Call Trace:
>[143150.020000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143150.024000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.028000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.036000]  =======================
>[143150.036000] ksuspend_usbd S F7E67F70     0   184     19           187    73 (L-TLB)
>[143150.044000]        f7e67f84 00000046 00000002 f7e67f70 f7e67f6c 00000000 c0101fce c392d770 
>[143150.052000]        c39d9590 c3836a80 00000001 c39d93d0 0e241fc5 00000000 000007e7 c39d94f4 
>[143150.064000]        c3839780 00000003 fffedbc5 00000000 f7cae8e4 00000246 f7e67f70 00000246 
>[143150.072000] Call Trace:
>[143150.072000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143150.080000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.084000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.088000]  =======================
>[143150.092000] khubd         S F7E63EEC     0   187     19           189   184 (L-TLB)
>[143150.100000]        f7e63f28 00000046 00000004 f7e63eec c0334314 00040004 000000a3 00000000 
>[143150.108000]        80000180 f7e18608 00000006 c39d8e70 376fac00 00000002 00000000 c39d8f94 
>[143150.116000]        c3839780 00000003 00000246 c04ebfc0 00000246 f7e63f10 00000246 f7e63f98 
>[143150.124000] Call Trace:
>[143150.128000]  [<c0331b2e>] hub_thread+0x8e9/0x951
>[143150.132000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.136000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.140000]  =======================
>[143150.144000] kseriod       S F7E61F74     0   189     19           237   187 (L-TLB)
>[143150.152000]        f7e61f88 00000046 00000002 f7e61f74 f7e61f70 00000000 c04ee260 f7fff300 
>[143150.160000]        00000000 f7e61f64 00000007 c39d8910 3bba4e00 00000002 00000000 c39d8a34 
>[143150.172000]        c3821780 00000000 fffee46f 00000000 00000246 f7e61f70 00000246 f7e61fa4 
>[143150.180000] Call Trace:
>[143150.180000]  [<c03430dd>] serio_thread+0x260/0x2ab
>[143150.188000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.192000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.196000]  =======================
>[143150.200000] rt-test-0     S F7E5FFA0     0   237     19           239   189 (L-TLB)
>[143150.208000]        f7e5ffb4 00000046 00000002 f7e5ffa0 f7e5ff9c 00000000 c054ae00 c3839780 
>[143150.216000]        c383a320 4247c600 00000007 c39d83b0 4247c600 00000000 00000000 c39d84d4 
>[143150.224000]        c3839780 00000003 fffedc21 00000000 c3839780 c39ea804 00000001 c057f180 
>[143150.232000] Call Trace:
>[143150.236000]  [<c013c329>] test_func+0x56/0x95
>[143150.240000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.244000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.248000]  =======================
>[143150.252000] rt-test-1     S C398FFA0     0   239     19           241   237 (L-TLB)
>[143150.260000]        c398ffb4 00000046 00000002 c398ffa0 c398ff9c 00000000 c054ae00 c3839780 
>[143150.268000]        c383a320 4247c600 00000007 f7e4b190 4247c600 00000000 00000000 f7e4b2b4 
>[143150.276000]        c3839780 00000003 fffedc21 00000000 c3839780 c39ea678 00000001 c057f204 
>[143150.284000] Call Trace:
>[143150.288000]  [<c013c329>] test_func+0x56/0x95
>[143150.292000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.296000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.304000]  =======================
>[143150.308000] rt-test-2     S C398DFA0     0   241     19           243   239 (L-TLB)
>[143150.316000]        c398dfb4 00000046 00000002 c398dfa0 c398df9c 00000000 c054ae00 c3839780 
>[143150.324000]        c383a320 4247c600 00000007 f7c1f9b0 4247c600 00000000 00000000 f7c1fad4 
>[143150.332000]        c3839780 00000003 fffedc21 00000000 c3839780 c39ea4ec 00000001 c057f288 
>[143150.340000] Call Trace:
>[143150.344000]  [<c013c329>] test_func+0x56/0x95
>[143150.348000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.352000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.356000]  =======================
>[143150.360000] rt-test-3     S C398BFA0     0   243     19           245   241 (L-TLB)
>[143150.368000]        c398bfb4 00000046 00000002 c398bfa0 c398bf9c 00000000 c054ae00 c3839780 
>[143150.376000]        c383a320 4247c600 00000007 f7c1f450 4247c600 00000000 00000000 f7c1f574 
>[143150.384000]        c3839780 00000003 fffedc21 00000000 c3839780 c39ea360 00000001 c057f30c 
>[143150.392000] Call Trace:
>[143150.396000]  [<c013c329>] test_func+0x56/0x95
>[143150.400000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.404000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.408000]  =======================
>[143150.412000] rt-test-4     S F7E69FA0     0   245     19           247   243 (L-TLB)
>[143150.420000]        f7e69fb4 00000046 00000002 f7e69fa0 f7e69f9c 00000000 c054ae00 c3839780 
>[143150.428000]        c383a320 4247c600 00000007 f7c1eef0 4247c600 00000000 00000000 f7c1f014 
>[143150.436000]        c3839780 00000003 fffedc21 00000000 c3839780 c39ea1d4 00000001 c057f390 
>[143150.448000] Call Trace:
>[143150.448000]  [<c013c329>] test_func+0x56/0x95
>[143150.452000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.456000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.464000]  =======================
>[143150.468000] rt-test-5     S F7C8FFA0     0   247     19           249   245 (L-TLB)
>[143150.476000]        f7c8ffb4 00000046 00000002 f7c8ffa0 f7c8ff9c 00000000 c054ae00 c3839780 
>[143150.484000]        c383a320 4247c600 00000007 f7c1e990 4247c600 00000000 00000000 f7c1eab4 
>[143150.492000]        c3839780 00000003 fffedc21 00000000 c3839780 c39ea048 00000001 c057f414 
>[143150.500000] Call Trace:
>[143150.504000]  [<c013c329>] test_func+0x56/0x95
>[143150.508000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.512000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.516000]  =======================
>[143150.520000] rt-test-6     S F7C87FA0     0   249     19           251   247 (L-TLB)
>[143150.528000]        f7c87fb4 00000046 00000002 f7c87fa0 f7c87f9c 00000000 c054ae00 c3839780 
>[143150.536000]        c383a320 4247c600 00000007 f7c2c9d0 4247c600 00000000 00000000 f7c2caf4 
>[143150.544000]        c3839780 00000003 fffedc21 00000000 c3839780 f7fe1e34 00000001 c057f498 
>[143150.552000] Call Trace:
>[143150.556000]  [<c013c329>] test_func+0x56/0x95
>[143150.560000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.564000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.572000]  =======================
>[143150.572000] rt-test-7     S C39BFFA0     0   251     19           254   249 (L-TLB)
>[143150.580000]        c39bffb4 00000046 00000002 c39bffa0 c39bff9c 00000000 c054ae00 c3839780 
>[143150.588000]        c383a320 4247c600 00000007 f7c2c470 4247c600 00000000 00000000 f7c2c594 
>[143150.600000]        c3839780 00000003 fffedc21 00000000 c3839780 f7fe1ca8 00000001 c057f51c 
>[143150.608000] Call Trace:
>[143150.608000]  [<c013c329>] test_func+0x56/0x95
>[143150.612000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.620000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.624000]  =======================
>[143150.628000] pdflush       S C3839780     0   254     19           255   251 (L-TLB)
>[143150.636000]        f7eb9f98 00000046 c054ae00 c3839780 f7eb9f8c f7eb9f3c c03c8d49 f7eb9f54 
>[143150.644000]        c011d34a 00000286 00000007 f7c47a30 4247c600 00000000 00000000 f7c47b54 
>[143150.652000]        c3839780 00000003 c04cd244 f7c47a30 f7c47ee0 f7eb9fa4 f7eb9f80 c03c7ed7 
>[143150.660000] Call Trace:
>[143150.664000]  [<c01544aa>] pdflush+0xb2/0x19f
>[143150.668000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.672000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.676000]  =======================
>[143150.680000] pdflush       D F7EBBD68     0   255     19           256   254 (L-TLB)
>[143150.688000]        f7ebbd7c 00000046 00000002 f7ebbd68 f7ebbd64 00000000 00000001 00000002 
>[143150.696000]        00000003 000037e7 0000000a f7c474d0 e44d0000 00000082 003d0900 f7c475f4 
>[143150.704000]        c3829780 00000001 0001000a 00000000 00000000 db287b30 db287b30 f7ffdec8 
>[143150.712000] Call Trace:
>[143150.716000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143150.724000]  [<c03c89b4>] rwsem_down_read_failed+0x1d/0x25
>[143150.728000]  [<c03c8a37>] call_rwsem_down_read_failed+0x7/0xc
>[143150.732000]  [<c02478af>] gfs2_log_reserve+0xef/0x100
>[143150.740000]  [<c02581d5>] gfs2_trans_begin+0xdb/0x112
>[143150.744000]  [<c024b644>] gfs2_writepage+0xe4/0x169
>[143150.748000]  [<c0153b7d>] generic_writepages+0x187/0x2ac
>[143150.756000]  [<c0153cce>] do_writepages+0x2c/0x34
>[143150.760000]  [<c0182a61>] __writeback_single_inode+0x17e/0x306
>[143150.764000]  [<c0182f4c>] sync_sb_inodes+0x194/0x25b
>[143150.772000]  [<c018335b>] writeback_inodes+0x7f/0xd0
>[143150.776000]  [<c0154120>] wb_kupdate+0x7a/0xdb
>[143150.780000]  [<c0154501>] pdflush+0x109/0x19f
>[143150.784000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.788000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.792000]  =======================
>[143150.796000] kswapd0       S F7EBDF2C     0   256     19           257   255 (L-TLB)
>[143150.804000]        f7ebdf40 00000046 00000002 f7ebdf2c f7ebdf28 00000000 00000000 ffffffff 
>[143150.812000]        f7c2f958 c04d71c0 0000000a f7c46f70 be6e9a00 00003dbe 00000000 f7c47094 
>[143150.820000]        c3839780 00000003 0101d553 00000000 00000246 f7ebdf28 00000246 f7ebdf94 
>[143150.832000] Call Trace:
>[143150.832000]  [<c0156c17>] kswapd+0xb6/0x39c
>[143150.836000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.840000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.848000]  =======================
>[143150.852000] aio/0         S F7EBFF70     0   257     19           258   256 (L-TLB)
>[143150.860000]        f7ebff84 00000046 00000002 f7ebff70 f7ebff6c 00000000 c0101fce c04c6640 
>[143150.868000]        f7c46bd0 c381ea80 00000006 f7c46a10 4247c600 00000000 00000000 f7c46b34 
>[143150.876000]        c3821780 00000000 fffedc27 00000000 f7c2ea88 00000246 f7ebff70 00000246 
>[143150.884000] Call Trace:
>[143150.888000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143150.892000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.896000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.900000]  =======================
>[143150.904000] aio/1         S F7C99F70     0   258     19           259   257 (L-TLB)
>[143150.912000]        f7c99f84 00000046 00000002 f7c99f70 f7c99f6c 00000000 c0101fce c392d210 
>[143150.920000]        f7c46670 c3826a80 00000006 f7c464b0 4247c600 00000000 00000000 f7c465d4 
>[143150.928000]        c3829780 00000001 fffedc27 00000000 f7c2e9fc 00000246 f7c99f70 00000246 
>[143150.936000] Call Trace:
>[143150.940000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143150.944000]  [<c0134edb>] kthread+0xb3/0xdc
>[143150.948000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143150.956000]  =======================
>[143150.960000] aio/2         S F7C9BF70     0   259     19           260   258 (L-TLB)
>[143150.968000]        f7c9bf84 00000046 00000002 f7c9bf70 f7c9bf6c 00000000 c0101fce c392ccb0 
>[143150.976000]        f7c55770 c382ea80 00000007 f7c555b0 4247c600 00000000 00000000 f7c556d4 
>[143150.984000]        c3831780 00000002 fffedc20 00000000 f7c2e970 00000246 f7c9bf70 00000246 
>[143150.992000] Call Trace:
>[143150.996000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.000000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.004000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.008000]  =======================
>[143151.012000] aio/3         S F7C9DF70     0   260     19           262   259 (L-TLB)
>[143151.020000]        f7c9df84 00000046 00000002 f7c9df70 f7c9df6c 00000000 c0101fce c392d770 
>[143151.028000]        f7c55210 c3836a80 00000007 f7c55050 4247c600 00000000 00000000 f7c55174 
>[143151.036000]        c3839780 00000003 fffedc21 00000000 f7f9a8e4 00000246 f7c9df70 00000246 
>[143151.044000] Call Trace:
>[143151.048000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.052000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.056000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.064000]  =======================
>[143151.068000] xfslogd/0     S F7CEFF70     0   262     19           263   260 (L-TLB)
>[143151.076000]        f7ceff84 00000046 00000002 f7ceff70 f7ceff6c 00000000 f7f8d640 00000082 
>[143151.084000]        f7ceff38 00000082 0000000a f7c54af0 b6a2ed00 00000004 00000000 f7c54c14 
>[143151.092000]        c3821780 00000000 fffeeecd 00000000 f7f8d628 00000246 f7ceff70 00000246 
>[143151.100000] Call Trace:
>[143151.104000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.108000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.112000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.116000]  =======================
>[143151.120000] xfslogd/1     S F630705C     0   263     19           264   262 (L-TLB)
>[143151.128000]        f7f01f84 00000046 f6307060 f630705c 00000001 00000003 f7f8d5b4 00000082 
>[143151.136000]        f7f01f38 00000082 0000000a f7c54590 f6c5cc00 00001da4 00000000 f7c546b4 
>[143151.144000]        c3829780 00000001 f7f8d594 00000246 f7f8d59c 00000246 f7f01f70 00000246 
>[143151.152000] Call Trace:
>[143151.156000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.160000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.164000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.172000]  =======================
>[143151.176000] xfslogd/2     S F7F03F70     0   264     19           265   263 (L-TLB)
>[143151.180000]        f7f03f84 00000046 00000002 f7f03f70 f7f03f6c 00000000 c0101fce c392ccb0 
>[143151.192000]        f7c1e5f0 c382ea80 00000007 f7c1e430 4247c600 00000000 00000000 f7c1e554 
>[143151.200000]        c3831780 00000002 fffedc20 00000000 f7f8d510 00000246 f7f03f70 00000246 
>[143151.208000] Call Trace:
>[143151.208000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.216000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.220000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.224000]  =======================
>[143151.228000] xfslogd/3     S F7E4B6F0     0   265     19           266   264 (L-TLB)
>[143151.236000]        f7f05f84 00000046 c3839780 f7e4b6f0 c39c0330 f7f05f44 c0101fce f7e4b8b0 
>[143151.244000]        c39c04f0 c3836a80 00000007 c39c0330 4247c600 00000000 00000000 c39c0454 
>[143151.252000]        c3839780 00000003 f7f89b1c 00000246 f7f8d484 00000246 f7f05f70 00000246 
>[143151.260000] Call Trace:
>[143151.264000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.268000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.272000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.280000]  =======================
>[143151.280000] xfsdatad/0    S F7F07F70     0   266     19           267   265 (L-TLB)
>[143151.288000]        f7f07f84 00000046 00000002 f7f07f70 f7f07f6c 00000000 c0101fce c04c6640 
>[143151.300000]        c39cdab0 c381ea80 00000007 c39cd8f0 4247c600 00000000 00000000 c39cda14 
>[143151.308000]        c3821780 00000000 fffedc27 00000000 f7f8d36c 00000246 f7f07f70 00000246 
>[143151.316000] Call Trace:
>[143151.316000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.324000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.328000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.332000]  =======================
>[143151.336000] xfsdatad/1    S F7F09F70     0   267     19           268   266 (L-TLB)
>[143151.344000]        f7f09f84 00000046 00000002 f7f09f70 f7f09f6c 00000000 c0101fce c392d210 
>[143151.352000]        c39cd550 c3826a80 00000007 c39cd390 4247c600 00000000 00000000 c39cd4b4 
>[143151.360000]        c3829780 00000001 fffedc27 00000000 f7f8d2e0 00000246 f7f09f70 00000246 
>[143151.368000] Call Trace:
>[143151.372000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.376000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.380000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.388000]  =======================
>[143151.388000] xfsdatad/2    S F7F0BF70     0   268     19           269   267 (L-TLB)
>[143151.396000]        f7f0bf84 00000046 00000002 f7f0bf70 f7f0bf6c 00000000 c0101fce c392ccb0 
>[143151.404000]        c39ccff0 c382ea80 00000007 c39cce30 4247c600 00000000 00000000 c39ccf54 
>[143151.416000]        c3831780 00000002 fffedc20 00000000 f7f8d254 00000246 f7f0bf70 00000246 
>[143151.424000] Call Trace:
>[143151.424000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.432000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.436000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.440000]  =======================
>[143151.444000] xfsdatad/3    S F7F0DF70     0   269     19           414   268 (L-TLB)
>[143151.452000]        f7f0df84 00000046 00000002 f7f0df70 f7f0df6c 00000000 c0101fce c392d770 
>[143151.460000]        c39cca90 c3836a80 00000007 c39cc8d0 4284cf00 00000000 00000000 c39cc9f4 
>[143151.468000]        c3839780 00000003 fffedc21 00000000 f7f8d1c8 00000246 f7f0df70 00000246 
>[143151.476000] Call Trace:
>[143151.480000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.484000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.488000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.492000]  =======================
>[143151.496000] scsi_eh_0     S F7E5DF4C     0   414     19           470   269 (L-TLB)
>[143151.504000]        f7e5df60 00000046 00000002 f7e5df4c f7e5df48 00000000 c03c8d49 c3839780 
>[143151.512000]        f7e4b6f0 00000096 00000009 c39cc370 0b076000 00000001 00000000 c39cc494 
>[143151.524000]        c3839780 00000003 fffedf6c 00000000 c3839780 c04c6280 c04c6280 f7e5df4c 
>[143151.532000] Call Trace:
>[143151.532000]  [<c030d5d9>] scsi_error_handler+0x6b/0x9b0
>[143151.540000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.544000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.548000]  =======================
>[143151.552000] kpsmoused     S F7D5BF70     0   470     19           492   414 (L-TLB)
>[143151.560000]        f7d5bf84 00000046 00000002 f7d5bf70 f7d5bf6c 00000000 c0101fce c392d770 
>[143151.568000]        f7f38330 c3836a80 00000009 f7f38170 14267800 00000002 00000000 f7f38294 
>[143151.576000]        c3839780 00000003 fffee3c3 00000000 c397913c 00000246 f7d5bf70 00000246 
>[143151.584000] Call Trace:
>[143151.588000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.592000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.596000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.604000]  =======================
>[143151.604000] kirqd         S F7D51F34     0   474      1           545    19 (L-TLB)
>[143151.612000]        f7d51f48 00000046 00000002 f7d51f34 f7d51f30 00000000 00000000 f7d51ef4 
>[143151.620000]        c03c8ce2 00000001 0000000a f7edccf0 42b27500 00008231 00000000 f7edce14 
>[143151.632000]        c3821780 00000000 0220ec17 00000000 00000286 f7d51f58 f7d51f48 c012c2e5 
>[143151.640000] Call Trace:
>[143151.640000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143151.648000]  [<c03c78c0>] schedule_timeout_interruptible+0x15/0x17
>[143151.652000]  [<c0117b53>] balanced_irq+0x50/0x495
>[143151.656000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.664000]  =======================
>[143151.668000] kjournald     S F7EDFF80     0   492     19           515   470 (L-TLB)
>[143151.676000]        f7edff94 00000046 00000002 f7edff80 f7edff7c 00000000 c39d5af8 f07c48c4 
>[143151.684000]        00000000 f7ce4cf8 00000009 f7f38c30 cf4b7200 0000822c 00000000 f7f38d54 
>[143151.692000]        c3831780 00000002 0220d96d 00000000 00000246 f7edff7c 00000246 f7edffa4 
>[143151.700000] Call Trace:
>[143151.704000]  [<c01caa77>] kjournald+0x169/0x1e8
>[143151.708000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.712000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.716000]  =======================
>[143151.720000] kauditd       S C399D450     0   515     19           965   492 (L-TLB)
>[143151.728000]        f66e5f98 00000046 f7d52f30 c399d450 c3836a80 00000003 00000000 c3955238 
>[143151.736000]        00000000 f66e5f4c 00000009 c399d290 c41a6a00 00000002 00000000 c399d3b4 
>[143151.744000]        c3839780 00000003 00000282 00000246 c04cd588 00000246 f66e5f84 00000246 
>[143151.752000] Call Trace:
>[143151.756000]  [<c01479c1>] kauditd_thread+0x102/0x131
>[143151.760000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.764000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.772000]  =======================
>[143151.776000] udevd         S F674FB30     0   545      1          5139   474 (NOTLB)
>[143151.784000]        f674fb44 00000086 00000002 f674fb30 f674fb2c 00000000 f54daa10 0000000e 
>[143151.792000]        f7f2b000 00000000 00000005 f7e64750 58cf0500 0000003c 00000000 f7e64874 
>[143151.800000]        c3821780 00000000 ffffd828 00000000 c03c80fb c3957da8 00000246 f674fb34 
>[143151.808000] Call Trace:
>[143151.812000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143151.816000]  [<c0175c30>] do_select+0x466/0x4ce
>[143151.820000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143151.824000]  [<c0176512>] sys_select+0xca/0x16c
>[143151.828000]  [<c0102eac>] syscall_call+0x7/0xb
>[143151.836000]  =======================
>[143151.836000] scsi_eh_1     S C3839780     0   965     19          1228   515 (L-TLB)
>[143151.844000]        f6185f60 00000046 c054ae00 c3839780 f6185f6c f6185f04 c03c8d49 c3839780 
>[143151.852000]        f7e4b6f0 00000096 00000006 c3990790 f3125700 00000002 00000000 c39908b4 
>[143151.864000]        c3839780 00000003 c3836a80 00000003 00000000 c3921430 00000000 f6185f4c 
>[143151.872000] Call Trace:
>[143151.872000]  [<c030d5d9>] scsi_error_handler+0x6b/0x9b0
>[143151.880000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.884000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.888000]  =======================
>[143151.892000] qla2xxx_1_dpc S F6189F88     0  1228     19          1229   965 (L-TLB)
>[143151.900000]        f6189f9c 00000046 00000002 f6189f88 f6189f84 00000000 00000000 0200001e 
>[143151.908000]        f6189f44 f6189f44 0000000a f608a2f0 52ad7000 00000003 00000000 f608a414 
>[143151.916000]        c3839780 00000003 fffee8f7 00000000 f6174378 00000100 00000000 f6189f9c 
>[143151.924000] Call Trace:
>[143151.928000]  [<f89e9332>] qla2x00_do_dpc+0x33/0x43e [qla2xxx]
>[143151.932000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.940000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.944000]  =======================
>[143151.948000] scsi_wq_1     S F66A9F40     0  1229     19          1230  1228 (L-TLB)
>[143151.956000]        f66a9f84 00000046 c03c7ed7 f66a9f40 c0311cbf 00000003 f615ad5c 00000082 
>[143151.964000]        f66a9f38 00000082 0000000a f7ec4c70 5da50e00 00000003 00000000 f7ec4d94 
>[143151.972000]        c3831780 00000002 f615ad3c 00000246 f615ad44 00000246 f66a9f70 00000246 
>[143151.980000] Call Trace:
>[143151.984000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143151.988000]  [<c0134edb>] kthread+0xb3/0xdc
>[143151.992000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143151.996000]  =======================
>[143152.000000] fc_wq_1       S F6027F70     0  1230     19          1231  1229 (L-TLB)
>[143152.008000]        f6027f84 00000046 00000002 f6027f70 f6027f6c 00000000 c0101fce c04c6640 
>[143152.016000]        f608ba30 c381ea80 00000008 f608b870 165b8b00 00000003 00000000 f608b994 
>[143152.024000]        c3821780 00000000 fffee7ff 00000000 f615a970 00000246 f6027f70 00000246 
>[143152.036000] Call Trace:
>[143152.036000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.040000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.044000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.052000]  =======================
>[143152.056000] fc_dl_1       S F61DBF70     0  1231     19          4711  1230 (L-TLB)
>[143152.064000]        f61dbf84 00000046 00000002 f61dbf70 f61dbf6c 00000000 c0101fce c04c6640 
>[143152.072000]        f61765b0 c381ea80 00000008 f61763f0 165b8b00 00000003 00000000 f6176514 
>[143152.080000]        c3821780 00000000 fffee7ff 00000000 f61326b4 00000246 f61dbf70 00000246 
>[143152.088000] Call Trace:
>[143152.092000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.096000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.100000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.104000]  =======================
>[143152.108000] kmirrord      S F7EDC230     0  4711     19          4736  1231 (L-TLB)
>[143152.116000]        f617ff84 00000046 00000000 f7edc230 c39c0df0 f617ff44 c0101fce f7edc3f0 
>[143152.124000]        c39c0fb0 c381ea80 00000001 c39c0df0 5f230600 00000004 00000000 c39c0f14 
>[143152.132000]        c3821780 00000000 f6117804 00000246 f60d17cc 00000246 f617ff70 00000246 
>[143152.140000] Call Trace:
>[143152.144000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.148000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.152000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.160000]  =======================
>[143152.164000] kjournald     S F5C61F80     0  4736     19          4738  4711 (L-TLB)
>[143152.172000]        f5c61f94 00000046 00000002 f5c61f80 f5c61f7c 00000000 f62ba494 f07c4234 
>[143152.180000]        00000000 f67cbd44 0000000a f6176950 4f50a700 00006c4c 00000000 f6176a74 
>[143152.188000]        c3831780 00000002 01c5173a 00000000 00000246 f5c61f7c 00000246 f5c61fa4 
>[143152.196000] Call Trace:
>[143152.200000]  [<c01caa77>] kjournald+0x169/0x1e8
>[143152.204000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.208000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.212000]  =======================
>[143152.216000] kjournald     S 00000000     0  4738     19          4740  4736 (L-TLB)
>[143152.224000]        f7d4df94 00000046 00000013 00000000 f5c2cc14 00000041 c055b512 8bb89900 
>[143152.232000]        1f3b9b00 363c4fb6 00000008 c399d7f0 8bb89900 00000004 00000000 c399d914 
>[143152.240000]        c3821780 00000000 00000246 f5c2cc34 00000246 f7d4df7c 00000246 f7d4dfa4 
>[143152.248000] Call Trace:
>[143152.252000]  [<c01caa77>] kjournald+0x169/0x1e8
>[143152.256000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.260000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.268000]  =======================
>[143152.272000] xfsbufd       S F5C17F4C     0  4740     19          4741  4738 (L-TLB)
>[143152.276000]        f5c17f60 00000046 00000002 f5c17f4c f5c17f48 00000000 00000000 c057bc00 
>[143152.288000]        f5c17f60 c010387f 0000000a f7ec51d0 67e3f100 00008231 00000000 f7ec52f4 
>[143152.296000]        c3821780 00000000 0220ecb4 00000000 00000286 f5c17f70 f5c17f60 c012c2e5 
>[143152.304000] Call Trace:
>[143152.304000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143152.312000]  [<c03c78c0>] schedule_timeout_interruptible+0x15/0x17
>[143152.316000]  [<c0232f74>] xfsbufd+0x6b/0x187
>[143152.320000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.324000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.332000]  =======================
>[143152.336000] xfssyncd      S F628DF58     0  4741     19          4859  4740 (L-TLB)
>[143152.344000]        f628df6c 00000046 00000002 f628df58 f628df54 00000000 22222222 22222222 
>[143152.352000]        f6307e90 f7e56008 0000000a f61de430 b4f21000 0000822a 00000000 f61de554 
>[143152.360000]        c3821780 00000000 0220d099 00000000 00000286 f628df7c f628df6c c012c2e5 
>[143152.368000] Call Trace:
>[143152.372000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143152.376000]  [<c03c78c0>] schedule_timeout_interruptible+0x15/0x17
>[143152.380000]  [<c0237b86>] xfssyncd+0x26/0x117
>[143152.388000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.392000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.396000]  =======================
>[143152.400000] kondemand/0   S F5CF5F70     0  4859     19          4860  4741 (L-TLB)
>[143152.408000]        f5cf5f84 00000046 00000002 f5cf5f70 f5cf5f6c 00000000 f58dbf00 00000082 
>[143152.416000]        f5cf5f38 00000082 0000000a f5c676f0 473a2000 00008231 00000000 f5c67814 
>[143152.424000]        c3821780 00000000 0220ec2a 00000000 f58dbee8 00000246 f5cf5f70 00000246 
>[143152.432000] Call Trace:
>[143152.436000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.440000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.444000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.448000]  =======================
>[143152.452000] kondemand/1   S F5CA9F70     0  4860     19          4861  4859 (L-TLB)
>[143152.460000]        f5ca9f84 00000046 00000002 f5ca9f70 f5ca9f6c 00000000 c0101fce f7ee99b0 
>[143152.468000]        f628f770 c3826a80 00000008 f628f5b0 d09fc200 00000004 00000000 f628f6d4 
>[143152.480000]        c3829780 00000001 fffeef3d 00000000 f58dbe5c 00000246 f5ca9f70 00000246 
>[143152.488000] Call Trace:
>[143152.488000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.496000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.500000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.504000]  =======================
>[143152.508000] kondemand/2   S F63A7F70     0  4861     19          4862  4860 (L-TLB)
>[143152.516000]        f63a7f84 00000046 00000002 f63a7f70 f63a7f6c 00000000 f58dbde8 00000082 
>[143152.524000]        f63a7f38 00000082 0000000a f5c500f0 30e09400 00008232 00000000 f5c50214 
>[143152.532000]        c3831780 00000002 0220efff 00000000 f58dbdd0 00000246 f63a7f70 00000246 
>[143152.540000] Call Trace:
>[143152.544000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.548000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.552000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.556000]  =======================
>[143152.560000] kondemand/3   S F5C15F70     0  4862     19          5471  4861 (L-TLB)
>[143152.568000]        f5c15f84 00000046 00000002 f5c15f70 f5c15f6c 00000000 c0101fce c392c750 
>[143152.576000]        f5ceb610 c3836a80 00000008 f5ceb450 d09fc200 00000004 00000000 f5ceb574 
>[143152.584000]        c3839780 00000003 fffeef3d 00000000 f58dbd44 00000246 f5c15f70 00000246 
>[143152.596000] Call Trace:
>[143152.596000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143152.600000]  [<c0134edb>] kthread+0xb3/0xdc
>[143152.608000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143152.612000]  =======================
>[143152.616000] dhclient      S F6069B30     0  5139      1          5195   545 (NOTLB)
>[143152.624000]        f6069b44 00000082 00000002 f6069b30 f6069b2c 00000000 00000000 f6069b44 
>[143152.632000]        c010393c f6069b54 0000000a f5c90230 642ee700 0000822b 00000000 f5c90354 
>[143152.640000]        c3821780 00000000 0220d37b 00000000 00000286 f6069b54 f6069b44 c012c2e5 
>[143152.648000] Call Trace:
>[143152.652000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143152.656000]  [<c0175c30>] do_select+0x466/0x4ce
>[143152.660000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143152.664000]  [<c01764e1>] sys_select+0x99/0x16c
>[143152.672000]  [<c0102eac>] syscall_call+0x7/0xb
>[143152.676000]  =======================
>[143152.680000] syslogd       S F7D53B30     0  5195      1          5198  5139 (NOTLB)
>[143152.688000]        f7d53b44 00000086 00000002 f7d53b30 f7d53b2c 00000000 00000000 ebe3e5c4 
>[143152.696000]        00000000 f7d53b18 00000007 f5cbadf0 dadaa700 0000822a 00000000 f5cbaf14 
>[143152.704000]        c3821780 00000000 0220d13a 00000000 00000246 f7d53b2c 00000246 f7d53bec 
>[143152.712000] Call Trace:
>[143152.716000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143152.720000]  [<c0175c30>] do_select+0x466/0x4ce
>[143152.724000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143152.728000]  [<c0176512>] sys_select+0xca/0x16c
>[143152.732000]  [<c0102eac>] syscall_call+0x7/0xb
>[143152.740000]  =======================
>[143152.744000] irqbalance    S F66A7F18     0  5208      1          5226  5198 (NOTLB)
>[143152.748000]        f66a7f2c 00000082 00000002 f66a7f18 f66a7f14 00000000 c38228d8 f66a7ee0 
>[143152.760000]        c013795c c38228ec 0000000a f7ed0750 bfbe6600 0000822f 00000000 f7ed0874 
>[143152.768000]        c3821780 00000000 0220e5c1 00000000 c01377bb bb4bbdb9 00008236 00000000 
>[143152.776000] Call Trace:
>[143152.776000]  [<c03c8160>] do_nanosleep+0x3e/0x6a
>[143152.784000]  [<c0137812>] hrtimer_nanosleep+0x4c/0xf9
>[143152.788000]  [<c0137906>] sys_nanosleep+0x47/0x58
>[143152.792000]  [<c0102eac>] syscall_call+0x7/0xb
>[143152.796000]  =======================
>[143152.800000] portmap       S F5CBFBD8     0  5226      1          5245  5208 (NOTLB)
>[143152.808000]        f5cbfbec 00000082 00000002 f5cbfbd8 f5cbfbd4 00000000 00000000 00000490 
>[143152.816000]        00000000 00000850 0000000a f5c91250 9ecaba00 0000819b 00000000 f5c91374 
>[143152.824000]        c3821780 00000000 021e7876 00000000 f5cbfbd8 c01351a5 00000000 f5cbfc50 
>[143152.832000] Call Trace:
>[143152.836000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143152.840000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143152.848000]  [<c01757c8>] sys_poll+0x34/0x36
>[143152.852000]  [<c0102eac>] syscall_call+0x7/0xb
>[143152.856000]  =======================
>[143152.860000] rpc.statd     S F62F7B30     0  5245      1          5278  5226 (NOTLB)
>[143152.868000]        f62f7b44 00000082 00000002 f62f7b30 f62f7b2c 00000000 00000000 00000000 
>[143152.876000]        00000000 00000000 00000001 f7e65770 da85b700 00000006 00000000 f7e65894 
>[143152.884000]        c3839780 00000003 fffef7cb 00000000 f62f7b30 c01351a5 00000000 f62f7bd8 
>[143152.892000] Call Trace:
>[143152.896000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143152.900000]  [<c0175c30>] do_select+0x466/0x4ce
>[143152.904000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143152.908000]  [<c0176512>] sys_select+0xca/0x16c
>[143152.916000]  [<c0102eac>] syscall_call+0x7/0xb
>[143152.920000]  =======================
>[143152.924000] rpc.idmapd    S F7E59F14     0  5278      1          5292  5245 (NOTLB)
>[143152.932000]        f7e59f28 00200086 00000002 f7e59f14 f7e59f10 00000000 00000000 c0138a06 
>[143152.940000]        f7e59f5c ffffe83a 0000000a f5ce7970 46830500 00008231 00000000 f5ce7a94 
>[143152.948000]        c3839780 00000003 0220ec28 00000000 00200286 f7e59f38 f7e59f28 c012c2e5 
>[143152.956000] Call Trace:
>[143152.960000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143152.964000]  [<c018f2d1>] sys_epoll_wait+0x13f/0x39a
>[143152.968000]  [<c0102eac>] syscall_call+0x7/0xb
>[143152.972000]  =======================
>[143152.976000] dbus-daemon   S F62D5BD8     0  5292      1          5330  5278 (NOTLB)
>[143152.984000]        f62d5bec 00000082 00000002 f62d5bd8 f62d5bd4 00000000 5a5a5a5a 5a5a5a5a 
>[143152.992000]        c1883380 f62d5ba0 00000003 f5ceaef0 58cf0500 0000003c 00000000 f5ceb014 
>[143153.000000]        c3829780 00000001 ffffd828 00000000 f62d5bd0 00000246 f62d5cb8 f62d5be4 
>[143153.008000] Call Trace:
>[143153.012000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.016000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143153.020000]  [<c01757c8>] sys_poll+0x34/0x36
>[143153.028000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.032000]  =======================
>[143153.036000] automount     S F5C59E78     0  5330      1          5331  5292 (NOTLB)
>[143153.044000]        f5c59ecc 00000082 00000000 f5c59e78 c017be3b 459b0004 017691d5 00000004 
>[143153.052000]        c2caaa20 f5c59ec4 00000001 c39720f0 f6a7dd00 00000006 00000000 c3972214 
>[143153.060000]        c3829780 00000001 f5c59f78 c013a195 f553c000 00000000 f62b828c c39720f0 
>[143153.068000] Call Trace:
>[143153.072000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.076000]  [<c03c78c0>] schedule_timeout_interruptible+0x15/0x17
>[143153.080000]  [<c012f286>] sys_rt_sigtimedwait+0x1b9/0x2bc
>[143153.088000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.092000]  =======================
>[143153.096000] automount     S F7EEBE58     0  5331      1          5332  5330 (NOTLB)
>[143153.104000]        f7eebe6c 00000082 00000002 f7eebe58 f7eebe54 00000000 f5cb1f34 f5cb1f0c 
>[143153.112000]        00000000 f7eebe20 00000002 f5cea990 18aa8400 0000822c 00000000 f5ceaab4 
>[143153.120000]        c3821780 00000000 0220d66f 00000000 00000286 f7eebe7c f7eebe6c c012c2e5 
>[143153.128000] Call Trace:
>[143153.132000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143153.136000]  [<c013a11c>] do_futex+0x1a7/0xc2b
>[143153.140000]  [<c013aca7>] sys_futex+0x107/0x11a
>[143153.144000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.148000]  =======================
>[143153.152000] automount     S F5CB1E58     0  5332      1          5335  5331 (NOTLB)
>[143153.160000]        f5cb1e6c 00000082 00000002 f5cb1e58 f5cb1e54 00000000 00000000 f521866c 
>[143153.168000]        00000020 00000000 0000000a f7f39190 37f7e000 00008231 00000000 f7f392b4 
>[143153.180000]        c3821780 00000000 0220ebeb 00000000 00000286 f5cb1e7c f5cb1e6c c012c2e5 
>[143153.188000] Call Trace:
>[143153.188000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143153.196000]  [<c013a11c>] do_futex+0x1a7/0xc2b
>[143153.200000]  [<c013aca7>] sys_futex+0x107/0x11a
>[143153.204000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.208000]  =======================
>[143153.212000] automount     S F6313BD8     0  5335      1          5338  5332 (NOTLB)
>[143153.220000]        f6313bec 00000082 00000002 f6313bd8 f6313bd4 00000000 f6313bbc f7e32d40 
>[143153.228000]        00000000 f6313bc8 0000000a f7ca00b0 18aa8400 0000822c 00000000 f7ca01d4 
>[143153.236000]        c3821780 00000000 0156ba25 00000246 f5c3a6e0 00000246 f6313bd8 c38217d4 
>[143153.244000] Call Trace:
>[143153.248000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.252000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143153.256000]  [<c01757c8>] sys_poll+0x34/0x36
>[143153.260000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.268000]  =======================
>[143153.268000] automount     S F553DBD8     0  5338      1          5348  5335 (NOTLB)
>[143153.276000]        f553dbec 00000082 00000002 f553dbd8 f553dbd4 00000000 f553db98 c027c67f 
>[143153.284000]        00000000 f553dc24 0000000a f5c6d730 93cc2e00 00008226 00000000 f5c6d854 
>[143153.296000]        c3821780 00000000 00013aa4 00000246 f62bb6e0 00000246 f553dbd8 c38217d4 
>[143153.304000] Call Trace:
>[143153.304000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.312000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143153.316000]  [<c01757c8>] sys_poll+0x34/0x36
>[143153.320000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.324000]  =======================
>[143153.328000] smartd        S F5C63F18     0  5348      1          5357  5338 (NOTLB)
>[143153.336000]        f5c63f2c 00000086 00000002 f5c63f18 f5c63f14 00000000 c38328d8 f5c63ee0 
>[143153.344000]        c013795c c38328ec 0000000a f61e99f0 018aa000 00008157 00000000 f61e9b14 
>[143153.352000]        c3831780 00000002 021d58aa 00000000 c01377bb c1762749 000082fe 00000000 
>[143153.360000] Call Trace:
>[143153.364000]  [<c03c8160>] do_nanosleep+0x3e/0x6a
>[143153.368000]  [<c0137812>] hrtimer_nanosleep+0x4c/0xf9
>[143153.372000]  [<c0137906>] sys_nanosleep+0x47/0x58
>[143153.380000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.384000]  =======================
>[143153.388000] acpid         S F7427BD8     0  5357      1          5366  5348 (NOTLB)
>[143153.396000]        f7427bec 00000082 00000002 f7427bd8 f7427bd4 00000000 f7c8dc04 f7c7b360 
>[143153.404000]        c2dd4c48 f7ec7af4 00000007 f5ce6eb0 46966600 00000008 00000000 f5ce6fd4 
>[143153.412000]        c3829780 00000001 fffefdc1 00000000 f7427bd0 00000246 f7427c80 f7427be4 
>[143153.420000] Call Trace:
>[143153.424000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.428000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143153.432000]  [<c01757c8>] sys_poll+0x34/0x36
>[143153.436000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.440000]  =======================
>[143153.444000] hpiod         S F61A7E18     0  5366      1          5371  5357 (NOTLB)
>[143153.452000]        f61a7e2c 00200082 00000002 f61a7e18 f61a7e14 00000000 ffffffff 00000000 
>[143153.460000]        00000000 00000000 00000001 f7ec5730 2f663b00 00000007 00000000 f7ec5854 
>[143153.468000]        c3829780 00000001 fffef92b 00000000 f7ff2980 f7ff2980 f554b4c4 f61a7e38 
>[143153.476000] Call Trace:
>[143153.480000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.484000]  [<c0390f37>] inet_csk_accept+0x93/0x1f8
>[143153.492000]  [<c03aba65>] inet_accept+0x20/0xa1
>[143153.496000]  [<c03686eb>] sys_accept+0xbc/0x158
>[143153.500000]  [<c0368857>] sys_socketcall+0xd0/0x241
>[143153.504000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.508000]  =======================
>[143153.512000] python        S F6169B30     0  5371      1          5381  5366 (NOTLB)
>[143153.520000]        f6169b44 00200082 00000002 f6169b30 f6169b2c 00000000 00000000 c03c8df5 
>[143153.528000]        f6169b64 f6169af8 0000000a f7e65210 39b2e100 00008232 00000000 f7e65334 
>[143153.536000]        c3829780 00000001 0220f024 00000000 00200286 f6169b54 f6169b44 c012c2e5 
>[143153.548000] Call Trace:
>[143153.548000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143153.552000]  [<c0175c30>] do_select+0x466/0x4ce
>[143153.560000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143153.564000]  [<c01764e1>] sys_select+0x99/0x16c
>[143153.568000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.572000]  =======================
>[143153.576000] cupsd         S F6663B30     0  5381      1          5398  5371 (NOTLB)
>[143153.584000]        f6663b44 00200082 00000002 f6663b30 f6663b2c 00000000 00001000 00000007 
>[143153.592000]        f7f28494 f7f28494 00000007 f7ec41b0 9257d200 00000007 00000000 f7ec42d4 
>[143153.600000]        c3821780 00000000 fffefacd 00000000 f5c0392c 00200246 f6663b30 00200246 
>[143153.608000] Call Trace:
>[143153.612000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.616000]  [<c0175c30>] do_select+0x466/0x4ce
>[143153.620000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143153.628000]  [<c0176512>] sys_select+0xca/0x16c
>[143153.632000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.636000]  =======================
>[143153.640000] sshd          S F55E7B30     0  5398      1  5768    5408  5381 (NOTLB)
>[143153.648000]        f55e7b44 00200082 00000002 f55e7b30 f55e7b2c 00000000 00001000 00000007 
>[143153.656000]        f7f28494 f7f28494 0000000a f6147930 df27f700 00000097 00000000 f6147a54 
>[143153.664000]        c3821780 00000000 00200246 f55e7bec f55e7b30 c01351a5 00000000 f55e7bd8 
>[143153.672000] Call Trace:
>[143153.676000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.680000]  [<c0175c30>] do_select+0x466/0x4ce
>[143153.684000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143153.692000]  [<c0176512>] sys_select+0xca/0x16c
>[143153.696000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.700000]  =======================
>[143153.704000] xinetd        S FFFEFB85     0  5408      1          5420  5398 (NOTLB)
>[143153.712000]        f6347b44 00000086 c012c2e5 fffefb85 00000000 00000086 f7c8dc04 f7c8dc04 
>[143153.720000]        00000000 00000003 0000000a f7ce0650 e1056c00 00000007 00000000 f7ce0774 
>[143153.728000]        c3829780 00000001 00000246 f6347c24 f6347b30 c01351a5 00000000 f6347bd8 
>[143153.736000] Call Trace:
>[143153.740000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.744000]  [<c0175c30>] do_select+0x466/0x4ce
>[143153.748000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143153.752000]  [<c0176512>] sys_select+0xca/0x16c
>[143153.760000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.764000]  =======================
>[143153.768000] ntpd          S F5CA5B30     0  5420      1          5439  5408 (NOTLB)
>[143153.776000]        f5ca5b44 00200086 00000002 f5ca5b30 f5ca5b2c 00000000 00000000 00000007 
>[143153.784000]        f7f28494 f7f28494 0000000a f5c50650 39661600 00008231 00000000 f5c50774 
>[143153.792000]        c3821780 00000000 0220ebf1 00000000 f7d6e530 f5ca5b38 c017626f c39f3640 
>[143153.800000] Call Trace:
>[143153.804000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.808000]  [<c0175c30>] do_select+0x466/0x4ce
>[143153.812000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143153.816000]  [<c0176512>] sys_select+0xca/0x16c
>[143153.820000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.828000]  =======================
>[143153.832000] rpc.rquotad   S 5A5A5A5A     0  5439      1          5472  5420 (NOTLB)
>[143153.836000]        f5c85bec 00000082 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 
>[143153.848000]        5a5a5a5a 5a5a5a5a 00000001 f5cba890 76afbe00 00000007 00000000 f5cba9b4 
>[143153.856000]        c3829780 00000001 00000246 f5c85c80 f5c85bd8 c01351a5 00000000 f5c85c50 
>[143153.864000] Call Trace:
>[143153.864000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.872000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143153.876000]  [<c01757c8>] sys_poll+0x34/0x36
>[143153.880000]  [<c0102eac>] syscall_call+0x7/0xb
>[143153.884000]  =======================
>[143153.888000] nfsd4         S F5617F70     0  5471     19          5473  4862 (L-TLB)
>[143153.896000]        f5617f84 00000046 00000002 f5617f70 f5617f6c 00000000 f560b384 00000082 
>[143153.904000]        f5617f38 00000082 00000004 f5caa850 904c6700 0000822f 00000000 f5caa974 
>[143153.912000]        c3821780 00000000 0220e4fa 00000000 f560b36c 00000246 f5617f70 00000246 
>[143153.920000] Call Trace:
>[143153.924000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143153.928000]  [<c0134edb>] kthread+0xb3/0xdc
>[143153.932000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143153.940000]  =======================
>[143153.944000] lockd         S F561DF38     0  5472      1          5477  5439 (L-TLB)
>[143153.948000]        f561df4c 00000046 00000002 f561df38 f561df34 00000000 c0367c18 c03ec720 
>[143153.960000]        fffffff5 f561df6c 00000001 f5c66170 81e46500 00000007 00000000 f5c66294 
>[143153.968000]        c3829780 00000001 fffefa87 00000000 00000000 00000000 c03c8cd8 f561df4c 
>[143153.976000] Call Trace:
>[143153.976000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143153.984000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143153.988000]  [<f898f356>] lockd+0x10e/0x21c [lockd]
>[143153.992000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.000000]  =======================
>[143154.004000] rpciod/0      S F6083F2C     0  5473     19          5474  5471 (L-TLB)
>[143154.012000]        f6083f84 00000046 f08dae70 f6083f2c f8935ef7 00000003 f560be74 00000082 
>[143154.020000]        f6083f38 00000082 0000000a f5cea430 d868a000 000041c8 00000000 f5cea554 
>[143154.028000]        c3821780 00000000 f560be54 00000246 f560be5c 00000246 f6083f70 c3821c4c 
>[143154.036000] Call Trace:
>[143154.040000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143154.044000]  [<c0134edb>] kthread+0xb3/0xdc
>[143154.048000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.052000]  =======================
>[143154.056000] rpciod/1      S F561FF70     0  5474     19          5475  5473 (L-TLB)
>[143154.064000]        f561ff84 00000046 00000002 f561ff70 f561ff6c 00000000 c0101fce f5c66330 
>[143154.072000]        f5c66df0 c3826a80 00000007 f5c66c30 81e46500 00000007 00000000 f5c66d54 
>[143154.080000]        c3829780 00000001 fffefa87 00000000 f560bdd0 00000246 f561ff70 00000246 
>[143154.088000] Call Trace:
>[143154.092000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143154.096000]  [<c0134edb>] kthread+0xb3/0xdc
>[143154.100000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.108000]  =======================
>[143154.112000] rpciod/2      S F561BF70     0  5475     19          5476  5474 (L-TLB)
>[143154.116000]        f561bf84 00000046 00000002 f561bf70 f561bf6c 00000000 c0101fce c392ccb0 
>[143154.128000]        f5c67350 c382ea80 00000007 f5c67190 81e46500 00000007 00000000 f5c672b4 
>[143154.136000]        c3831780 00000002 fffefa89 00000000 f560bd44 00000246 f561bf70 00000246 
>[143154.144000] Call Trace:
>[143154.144000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143154.152000]  [<c0134edb>] kthread+0xb3/0xdc
>[143154.156000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.160000]  =======================
>[143154.164000] rpciod/3      S F5619F70     0  5476     19          5836  5475 (L-TLB)
>[143154.172000]        f5619f84 00000046 00000002 f5619f70 f5619f6c 00000000 c0101fce c392c750 
>[143154.180000]        f6127550 c3836a80 00000007 f6127390 81e46500 00000007 00000000 f61274b4 
>[143154.188000]        c3839780 00000003 fffefa89 00000000 f560bcb8 00000246 f5619f70 00000246 
>[143154.196000] Call Trace:
>[143154.200000]  [<c01326eb>] worker_thread+0xca/0x11e
>[143154.204000]  [<c0134edb>] kthread+0xb3/0xdc
>[143154.208000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.216000]  =======================
>[143154.216000] nfsd          S F55B9F24     0  5477      1          5478  5472 (L-TLB)
>[143154.224000]        f55b9f38 00000046 00000002 f55b9f24 f55b9f20 00000000 f55b9ee4 00000086 
>[143154.236000]        00000001 f55b9f44 0000000a f5caadb0 ee04c500 00007fb8 00000000 f5caaed4 
>[143154.244000]        c3821780 00000000 00000286 f55b9f18 00000286 f55b9f48 f55b9f38 c38217d4 
>[143154.252000] Call Trace:
>[143154.252000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.260000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.264000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.268000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.276000]  =======================
>[143154.276000] nfsd          S F55F1F24     0  5478      1          5479  5477 (L-TLB)
>[143154.284000]        f55f1f38 00000046 00000002 f55f1f24 f55f1f20 00000000 c0367c18 c03ec720 
>[143154.292000]        fffffff5 f55f1f58 0000000a f5cbb8b0 228f6700 00007f4e 00000000 f5cbb9d4 
>[143154.304000]        c3829780 00000001 00000286 f55f1f18 00000286 f55f1f48 f55f1f38 c38297d4 
>[143154.312000] Call Trace:
>[143154.312000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.320000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.324000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.328000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.336000]  =======================
>[143154.336000] nfsd          S F55BDF24     0  5479      1          5480  5478 (L-TLB)
>[143154.344000]        f55bdf38 00000046 00000002 f55bdf24 f55bdf20 00000000 f55bdee4 00000086 
>[143154.352000]        00000001 f55bdf44 0000000a f628eaf0 22cc7000 00007f4e 00000000 f628ec14 
>[143154.364000]        c3831780 00000002 00000286 f55bdf18 00000286 f55bdf48 f55bdf38 c3831c4c 
>[143154.372000] Call Trace:
>[143154.372000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.380000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.384000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.388000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.392000]  =======================
>[143154.396000] nfsd          S F7E53F24     0  5480      1          5481  5479 (L-TLB)
>[143154.404000]        f7e53f38 00000046 00000002 f7e53f24 f7e53f20 00000000 f7e53ee4 00000086 
>[143154.412000]        00000001 f7e53f44 0000000a f5c80750 ee04c500 00007fb8 00000000 f5c80874 
>[143154.424000]        c3839780 00000003 00000286 f7e53f18 00000286 f7e53f48 f7e53f38 c3839c4c 
>[143154.432000] Call Trace:
>[143154.432000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.440000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.444000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.448000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.452000]  =======================
>[143154.456000] nfsd          S F5CFBF24     0  5481      1          5482  5480 (L-TLB)
>[143154.464000]        f5cfbf38 00000046 00000002 f5cfbf24 f5cfbf20 00000000 00000000 00000086 
>[143154.472000]        00000001 f5cfbf44 0000000a f61de990 ee04c500 00007fb8 00000000 f61deab4 
>[143154.480000]        c3821780 00000000 02168fe8 00000000 00000286 f5cfbf48 f5cfbf38 c012c2e5 
>[143154.492000] Call Trace:
>[143154.492000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.500000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.504000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.508000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.512000]  =======================
>[143154.516000] nfsd          S F5CEDF24     0  5482      1          5483  5481 (L-TLB)
>[143154.524000]        f5cedf38 00000046 00000002 f5cedf24 f5cedf20 00000000 f5cedee4 00000086 
>[143154.532000]        00000001 f5cedf44 0000000a c39b5310 228f6700 00007f4e 00000000 c39b5434 
>[143154.540000]        c3829780 00000001 0214cffa 00000000 00000286 f5cedf48 f5cedf38 c012c2e5 
>[143154.552000] Call Trace:
>[143154.552000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.560000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.564000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.568000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.572000]  =======================
>[143154.576000] nfsd          S F5171F24     0  5483      1          5484  5482 (L-TLB)
>[143154.584000]        f5171f38 00000046 00000002 f5171f24 f5171f20 00000000 f5171ee4 00000086 
>[143154.592000]        00000001 f5171f44 0000000a f61e9490 22cc7000 00007f4e 00000000 f61e95b4 
>[143154.600000]        c3831780 00000002 0214cffb 00000000 00000286 f5171f48 f5171f38 c012c2e5 
>[143154.612000] Call Trace:
>[143154.612000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.616000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.624000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.628000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.632000]  =======================
>[143154.636000] nfsd          S F55BBF24     0  5484      1          5487  5483 (L-TLB)
>[143154.644000]        f55bbf38 00000046 00000002 f55bbf24 f55bbf20 00000000 00000000 f7ffb440 
>[143154.652000]        00000000 f55bbf14 0000000a f7d56a10 ee04c500 00007fb8 00000000 f7d56b34 
>[143154.660000]        c3839780 00000003 02168fe8 00000000 00000286 f55bbf48 f55bbf38 c012c2e5 
>[143154.672000] Call Trace:
>[143154.672000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.676000]  [<f893ab2e>] svc_recv+0x209/0x36b [sunrpc]
>[143154.684000]  [<f899c6ac>] nfsd+0xc9/0x275 [nfsd]
>[143154.688000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143154.692000]  =======================
>[143154.696000] rpc.mountd    S 5A5A5A5A     0  5487      1          5517  5484 (NOTLB)
>[143154.704000]        f55f3b44 00000082 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 
>[143154.712000]        5a5a5a5a 5a5a5a5a 00000001 f5cab870 8577ec00 00000007 00000000 f5cab994 
>[143154.720000]        c3829780 00000001 00000246 f55f3c5c f55f3b30 c01351a5 00000000 f55f3bd8 
>[143154.728000] Call Trace:
>[143154.732000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143154.736000]  [<c0175c30>] do_select+0x466/0x4ce
>[143154.744000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143154.748000]  [<c0176512>] sys_select+0xca/0x16c
>[143154.752000]  [<c0102eac>] syscall_call+0x7/0xb
>[143154.756000]  =======================
>[143154.760000] sendmail      S F61E5B30     0  5517      1          5524  5487 (NOTLB)
>[143154.768000]        f61e5b44 00200082 00000002 f61e5b30 f61e5b2c 00000000 00000000 00200246 
>[143154.776000]        f7f35250 f7f35260 0000000a f5cab310 46fd1700 00008231 00000000 f5cab434 
>[143154.784000]        c3821780 00000000 0220ec2a 00000000 00200286 f61e5b54 f61e5b44 c012c2e5 
>[143154.792000] Call Trace:
>[143154.796000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.800000]  [<c0175c30>] do_select+0x466/0x4ce
>[143154.804000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143154.812000]  [<c01764e1>] sys_select+0x99/0x16c
>[143154.816000]  [<c0102eac>] syscall_call+0x7/0xb
>[143154.820000]  =======================
>[143154.824000] sendmail      S C39A9F94     0  5524      1          5533  5517 (NOTLB)
>[143154.832000]        c39a9fa8 00200082 00000002 c39a9f94 c39a9f90 00000000 c39a9f88 f628e590 
>[143154.840000]        c395e9e0 0000000f 00000009 f628e590 7b17d400 00007fb4 00000000 f628e6b4 
>[143154.848000]        c3821780 00000000 02167d3e 00000000 c39a9fb8 f60c5e3c f60c5e08 00000001 
>[143154.856000] Call Trace:
>[143154.860000]  [<c012ed51>] sys_pause+0x14/0x1b
>[143154.864000]  [<c0102eac>] syscall_call+0x7/0xb
>[143154.868000]  =======================
>[143154.872000] gpm           S F5239B30     0  5533      1          5551  5524 (NOTLB)
>[143154.880000]        f5239b44 00000086 00000002 f5239b30 f5239b2c 00000000 00001000 00000007 
>[143154.888000]        f7f28494 f7f28494 00000007 f61268d0 39d31700 00004e9c 00000000 f61269f4 
>[143154.896000]        c3821780 00000000 0148922a 00000000 00000286 f5239b54 f5239b44 c012c2e5 
>[143154.904000] Call Trace:
>[143154.908000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143154.912000]  [<c0175c30>] do_select+0x466/0x4ce
>[143154.916000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143154.924000]  [<c01764e1>] sys_select+0x99/0x16c
>[143154.928000]  [<c0102eac>] syscall_call+0x7/0xb
>[143154.932000]  =======================
>[143154.936000] crond         S F55E9F18     0  5551      1  6930    5599  5533 (NOTLB)
>[143154.944000]        f55e9f2c 00000082 00000002 f55e9f18 f55e9f14 00000000 c383a8d8 f55e9ee0 
>[143154.952000]        c013795c c383a8ec 00000007 f5c6cc70 8552ca00 0000822b 00000000 f5c6cd94 
>[143154.960000]        c3839780 00000003 0220d405 00000000 c01377bb 2514b5bb 0000823e 00000000 
>[143154.968000] Call Trace:
>[143154.972000]  [<c03c8160>] do_nanosleep+0x3e/0x6a
>[143154.976000]  [<c0137812>] hrtimer_nanosleep+0x4c/0xf9
>[143154.980000]  [<c0137906>] sys_nanosleep+0x47/0x58
>[143154.984000]  [<c0102eac>] syscall_call+0x7/0xb
>[143154.992000]  =======================
>[143154.996000] xfs           S F5C19B30     0  5599      1          5616  5551 (NOTLB)
>[143155.000000]        f5c19b44 00000086 00000002 f5c19b30 f5c19b2c 00000000 00000000 f5c19af4 
>[143155.012000]        00000282 00000003 0000000a f5c666d0 ea0aba00 000081e7 00000000 f5c667f4 
>[143155.020000]        c3821780 00000000 021fb875 00000000 00000286 f5c19b54 f5c19b44 c012c2e5 
>[143155.028000] Call Trace:
>[143155.028000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143155.036000]  [<c0175c30>] do_select+0x466/0x4ce
>[143155.040000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143155.044000]  [<c01764e1>] sys_select+0x99/0x16c
>[143155.048000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.052000]  =======================
>[143155.056000] atd           S F674DF18     0  5616      1          5633  5599 (NOTLB)
>[143155.064000]        f674df2c 00000086 00000002 f674df18 f674df14 00000000 c38228d8 f674dee0 
>[143155.072000]        c013795c c38228ec 00000007 f5c801f0 04c11500 0000822a 00000000 f5c80314 
>[143155.084000]        c3821780 00000000 0220cdb8 00000000 c01377bb 859f16d4 00008274 00000000 
>[143155.092000] Call Trace:
>[143155.092000]  [<c03c8160>] do_nanosleep+0x3e/0x6a
>[143155.096000]  [<c0137812>] hrtimer_nanosleep+0x4c/0xf9
>[143155.104000]  [<c0137906>] sys_nanosleep+0x47/0x58
>[143155.108000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.112000]  =======================
>[143155.116000] avahi-daemon  S F5263BD8     0  5633      1  5634    5643  5616 (NOTLB)
>[143155.124000]        f5263bec 00000082 00000002 f5263bd8 f5263bd4 00000000 00000046 ffffffff 
>[143155.132000]        00000046 f5263ba4 0000000a f608c330 df741300 0000822f 00000000 f608c454 
>[143155.140000]        c3821780 00000000 0220e646 00000000 00000286 f5263bfc f5263bec c012c2e5 
>[143155.148000] Call Trace:
>[143155.152000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143155.156000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143155.160000]  [<c01757c8>] sys_poll+0x34/0x36
>[143155.168000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.172000]  =======================
>[143155.176000] avahi-daemon  S F5615DAC     0  5634   5633                     (NOTLB)
>[143155.184000]        f5615dc0 00000086 00000002 f5615dac f5615da8 00000000 f5615f3c f5615d74 
>[143155.192000]        f5615e44 c03679f6 00000002 f7e641f0 d7b90d00 00000007 00000000 f7e64314 
>[143155.200000]        c3829780 00000001 fffefbf0 00000000 00000000 f5615dc4 f5615dc4 00000246 
>[143155.208000] Call Trace:
>[143155.212000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.216000]  [<c03c1f71>] unix_stream_recvmsg+0x1ec/0x438
>[143155.220000]  [<c0367145>] sock_aio_read+0xd8/0xe4
>[143155.224000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143155.232000]  [<c016b3c6>] vfs_read+0xa2/0x142
>[143155.236000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.240000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.244000]  =======================
>[143155.248000] hald          S F52F9BD8     0  5643      1  5644    5693  5633 (NOTLB)
>[143155.256000]        f52f9bec 00000082 00000002 f52f9bd8 f52f9bd4 00000000 00000000 00000001 
>[143155.264000]        f53bb9a8 f7eabd1c 00000007 f608adb0 8afddf00 0000822e 00000000 f608aed4 
>[143155.272000]        c3839780 00000003 0220e0b2 00000000 00000286 f52f9bfc f52f9bec c012c2e5 
>[143155.280000] Call Trace:
>[143155.284000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143155.288000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143155.292000]  [<c01757c8>] sys_poll+0x34/0x36
>[143155.296000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.300000]  =======================
>[143155.304000] hald-runner   S F51F5BD8     0  5644   5643  5650               (NOTLB)
>[143155.312000]        f51f5bec 00000082 00000002 f51f5bd8 f51f5bd4 00000000 00800000 f6183630 
>[143155.320000]        c1862ba8 f51f5ba0 00000001 f7ee97f0 75cb5c00 00000008 00000000 f7ee9914 
>[143155.328000]        c3829780 00000001 fffefe88 00000000 f51f5bd0 00000246 f51f5c64 f51f5be4 
>[143155.340000] Call Trace:
>[143155.340000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.348000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143155.352000]  [<c01757c8>] sys_poll+0x34/0x36
>[143155.356000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.360000]  =======================
>[143155.364000] hald-addon-ac S F5CCA7C4     0  5650   5644          5651       (NOTLB)
>[143155.372000]        f5621dc0 00000082 00000246 f5cca7c4 f5cca7c0 f5204f04 f5204ec8 c39d5af8 
>[143155.380000]        f5621da0 c01c4d70 00000007 c39c18b0 46966600 00000008 00000000 c39c19d4 
>[143155.388000]        c3831780 00000002 f5621d9c 00000001 00000001 f5621dc4 f5621dc4 00000246 
>[143155.396000] Call Trace:
>[143155.400000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.404000]  [<c03c1f71>] unix_stream_recvmsg+0x1ec/0x438
>[143155.408000]  [<c0367145>] sock_aio_read+0xd8/0xe4
>[143155.416000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143155.420000]  [<c016b3c6>] vfs_read+0xa2/0x142
>[143155.424000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.428000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.432000]  =======================
>[143155.436000] hald-addon-cp S F5C95BD8     0  5651   5644          5659  5650 (NOTLB)
>[143155.444000]        f5c95bec 00000082 00000002 f5c95bd8 f5c95bd4 00000000 0091f119 00000000 
>[143155.452000]        0091db41 00000000 00000007 f7edc230 46d36f00 00000008 00000000 f7edc354 
>[143155.460000]        c3831780 00000002 c39f3058 00000246 f5c95bd0 00000246 f5c95c64 f5c95be4 
>[143155.468000] Call Trace:
>[143155.472000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.476000]  [<c01756b6>] do_sys_poll+0x2cf/0x3ad
>[143155.480000]  [<c01757c8>] sys_poll+0x34/0x36
>[143155.488000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.492000]  =======================
>[143155.496000] hald-addon-ke S F51F1F30     0  5659   5644          5662  5651 (NOTLB)
>[143155.504000]        f51f1f44 00000082 00000002 f51f1f30 f51f1f2c 00000000 f53bcf78 f61c2238 
>[143155.512000]        f61c2238 f51f1f00 00000001 f7ec4710 4eb19800 00000008 003d0900 f7ec4834 
>[143155.520000]        c3831780 00000002 fffefde0 00000000 00000246 f51f1f2c 00000246 f51f1f50 
>[143155.528000] Call Trace:
>[143155.532000]  [<c03493d1>] evdev_read+0xce/0x160
>[143155.536000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.540000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.544000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.548000]  =======================
>[143155.552000] hald-addon-ke S F52F3F30     0  5662   5644                5659 (NOTLB)
>[143155.560000]        f52f3f44 00000086 00000002 f52f3f30 f52f3f2c 00000000 f4c97858 f61c2430 
>[143155.568000]        f61c2430 f52f3f00 00000001 c39c1350 4fe2c500 00000008 00000000 c39c1474 
>[143155.576000]        c3839780 00000003 fffefde8 00000000 00000246 f52f3f2c 00000246 f52f3f50 
>[143155.584000] Call Trace:
>[143155.588000]  [<c03493d1>] evdev_read+0xce/0x160
>[143155.592000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.596000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.600000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.604000]  =======================
>[143155.608000] mingetty      S 00000001     0  5693      1          5694  5643 (NOTLB)
>[143155.616000]        f6357ec0 00000086 00000086 00000001 f6357eb8 c011eda8 00000001 032e6980 
>[143155.624000]        00000000 00000003 00000005 c39d9930 7e60a000 00000008 00000000 c39d9a54 
>[143155.632000]        c3831780 00000002 00000080 032d6980 c3822320 f6357ec4 f6357ec4 00000246 
>[143155.644000] Call Trace:
>[143155.644000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.652000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143155.656000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143155.660000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.664000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.668000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.672000]  =======================
>[143155.676000] mingetty      S F5623EAC     0  5694      1          5695  5693 (NOTLB)
>[143155.684000]        f5623ec0 00000086 00000002 f5623eac f5623ea8 00000000 f7ce7628 f7ce7628 
>[143155.692000]        0000000f f583d478 00000004 c39b4850 7e60a000 00000008 00000000 c39b4974 
>[143155.700000]        c3839780 00000003 fffefeab 00000000 00000004 f5623ec4 f5623ec4 00000246 
>[143155.708000] Call Trace:
>[143155.712000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.716000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143155.720000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143155.724000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.732000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.736000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.740000]  =======================
>[143155.744000] mingetty      S F525FEAC     0  5695      1          5698  5694 (NOTLB)
>[143155.752000]        f525fec0 00000082 00000002 f525feac f525fea8 00000000 00000000 00000000 
>[143155.760000]        0000000f f583c03c 00000004 f628f050 7e9da900 00000008 00000000 f628f174 
>[143155.768000]        c3821780 00000000 fffefeab 00000000 00000282 f525fec4 f525fec4 00000246 
>[143155.776000] Call Trace:
>[143155.780000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.784000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143155.788000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143155.792000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.796000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.800000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.804000]  =======================
>[143155.808000] mingetty      S F7EDBEAC     0  5698      1          5699  5695 (NOTLB)
>[143155.816000]        f7edbec0 00000082 00000002 f7edbeac f7edbea8 00000000 00000000 00000000 
>[143155.824000]        0000000f f55ded60 00000004 f7f386d0 7e9da900 00000008 00000000 f7f387f4 
>[143155.836000]        c3831780 00000002 fffefea9 00000000 00000282 f7edbec4 f7edbec4 00000246 
>[143155.844000] Call Trace:
>[143155.844000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.852000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143155.856000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143155.860000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.864000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.868000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.872000]  =======================
>[143155.876000] mingetty      S C02D5D43     0  5699      1          5701  5698 (NOTLB)
>[143155.884000]        f523bec0 00000082 f523bef0 c02d5d43 00000020 00000000 00000000 00000000 
>[143155.892000]        0000000f f7422de0 00000005 f5c51110 7e9da900 00000008 00000000 f5c51234 
>[143155.900000]        c3831780 00000002 c01c4168 00000001 00000282 f523bec4 f523bec4 00000246 
>[143155.908000] Call Trace:
>[143155.912000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.916000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143155.920000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143155.928000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.932000]  [<c016b835>] sys_read+0x3d/0x61
>[143155.936000]  [<c0102eac>] syscall_call+0x7/0xb
>[143155.940000]  =======================
>[143155.944000] mingetty      S C02D5D43     0  5701      1          8524  5699 (NOTLB)
>[143155.952000]        f5265ec0 00000086 f5265ef0 c02d5d43 00000020 00000000 00000000 00000000 
>[143155.960000]        0000000f f583d884 00000005 f7d57a30 7edab200 00000008 00000000 f7d57b54 
>[143155.968000]        c3829780 00000001 c01c4168 00000001 00000286 f5265ec4 f5265ec4 c38297d4 
>[143155.976000] Call Trace:
>[143155.980000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143155.984000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143155.988000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143155.992000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143155.996000]  [<c016b835>] sys_read+0x3d/0x61
>[143156.000000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.008000]  =======================
>[143156.012000] sshd          S C03C8CE2     0  5768   5398  5781    5887       (NOTLB)
>[143156.016000]        f4d9fdc0 00000086 f4d9fd58 c03c8ce2 f4d9fd84 c0179648 f4d9fdf0 f7d76b7c 
>[143156.028000]        f6746e90 00000004 00000008 f608b310 2b2f8f00 0000002f 00000000 f608b434 
>[143156.036000]        c3821780 00000000 f4d9fdf0 f7d76b7c f7dc8b98 f4d9fdc4 f4d9fdc4 c3821c4c 
>[143156.044000] Call Trace:
>[143156.044000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143156.052000]  [<c03c1f71>] unix_stream_recvmsg+0x1ec/0x438
>[143156.056000]  [<c0367145>] sock_aio_read+0xd8/0xe4
>[143156.060000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143156.064000]  [<c016b3c6>] vfs_read+0xa2/0x142
>[143156.072000]  [<c016b835>] sys_read+0x3d/0x61
>[143156.076000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.080000]  =======================
>[143156.084000] sshd          S F5C9FB30     0  5781   5768  5782               (NOTLB)
>[143156.092000]        f5c9fb44 00000082 00000002 f5c9fb30 f5c9fb2c 00000000 00000000 00000001 
>[143156.100000]        00000000 f5c9fb08 0000000a f61278f0 2e6e7800 00008231 00000000 f6127a14 
>[143156.108000]        c3821780 00000000 0220ebc3 00000000 c04ddd90 f5c9fb40 c011d2c3 00000000 
>[143156.116000] Call Trace:
>[143156.120000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143156.124000]  [<c0175c30>] do_select+0x466/0x4ce
>[143156.128000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143156.132000]  [<c0176512>] sys_select+0xca/0x16c
>[143156.140000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.144000]  =======================
>[143156.148000] tcsh          S F4CD7F8C     0  5782   5781  5811               (NOTLB)
>[143156.156000]        f4cd7fa0 00000082 00000002 f4cd7f8c f4cd7f88 00000000 01200011 00000000 
>[143156.164000]        f4cd7f98 c0122bfc 0000000a f608a850 87e47700 0000003a 00000000 f608a974 
>[143156.172000]        c3821780 00000000 ffffd087 00000000 00000246 f4cd7f90 c012cb28 f4cd7fa8 
>[143156.180000] Call Trace:
>[143156.184000]  [<c012ee0a>] sys_rt_sigsuspend+0xb2/0xc8
>[143156.188000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.192000]  =======================
>[143156.196000] tcsh          S F4CABF8C     0  5811   5782  5858               (NOTLB)
>[143156.204000]        f4cabfa0 00200086 00000002 f4cabf8c f4cabf88 00000000 01200011 00000000 
>[143156.212000]        f4cabf98 c0122bfc 0000000a f7ca1630 d872c900 00000068 00000000 f7ca1754 
>[143156.220000]        c3831780 00000002 000092cc 00000000 00200246 f4cabf90 c012cb28 f4cabfa8 
>[143156.228000] Call Trace:
>[143156.232000]  [<c012ee0a>] sys_rt_sigsuspend+0xb2/0xc8
>[143156.236000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.240000]  =======================
>[143156.244000] gfs2_scand    S 00000001     0  5836     19          5837  5476 (L-TLB)
>[143156.252000]        f5927f78 00000046 f5927f50 00000001 0000007b 0000007b 00000003 f66c5394 
>[143156.260000]        00000292 f5927f30 0000000a f6177970 d627b400 0000822f 06ea0500 f6177a94 
>[143156.268000]        c3831780 00000002 00000286 f5927f58 00000286 f5927f88 f5927f78 c012c2e5 
>[143156.276000] Call Trace:
>[143156.280000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143156.284000]  [<c03c78c0>] schedule_timeout_interruptible+0x15/0x17
>[143156.292000]  [<c023b664>] gfs2_scand+0x36/0x46
>[143156.296000]  [<c0134edb>] kthread+0xb3/0xdc
>[143156.300000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143156.304000]  =======================
>[143156.308000] gfs2_glockd   D F4C73E70     0  5837     19          5841  5836 (L-TLB)
>[143156.316000]        f4c73e84 00000046 00000002 f4c73e70 f4c73e6c 00000000 c03c8d49 c3821780 
>[143156.324000]        00000000 f4c73e38 00000009 c39b5870 aa987f00 0000007e 00000000 c39b5994 
>[143156.332000]        c3829780 00000001 0000ee51 00000000 c0154ccd 0000000e f4c73e94 0000000d 
>[143156.344000] Call Trace:
>[143156.344000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143156.352000]  [<c03c898f>] rwsem_down_write_failed+0x1b/0x23
>[143156.356000]  [<c03c8a42>] call_rwsem_down_write_failed+0x6/0x8
>[143156.364000]  [<c0247167>] gfs2_log_flush+0x1b/0x2c3
>[143156.368000]  [<c024405d>] inode_go_sync+0x3b/0x91
>[143156.372000]  [<c0243bc0>] gfs2_glock_drop_th+0xd0/0x15c
>[143156.376000]  [<c0243fab>] inode_go_drop_th+0x12/0x15
>[143156.384000]  [<c024252f>] run_queue+0x110/0x31e
>[143156.388000]  [<c024348b>] gfs2_glmutex_unlock+0x2c/0x42
>[143156.392000]  [<c0243631>] gfs2_reclaim_glock+0xcd/0xdb
>[143156.396000]  [<c023b59a>] gfs2_glockd+0x16/0xaa
>[143156.404000]  [<c0134edb>] kthread+0xb3/0xdc
>[143156.408000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143156.412000]  =======================
>[143156.416000] gfs2_recoverd S F5261F64     0  5841     19          5842  5837 (L-TLB)
>[143156.424000]        f5261f78 00000046 00000002 f5261f64 f5261f60 00000000 00000000 c39b42f0 
>[143156.432000]        f5261f44 c0101fce 00000002 c39b42f0 82e0ae00 0000822a 00000000 c39b4414 
>[143156.440000]        c3821780 00000000 0220cfc9 00000000 00000286 f5261f88 f5261f78 c012c2e5 
>[143156.448000] Call Trace:
>[143156.452000]  [<c03c7876>] schedule_timeout+0x70/0x8e
>[143156.456000]  [<c03c78c0>] schedule_timeout_interruptible+0x15/0x17
>[143156.464000]  [<c023b6aa>] gfs2_recoverd+0x36/0x46
>[143156.468000]  [<c0134edb>] kthread+0xb3/0xdc
>[143156.472000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143156.476000]  =======================
>[143156.480000] gfs2_logd     D F4C9FECC     0  5842     19          5843  5841 (L-TLB)
>[143156.488000]        f4c9fee0 00000046 00000002 f4c9fecc f4c9fec8 00000000 f6177410 f4c9fea8 
>[143156.496000]        c0101fce c392d210 0000000a f6177410 49171d00 00000082 00000000 f6177534 
>[143156.504000]        c3829780 00000001 0000fd80 00000000 00000002 f4c9ff14 f4c9ff10 00000000 
>[143156.512000] Call Trace:
>[143156.516000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143156.524000]  [<c03c898f>] rwsem_down_write_failed+0x1b/0x23
>[143156.528000]  [<c03c8a42>] call_rwsem_down_write_failed+0x6/0x8
>[143156.536000]  [<c0247167>] gfs2_log_flush+0x1b/0x2c3
>[143156.540000]  [<c023b766>] gfs2_logd+0xac/0x14e
>[143156.544000]  [<c0134edb>] kthread+0xb3/0xdc
>[143156.548000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143156.552000]  =======================
>[143156.556000] gfs2_quotad   D F51FBD8C     0  5843     19                5842 (L-TLB)
>[143156.564000]        f51fbda0 00000046 00000002 f51fbd8c f51fbd88 00000000 f51fbd9c c0154ece 
>[143156.572000]        00000001 00000001 0000000a c39b4db0 099e0000 00000085 00000000 c39b4ed4 
>[143156.580000]        c3829780 00000001 0001090b 00000000 c03c8e09 f51fbd94 c0150d62 0000000e 
>[143156.588000] Call Trace:
>[143156.592000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143156.600000]  [<c03c898f>] rwsem_down_write_failed+0x1b/0x23
>[143156.604000]  [<c03c8a42>] call_rwsem_down_write_failed+0x6/0x8
>[143156.612000]  [<c0247167>] gfs2_log_flush+0x1b/0x2c3
>[143156.616000]  [<c024405d>] inode_go_sync+0x3b/0x91
>[143156.620000]  [<c0243bc0>] gfs2_glock_drop_th+0xd0/0x15c
>[143156.624000]  [<c0243fab>] inode_go_drop_th+0x12/0x15
>[143156.632000]  [<c024252f>] run_queue+0x110/0x31e
>[143156.636000]  [<c024348b>] gfs2_glmutex_unlock+0x2c/0x42
>[143156.640000]  [<c0243631>] gfs2_reclaim_glock+0xcd/0xdb
>[143156.644000]  [<c0242669>] run_queue+0x24a/0x31e
>[143156.652000]  [<c0242cf5>] gfs2_glock_nq+0x162/0x1ba
>[143156.656000]  [<c0255e6e>] gfs2_statfs_sync+0x43/0x227
>[143156.660000]  [<c023b854>] gfs2_quotad+0x4c/0x13c
>[143156.664000]  [<c0134edb>] kthread+0xb3/0xdc
>[143156.668000]  [<c0103afb>] kernel_thread_helper+0x7/0x10
>[143156.676000]  =======================
>[143156.680000] dbench        S C38228D8     0  5858   5811  5859               (NOTLB)
>[143156.684000]        f389df30 00200082 f389dee8 c38228d8 f389ded8 c013795c c38228ec f389df0c 
>[143156.696000]        f389def8 c39776c8 0000000a f5c81770 2e6e7800 00008231 00000000 f5c81894 
>[143156.704000]        c3821780 00000000 1187affc 00200246 c3977680 00200246 f389df1c c38217d4 
>[143156.712000] Call Trace:
>[143156.712000]  [<c012602f>] do_wait+0x89f/0x996
>[143156.720000]  [<c0126156>] sys_wait4+0x30/0x32
>[143156.724000]  [<c012617f>] sys_waitpid+0x27/0x29
>[143156.728000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.732000]  =======================
>[143156.736000] dbench        D F35D5D60     0  5859   5858          5860       (NOTLB)
>[143156.744000]        f35d5d74 00200086 00000002 f35d5d60 f35d5d5c 00000000 f35d5e3c f337051c 
>[143156.752000]        00000000 f337051c 00000007 f6146e70 aa987f00 0000007e 00000000 f6146f94 
>[143156.760000]        c3821780 00000000 0000ee4e 00000000 c014f8ac c164dc00 00000000 c164dc00 
>[143156.768000] Call Trace:
>[143156.772000]  [<c03c7205>] wait_for_completion+0x74/0xaa
>[143156.776000]  [<c0242a66>] glock_wait_internal+0xf2/0x21f
>[143156.784000]  [<c0242d0d>] gfs2_glock_nq+0x17a/0x1ba
>[143156.788000]  [<c024492e>] gfs2_change_nlink+0x78/0xaa
>[143156.792000]  [<c024e8e7>] gfs2_unlink+0xc3/0x100
>[143156.796000]  [<c0171b8b>] vfs_unlink+0x73/0xaf
>[143156.800000]  [<c01734ad>] do_unlinkat+0x87/0x112
>[143156.808000]  [<c0173548>] sys_unlink+0x10/0x12
>[143156.812000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.816000]  =======================
>[143156.820000] dbench        D F35F7C88     0  5860   5858          5861  5859 (NOTLB)
>[143156.828000]        f35f7c9c 00200082 00000002 f35f7c88 f35f7c84 00000000 f363812c f35f7c48 
>[143156.836000]        c03c7ed7 f35f7c58 00000007 f61473d0 aa987f00 0000007e 00000000 f61474f4 
>[143156.844000]        c3839780 00000003 0000ee50 00000000 f363812c 00000000 83bf2b46 f3638148 
>[143156.852000] Call Trace:
>[143156.856000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143156.860000]  [<c03c89b4>] rwsem_down_read_failed+0x1d/0x25
>[143156.868000]  [<c03c8a37>] call_rwsem_down_read_failed+0x7/0xc
>[143156.872000]  [<c02478af>] gfs2_log_reserve+0xef/0x100
>[143156.876000]  [<c02581d5>] gfs2_trans_begin+0xdb/0x112
>[143156.884000]  [<c024b832>] gfs2_prepare_write+0x169/0x230
>[143156.888000]  [<c01500fc>] generic_file_buffered_write+0x20b/0x572
>[143156.892000]  [<c01508b4>] __generic_file_aio_write_nolock+0x451/0x4be
>[143156.900000]  [<c0150977>] generic_file_aio_write+0x56/0xb4
>[143156.904000]  [<c016aab4>] do_sync_write+0xc2/0xff
>[143156.912000]  [<c016b270>] vfs_write+0x90/0x144
>[143156.916000]  [<c016b95b>] sys_pwrite64+0x45/0x5c
>[143156.920000]  [<c0102eac>] syscall_call+0x7/0xb
>[143156.924000]  =======================
>[143156.928000] dbench        D 00000000     0  5861   5858          5862  5860 (NOTLB)
>[143156.936000]        f35f9d08 00200086 c03c80fb 00000000 c382ea80 f35f9cc8 c03d3e20 f35f9cd4 
>[143156.944000]        00200246 22222222 00000007 f6146910 aa987f00 0000007e 00000000 f6146a34 
>[143156.952000]        c3831780 00000002 c03c7ed7 f35f9d20 c0255a90 c03c8df5 ffffffff f4c94e10 
>[143156.960000] Call Trace:
>[143156.964000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143156.968000]  [<c03c89b4>] rwsem_down_read_failed+0x1d/0x25
>[143156.976000]  [<c03c8a37>] call_rwsem_down_read_failed+0x7/0xc
>[143156.980000]  [<c02478af>] gfs2_log_reserve+0xef/0x100
>[143156.988000]  [<c02581d5>] gfs2_trans_begin+0xdb/0x112
>[143156.992000]  [<c023ad2b>] do_strip+0x1b7/0x3a6
>[143156.996000]  [<c0239b0b>] recursive_scan+0x108/0x193
>[143157.000000]  [<c0239c63>] trunc_dealloc+0xcd/0xea
>[143157.004000]  [<c0239c8c>] gfs2_file_dealloc+0xc/0xe
>[143157.012000]  [<c0250641>] gfs2_delete_inode+0xe3/0x157
>[143157.016000]  [<c017b6cb>] generic_delete_inode+0x8f/0xf9
>[143157.020000]  [<c017b747>] generic_drop_inode+0x12/0x13e
>[143157.028000]  [<c017ade6>] iput+0x63/0x66
>[143157.032000]  [<c01734cf>] do_unlinkat+0xa9/0x112
>[143157.036000]  [<c0173548>] sys_unlink+0x10/0x12
>[143157.040000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.044000]  =======================
>[143157.048000] dbench        D F35FBC88     0  5862   5858          5863  5861 (NOTLB)
>[143157.056000]        f35fbc9c 00200082 00000002 f35fbc88 f35fbc84 00000000 f3638074 f35fbc48 
>[143157.064000]        c03c7ed7 f35fbc8c 00000007 f61463b0 aa987f00 0000007e 00000000 f61464d4 
>[143157.072000]        c3831780 00000002 0000ee4f 00000000 f3638074 00000000 00000000 f3638090 
>[143157.080000] Call Trace:
>[143157.084000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143157.088000]  [<c03c89b4>] rwsem_down_read_failed+0x1d/0x25
>[143157.096000]  [<c03c8a37>] call_rwsem_down_read_failed+0x7/0xc
>[143157.100000]  [<c02478af>] gfs2_log_reserve+0xef/0x100
>[143157.108000]  [<c02581d5>] gfs2_trans_begin+0xdb/0x112
>[143157.112000]  [<c024b832>] gfs2_prepare_write+0x169/0x230
>[143157.116000]  [<c01500fc>] generic_file_buffered_write+0x20b/0x572
>[143157.124000]  [<c01508b4>] __generic_file_aio_write_nolock+0x451/0x4be
>[143157.128000]  [<c0150977>] generic_file_aio_write+0x56/0xb4
>[143157.136000]  [<c016aab4>] do_sync_write+0xc2/0xff
>[143157.140000]  [<c016b270>] vfs_write+0x90/0x144
>[143157.144000]  [<c016b95b>] sys_pwrite64+0x45/0x5c
>[143157.148000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.152000]  =======================
>[143157.156000] dbench        D 00000001     0  5863   5858                5862 (NOTLB)
>[143157.164000]        f35fdb8c 00200082 f35fdba8 00000001 00000001 00000000 f35fdb48 c0114a88 
>[143157.172000]        f35fdb3c c03c8d49 00000007 f5c90cf0 aa987f00 0000007e 00000000 f5c90e14 
>[143157.184000]        c3821780 00000000 f35fdbe0 f35fdbe0 f35fdba0 d9952244 f35fdb84 c0154ccd 
>[143157.192000] Call Trace:
>[143157.192000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143157.200000]  [<c03c898f>] rwsem_down_write_failed+0x1b/0x23
>[143157.204000]  [<c03c8a42>] call_rwsem_down_write_failed+0x6/0x8
>[143157.212000]  [<c0247167>] gfs2_log_flush+0x1b/0x2c3
>[143157.216000]  [<c024405d>] inode_go_sync+0x3b/0x91
>[143157.220000]  [<c0243bc0>] gfs2_glock_drop_th+0xd0/0x15c
>[143157.224000]  [<c0243fab>] inode_go_drop_th+0x12/0x15
>[143157.232000]  [<c024252f>] run_queue+0x110/0x31e
>[143157.236000]  [<c024348b>] gfs2_glmutex_unlock+0x2c/0x42
>[143157.240000]  [<c0243631>] gfs2_reclaim_glock+0xcd/0xdb
>[143157.244000]  [<c0242669>] run_queue+0x24a/0x31e
>[143157.252000]  [<c0242cf5>] gfs2_glock_nq+0x162/0x1ba
>[143157.256000]  [<c0244edf>] gfs2_inode_lookup+0x15f/0x1b3
>[143157.260000]  [<c0245e7d>] gfs2_createi+0xa68/0xcaa
>[143157.264000]  [<c024f58f>] gfs2_create+0x70/0x127
>[143157.272000]  [<c01715e0>] vfs_create+0xa7/0x118
>[143157.276000]  [<c0173cfa>] open_namei+0x170/0x581
>[143157.280000]  [<c0169962>] do_filp_open+0x26/0x3b
>[143157.284000]  [<c01699ba>] do_sys_open+0x43/0xc2
>[143157.288000]  [<c0169a71>] sys_open+0x1c/0x1e
>[143157.292000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.296000]  =======================
>[143157.300000] sshd          S C03C8CE2     0  5887   5398  5889          5768 (NOTLB)
>[143157.308000]        d7711dc0 00000082 d7711d58 c03c8ce2 d7711d84 c0179648 d7711df0 f7d76b7c 
>[143157.316000]        f6746e90 00000004 00000008 f5c6c710 e0592400 00000097 00000000 f5c6c834 
>[143157.328000]        c3821780 00000000 d7711df0 f7d76b7c f7dc8b98 d7711dc4 d7711dc4 c3821c4c 
>[143157.336000] Call Trace:
>[143157.336000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143157.344000]  [<c03c1f71>] unix_stream_recvmsg+0x1ec/0x438
>[143157.348000]  [<c0367145>] sock_aio_read+0xd8/0xe4
>[143157.352000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143157.356000]  [<c016b3c6>] vfs_read+0xa2/0x142
>[143157.360000]  [<c016b835>] sys_read+0x3d/0x61
>[143157.368000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.372000]  =======================
>[143157.376000] sshd          S EC83BB30     0  5889   5887  5890               (NOTLB)
>[143157.384000]        ec83bb44 00000082 00000002 ec83bb30 ec83bb2c 00000000 00000000 00000000 
>[143157.392000]        c0505800 00000014 0000000a c3948610 685e0300 00008231 00000000 c3948734 
>[143157.400000]        c3821780 00000000 0220ecb4 00000000 c04ddd90 ec83bb40 c011d2c3 00000000 
>[143157.408000] Call Trace:
>[143157.412000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143157.416000]  [<c0175c30>] do_select+0x466/0x4ce
>[143157.420000]  [<c0175f48>] core_sys_select+0x2b0/0x2d5
>[143157.424000]  [<c0176512>] sys_select+0xca/0x16c
>[143157.428000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.436000]  =======================
>[143157.436000] tcsh          S E9029F8C     0  5890   5889  5916               (NOTLB)
>[143157.444000]        e9029fa0 00000082 00000002 e9029f8c e9029f88 00000000 01200011 00000000 
>[143157.452000]        e9029f98 c0122bfc 00000009 f5c917b0 4e055000 00000098 00000000 f5c918d4 
>[143157.464000]        c3821780 00000000 000159cb 00000000 00000246 e9029f90 c012cb28 e9029fa8 
>[143157.472000] Call Trace:
>[143157.472000]  [<c012ee0a>] sys_rt_sigsuspend+0xb2/0xc8
>[143157.480000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.484000]  =======================
>[143157.488000] crond         S F2E652F4     0  6930   5551  6931   10171       (NOTLB)
>[143157.496000]        f4c5be48 00000082 00000044 f2e652f4 00000001 f4c5bdf8 f4c5bdf8 c011bda7 
>[143157.504000]        0185a597 00000004 00000005 c39480b0 4bfce000 00001d9b 00000000 c39481d4 
>[143157.512000]        c3839780 00000003 f4c5be40 00000246 00000246 f4c5be30 00000246 f4c5be50 
>[143157.520000] Call Trace:
>[143157.524000]  [<c0170078>] pipe_wait+0x53/0x72
>[143157.528000]  [<c017072b>] pipe_read+0x292/0x2fe
>[143157.532000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143157.536000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143157.540000]  [<c016b835>] sys_read+0x3d/0x61
>[143157.544000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.548000]  =======================
>[143157.552000] run-parts     S F2F1BF1C     0  6931   6930  7401               (NOTLB)
>[143157.560000]        f2f1bf30 00000082 00000002 f2f1bf1c f2f1bf18 00000000 080f92a8 e396e480 
>[143157.568000]        f67ee238 f7e4a170 00000007 f7ee87d0 2ead8b00 00001da3 00000000 f7ee88f4 
>[143157.580000]        c3821780 00000000 007b29fd 00000000 f61171dc 00000246 f2f1bf1c 00000246 
>[143157.588000] Call Trace:
>[143157.588000]  [<c012602f>] do_wait+0x89f/0x996
>[143157.592000]  [<c0126156>] sys_wait4+0x30/0x32
>[143157.600000]  [<c012617f>] sys_waitpid+0x27/0x29
>[143157.604000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.608000]  =======================
>[143157.612000] mlocate.cron  S F60F9F1C     0  7401   6931  7406    7402       (NOTLB)
>[143157.620000]        f60f9f30 00000082 00000002 f60f9f1c f60f9f18 00000000 080f6c78 e4058780 
>[143157.628000]        f67eee08 c2d80c88 00000004 f7ce00f0 2fa1af00 00001da3 00000000 f7ce0214 
>[143157.636000]        c3829780 00000001 007b2a02 00000000 c394c680 00000246 f60f9f1c 00000246 
>[143157.644000] Call Trace:
>[143157.648000]  [<c012602f>] do_wait+0x89f/0x996
>[143157.652000]  [<c0126156>] sys_wait4+0x30/0x32
>[143157.656000]  [<c012617f>] sys_waitpid+0x27/0x29
>[143157.660000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.664000]  =======================
>[143157.668000] awk           S C6BE3E34     0  7402   6931                7401 (NOTLB)
>[143157.676000]        c6be3e48 00000086 00000002 c6be3e34 c6be3e30 00000000 00000000 ebe79e98 
>[143157.684000]        00000000 c6be3e70 00000006 f7e4a170 2eea9400 00001da3 003d0900 f7e4a294 
>[143157.692000]        c3829780 00000001 007b29fe 00000000 00000246 c6be3e30 00000246 c6be3e50 
>[143157.700000] Call Trace:
>[143157.704000]  [<c0170078>] pipe_wait+0x53/0x72
>[143157.708000]  [<c017072b>] pipe_read+0x292/0x2fe
>[143157.712000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143157.720000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143157.724000]  [<c016b835>] sys_read+0x3d/0x61
>[143157.728000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.732000]  =======================
>[143157.736000] updatedb      D CEF27DEC     0  7406   7401                     (NOTLB)
>[143157.744000]        cef27e00 00000086 00000002 cef27dec cef27de8 00000000 00000001 00000246 
>[143157.752000]        c0241fdc 00000050 0000000a f7e4ac30 61588300 00001da5 00000000 f7e4ad54 
>[143157.760000]        c3829780 00000001 007b3336 00000000 00000010 00000282 c04cec80 f7ffdec8 
>[143157.768000] Call Trace:
>[143157.772000]  [<c027efe2>] rwsem_down_failed_common+0x14c/0x16b
>[143157.776000]  [<c03c89b4>] rwsem_down_read_failed+0x1d/0x25
>[143157.784000]  [<c03c8a37>] call_rwsem_down_read_failed+0x7/0xc
>[143157.788000]  [<c02478af>] gfs2_log_reserve+0xef/0x100
>[143157.792000]  [<c02581d5>] gfs2_trans_begin+0xdb/0x112
>[143157.800000]  [<c02445a4>] gfs2_glock_nq_atime+0x1ab/0x2c1
>[143157.804000]  [<c024ca99>] gfs2_readdir+0x62/0xa7
>[143157.808000]  [<c01750c0>] vfs_readdir+0x4e/0x7a
>[143157.812000]  [<c017514a>] sys_getdents64+0x5e/0xa0
>[143157.820000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.824000]  =======================
>[143157.828000] agetty        S F067BEAC     0  8524      1          8561  5701 (NOTLB)
>[143157.836000]        f067bec0 00000082 00000002 f067beac f067bea8 00000000 00000000 bf89f61c 
>[143157.844000]        f067bf48 c02cab8d 0000000a f5caa2f0 efc7ba00 00003e36 00000000 f5caa414 
>[143157.852000]        c3829780 00000001 0103cd74 00000000 00000286 f067bec4 f067bec4 00000246 
>[143157.860000] Call Trace:
>[143157.864000]  [<c03c781c>] schedule_timeout+0x16/0x8e
>[143157.868000]  [<c02ccbd4>] read_chan+0x2e8/0x510
>[143157.872000]  [<c02c9de6>] tty_read+0x66/0xb5
>[143157.876000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143157.880000]  [<c016b835>] sys_read+0x3d/0x61
>[143157.884000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.888000]  =======================
>[143157.892000] lsof          D F06C5DE8     0  8561      1                8524 (NOTLB)
>[143157.900000]        f06c5dfc 00200082 00000002 f06c5de8 f06c5de4 00000000 f06c5da4 f5cc8048 
>[143157.908000]        f5cc80c8 f5cc8048 00000007 f61df9b0 2b7fba00 00003f29 00000000 f61dfad4 
>[143157.916000]        c3821780 00000000 0107c574 00000000 c017d488 f06c5ef4 00000000 f06c5e44 
>[143157.928000] Call Trace:
>[143157.928000]  [<c03c7205>] wait_for_completion+0x74/0xaa
>[143157.936000]  [<c0242a66>] glock_wait_internal+0xf2/0x21f
>[143157.940000]  [<c0242d0d>] gfs2_glock_nq+0x17a/0x1ba
>[143157.944000]  [<c024e6b4>] gfs2_getattr+0x91/0xa3
>[143157.948000]  [<c016d571>] vfs_getattr+0x24/0x34
>[143157.952000]  [<c016da8a>] vfs_stat_fd+0x2e/0x41
>[143157.960000]  [<c016db1f>] vfs_stat+0x11/0x13
>[143157.964000]  [<c016db35>] sys_stat64+0x14/0x28
>[143157.968000]  [<c0102eac>] syscall_call+0x7/0xb
>[143157.972000]  =======================
>[143157.976000] crond         S F00CBE34     0 10171   5551 10172          6930 (NOTLB)
>[143157.984000]        f00cbe48 00000086 00000002 f00cbe34 f00cbe30 00000000 f00cbdf8 c011bda7 
>[143157.992000]        0185a597 00000004 00000005 f61deef0 a4ab0700 00006c2d 00000000 f61df014 
>[143158.000000]        c3829780 00000001 01c49697 00000000 00000246 f00cbe30 00000246 f00cbe50 
>[143158.008000] Call Trace:
>[143158.012000]  [<c0170078>] pipe_wait+0x53/0x72
>[143158.016000]  [<c017072b>] pipe_read+0x292/0x2fe
>[143158.020000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143158.024000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143158.028000]  [<c016b835>] sys_read+0x3d/0x61
>[143158.032000]  [<c0102eac>] syscall_call+0x7/0xb
>[143158.040000]  =======================
>[143158.044000] run-parts     S C7AB7F1C     0 10172  10171 10640               (NOTLB)
>[143158.048000]        c7ab7f30 00000082 00000002 c7ab7f1c c7ab7f18 00000000 080f92a8 df8ebba0 
>[143158.060000]        c3921c10 c3991250 00000005 c39c0890 08346b00 00006c4b 00000000 c39c09b4 
>[143158.068000]        c3831780 00000002 01c511dd 00000000 e34aa050 00000246 c7ab7f1c 00000246 
>[143158.076000] Call Trace:
>[143158.076000]  [<c012602f>] do_wait+0x89f/0x996
>[143158.084000]  [<c0126156>] sys_wait4+0x30/0x32
>[143158.088000]  [<c012617f>] sys_waitpid+0x27/0x29
>[143158.092000]  [<c0102eac>] syscall_call+0x7/0xb
>[143158.096000]  =======================
>[143158.100000] mlocate.cron  S C6945F1C     0 10640  10172 10645   10641       (NOTLB)
>[143158.108000]        c6945f30 00000082 00000002 c6945f1c c6945f18 00000000 080f6c78 e614bae0 
>[143158.116000]        f7ce7e08 c2cd5e00 00000005 f5c81210 08717400 00006c4b 00000000 f5c81334 
>[143158.124000]        c3829780 00000001 01c511dd 00000000 c39771dc 00000246 c6945f1c 00000246 
>[143158.132000] Call Trace:
>[143158.136000]  [<c012602f>] do_wait+0x89f/0x996
>[143158.140000]  [<c0126156>] sys_wait4+0x30/0x32
>[143158.144000]  [<c012617f>] sys_waitpid+0x27/0x29
>[143158.148000]  [<c0102eac>] syscall_call+0x7/0xb
>[143158.152000]  =======================
>[143158.156000] awk           S F010FE34     0 10641  10172               10640 (NOTLB)
>[143158.164000]        f010fe48 00000082 00000002 f010fe34 f010fe30 00000000 00000000 d7ba9c94 
>[143158.172000]        00000000 f010fe70 00000007 c3991250 08346b00 00006c4b 00000000 c3991374 
>[143158.180000]        c3821780 00000000 01c511dd 00000000 00000246 f010fe30 00000246 f010fe50 
>[143158.192000] Call Trace:
>[143158.192000]  [<c0170078>] pipe_wait+0x53/0x72
>[143158.196000]  [<c017072b>] pipe_read+0x292/0x2fe
>[143158.200000]  [<c016abb3>] do_sync_read+0xc2/0xff
>[143158.208000]  [<c016b3b2>] vfs_read+0x8e/0x142
>[143158.212000]  [<c016b835>] sys_read+0x3d/0x61
>[143158.216000]  [<c0102eac>] syscall_call+0x7/0xb
>[143158.220000]  =======================
>[143158.224000] updatedb      D F076BDE8     0 10645  10640                     (NOTLB)
>[143158.232000]        f076bdfc 00000082 00000002 f076bde8 f076bde4 00000000 f4c4ccbc 00000007 
>[143158.240000]        00c82e74 e34fe000 00000002 f7c54030 23256400 00006c4b 003d0900 f7c54154 
>[143158.248000]        c3821780 00000000 01c5121d 00000000 c0179b26 f076bef4 f35d006c f076be44 
>[143158.256000] Call Trace:
>[143158.260000]  [<c03c7205>] wait_for_completion+0x74/0xaa
>[143158.264000]  [<c0242a66>] glock_wait_internal+0xf2/0x21f
>[143158.272000]  [<c0242d0d>] gfs2_glock_nq+0x17a/0x1ba
>[143158.276000]  [<c024e6b4>] gfs2_getattr+0x91/0xa3
>[143158.280000]  [<c016d571>] vfs_getattr+0x24/0x34
>[143158.284000]  [<c016d9e1>] vfs_lstat_fd+0x2b/0x3e
>[143158.288000]  [<c016da32>] vfs_lstat+0x11/0x13
>[143158.292000]  [<c016da48>] sys_lstat64+0x14/0x28
>[143158.300000]  [<c0102eac>] syscall_call+0x7/0xb
>[143158.304000]  =======================
>
>telnet> quit
>Connection closed.

>xenon[10:46am]-=>pwd

>/tmp/lockup
>xenon[10:46am]-=>ls

>b03-lockdump  b03-lockdump-3  b03-lockdump-4  b03-lockdump-again
>xenon[10:46am]-=>exit

>exit
>
>Script done on Thu 04 Jan 2007 10:46:14 AM CST

Actions: View

Attachments on bug 217356: 142167 | 142168 | 142255 | 142256 | 142257 | 142258 | 142905 | 144819 | 146881 | 152908