Bug 168134 - unkillable ipop3d, CROND, sshd processes
unkillable ipop3d, CROND, sshd processes
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Dave Anderson
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2005-09-12 15:31 EDT by Michael T. Rampson
Modified: 2007-11-30 17:07 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-10-19 14:54:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Michael T. Rampson 2005-09-12 15:31:24 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.10) Gecko/20050716 Firefox/1.0.6

Description of problem:
machine gets into a state where ipop3d, sshd, CROND become unkillable and a bunch of them have been spawned and cpu seems to go to 100%.  You have to reboot the machine to clear everything up.  Seems to happen once a week.

Version-Release number of selected component (if applicable):
kernel-smp-2.4.21-32.0.1.EL

How reproducible:
Always

Steps to Reproduce:
1. In this case, just let the machine run and usually in about a week, it gets into this state.  You can't login via ssh, but httpd still alive and serving.  Can't get mail, etc
2.
3.
  

Additional info:
Comment 1 Ernie Petrides 2005-09-12 16:00:58 EDT
Hello, Michael.  Thanks for your bug report.  Please report this to
Customer Support to see if they can help you determine the what's
leading to this problem.  At the very least, we'll need something
more tangible to go on to debug this problem.  My RHEL3 machine
running that release does not incur the same problem, so we'd need
a more reliable way to reproduce it.
Comment 2 Michael T. Rampson 2005-09-14 11:51:06 EDT
This may be related to an sshd issue.  In each case, someone was running a
program to try to login via ssh.  Is it possible that this causes the kernel to
run out of some resource that causes ipop3, crond, and vsftpd to wedge?  current
version of openssh is openssh-3.6.1p2-33.30.4.
Comment 3 Michael T. Rampson 2005-09-14 11:52:51 EDT
Also, this has happened on a 2nd server also running RHEL 3.0, completely
different HW.  In both cases, they are running SMP.
Comment 4 Dave Anderson 2005-09-14 13:47:49 EDT
It's impossible to speculate without evidence.

A netdump/diskdump would be best, which you could
generate with an alt-sysrq-c on the console when the
system gets into that state (presuming it still can
receive keyboard interrupts).

You might try this sequence of keyboard entries on
the system console when it gets into that state:

alt-sysrq-w
alt-sysrq-m
alt-sysrq-t

and capture the output using a serial console.

The "w" will show a trace of the active task on
each cpu, the "m" will show memory statistics,
and the "t" will trace all tasks on the system.


Comment 5 Michael T. Rampson 2005-09-14 14:39:50 EDT
Is there a way of generating this from the command-line?  I can't get into the
box when this happens.  I'm able to get in and execute commands via webmin as it
is an already running process, but trying to get in any other way seems
problematic.  You can login in, but the system never comes back with a prompt. 
I can issue commands via webmin however, although not all commands work or
affect anything (for instance restart /etc/init.d/sshd does restart ssh, but the
old ssh's don't go away and you can't get in.  /etc/init.d/crond fails.)
Comment 6 Dave Anderson 2005-09-14 15:00:04 EDT
You can:

 echo c > /proc/sysrq-trigger

or w, m or t.

BTW, this all presumes that /proc/sys/kernel/sysrq is equal to 1.
If not, you can do either of these for per-session enablement:

  echo 1 > /proc/sys/kernel/sysrq
  sysctl -w kernel.sysrq=1

or put this in your /etc/sysctl.conf to have it on permanently:

  kernel.sysrq = 1

Comment 7 Michael T. Rampson 2005-09-18 14:46:19 EDT
after doing echo 1> /proc/sys/kernel/sysrq and then doing echo c >
/proc/sysrq-trigger caused the machine to reboot.  This after trying to run the
reboot command does nothing.  Also, is there a way to direct this output to a
file in the filesystem instead of serial port?  Also, someone tried to run a
hack ipop3 which xinetd limited when it ran out of instances, but the ipop3
continued to hang around.
Comment 8 Dave Anderson 2005-09-19 09:01:02 EDT
Netdump and/or diskdump would have to be set up in order
for alt-sysrq-c to do anything other than crash/reboot
the machine.

Comment 9 Michael T. Rampson 2005-09-19 14:01:40 EDT
Here is an echo w from a machine that exhibits this:

Sep 19 09:31:07 m2 audbin[20300]: saving binary audit log /var/log/audit.d/bin.1
Sep 19 09:31:07 m2 audbin[20300]: threshold 20.00 exceeded for filesystem
/var/log/audit.d/. - free blocks down to 18.74%
Sep 19 09:31:07 m2 auditd[1550]: Notify command /usr/sbin/audbin -S
/var/log/audit.d/save.%u -C -T 20% exited with status 1
Sep 19 09:31:07 m2 auditd[1550]: output error
Sep 19 09:31:07 m2 auditd[1550]: output error
Sep 19 09:31:07 m2 auditd[1550]: output error; suspending execution
Sep 19 11:59:39 m2 kernel: SysRq : Show CPUs
Sep 19 11:59:39 m2 kernel: CPU3:
Sep 19 11:59:39 m2 kernel: edb45f1c 00000003 c01cfeaf 00000000 c03b5df4 00000077
00000006 c01d013a 
Sep 19 11:59:39 m2 kernel:        00000077 00000000 00000000 00000000 00000000
00000000 00000000 c01d009d 
Sep 19 11:59:39 m2 kernel:        00000077 00000000 00000000 00000000 00000002
f7066a00 ffffffea 00000002 
Sep 19 11:59:39 m2 kernel: Call Trace:   [<c01cfeaf>] sysrq_handle_showcpus
[kernel] 0xf (0xedb45f24)
Sep 19 11:59:39 m2 kernel: [<c01d013a>] __handle_sysrq_nolock [kernel] 0x7a
(0xedb45f38)
Sep 19 11:59:39 m2 kernel: [<c01d009d>] handle_sysrq [kernel] 0x5d (0xedb45f58)
Sep 19 11:59:39 m2 kernel: [<c01994a3>] write_sysrq_trigger [kernel] 0x53
(0xedb45f7c)
Sep 19 11:59:39 m2 kernel: [<c01648a7>] sys_write [kernel] 0x97 (0xedb45f94)
Sep 19 11:59:39 m2 kernel: 
Sep 19 11:59:39 m2 kernel: CPU2:
Sep 19 11:59:39 m2 kernel: c4eddf64 00000002 c011ca7f 00000000 00001f7c c03f6caa
c0109100 00000000 
Sep 19 11:59:39 m2 kernel:        c4edc000 c4edc000 c4edc000 c0109100 00000000
00000068 00000068 fffffffb 
Sep 19 11:59:39 m2 kernel:        c0109129 00000060 00000246 c01091c2 0602080b
00000000 00000000 00000000 
Sep 19 11:59:39 m2 kernel: Call Trace:   [<c011ca7f>]
smp_call_function_interrupt [kernel] 0x2f (0xc4eddf6c)
Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc4eddf7c)
Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc4eddf90)
Sep 19 11:59:39 m2 kernel: [<c0109129>] default_idle [kernel] 0x29 (0xc4eddfa4)
Sep 19 11:59:39 m2 kernel: [<c01091c2>] cpu_idle [kernel] 0x42 (0xc4eddfb0)
Sep 19 11:59:39 m2 kernel: [<c0129883>] printk [kernel] 0x153 (0xc4eddfcc)
Sep 19 11:59:39 m2 kernel: 
Sep 19 11:59:39 m2 kernel: CPU0:
Sep 19 11:59:39 m2 kernel: c03f5f88 00000000 c011ca7f 00000000 00001fa0 c03f6caa
c0109100 00000000 
Sep 19 11:59:39 m2 kernel:        c03f4000 c03f4000 c03f4000 c0109100 00000000
00000068 00000068 fffffffb 
Sep 19 11:59:39 m2 kernel:        c0109129 00000060 00000246 c01091c2 0002080b
00099800 c0107000 0008e000 
Sep 19 11:59:39 m2 kernel: Call Trace:   [<c011ca7f>]
smp_call_function_interrupt [kernel] 0x2f (0xc03f5f90)
Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc03f5fa0)
Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc03f5fb4)
Sep 19 11:59:39 m2 kernel: [<c0109129>] default_idle [kernel] 0x29 (0xc03f5fc8)
Sep 19 11:59:39 m2 kernel: [<c01091c2>] cpu_idle [kernel] 0x42 (0xc03f5fd4)
Sep 19 11:59:39 m2 kernel: [<c0107000>] stext [kernel] 0x0 (0xc03f5fe0)
Sep 19 11:59:39 m2 kernel: 
Sep 19 11:59:39 m2 kernel: CPU1:
Sep 19 11:59:39 m2 kernel: e9bb5ea0 00000001 c011ca7f 00000000 00001eb8 c03f6caa
00000000 e2c81d3c 
Sep 19 11:59:39 m2 kernel:        e9bb4000 00000000 00000000 00000000 ffffff00
ffff0068 00000068 fffffffb 
Sep 19 11:59:39 m2 kernel:        c017bf2d 00000060 00000286 e2c81d3c e9551520
f5af5380 00000000 00000b81 
Sep 19 11:59:39 m2 kernel: Call Trace:   [<c011ca7f>]
smp_call_function_interrupt [kernel] 0x2f (0xe9bb5ea8)
Sep 19 11:59:39 m2 kernel: [<c017bf2d>] posix_lock_file [kernel] 0x42d (0xe9bb5ee0)
Sep 19 11:59:39 m2 kernel: [<c017ced4>] fcntl_setlk [kernel] 0x174 (0xe9bb5f4c)
Sep 19 11:59:39 m2 kernel: [<c0177e66>] do_fcntl [kernel] 0x1c6 (0xe9bb5f7c)
Sep 19 11:59:39 m2 kernel: [<c0178054>] sys_fcntl64 [kernel] 0x54 (0xe9bb5f9c)
Sep 19 11:59:39 m2 kernel: 

Could this be an issue of audit shutting down because of disk space?
Comment 10 Michael T. Rampson 2005-09-19 14:05:31 EDT
Here is an 'm' dump:

Sep 19 12:03:53 m2 kernel: SysRq : Show Memory
Sep 19 12:03:53 m2 kernel: Mem-info:
Sep 19 12:03:53 m2 kernel: Zone:DMA freepages:  2880 min:     0 low:     0 high:
    0
Sep 19 12:03:53 m2 kernel: Zone:Normal freepages: 65550 min:  1278 low:  4543
high:  6303
Sep 19 12:03:53 m2 kernel: Zone:HighMem freepages: 15366 min:   255 low:  4606
high:  6909
Sep 19 12:03:53 m2 kernel: Free pages:       83797 ( 15366 HighMem)
Sep 19 12:03:53 m2 kernel: ( Active: 273633/67421, inactive_laundry: 13324,
inactive_clean: 6930, free: 83797 )
Sep 19 12:03:53 m2 kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2880
Sep 19 12:03:53 m2 kernel:   aa:2912 ac:55430 id:21437 il:3573 ic:2871 fr:65551
Sep 19 12:03:53 m2 kernel:   aa:16051 ac:199240 id:45984 il:9751 ic:4059 fr:15366
Sep 19 12:03:53 m2 kernel: 2*4kB 1*8kB 1*16kB 3*32kB 4*64kB 1*128kB 1*256kB
1*512kB 0*1024kB 1*2048kB 2*4096kB = 11520kB)
Sep 19 12:03:53 m2 kernel: 29360*4kB 12959*8kB 2288*16kB 136*32kB 2*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 262200kB)
Sep 19 12:03:53 m2 kernel: 10540*4kB 2017*8kB 136*16kB 1*32kB 1*64kB 1*128kB
1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 61464kB)
Sep 19 12:03:53 m2 kernel: Swap cache: add 4362, delete 4275, find
1067976/1068072, race 0+0
Sep 19 12:03:53 m2 kernel: 60251 pages of slabcache
Sep 19 12:03:53 m2 kernel: 392 pages of kernel stacks
Sep 19 12:03:53 m2 kernel: 0 lowmem pagetables, 2657 highmem pagetables
Sep 19 12:03:53 m2 kernel: 40 bounce buffer pages, 32 are on the emergency list
Sep 19 12:03:53 m2 kernel: Free swap:       2047840kB
Sep 19 12:03:53 m2 kernel: 524160 pages of RAM
Sep 19 12:03:53 m2 kernel: 294768 pages of HIGHMEM
Sep 19 12:03:53 m2 kernel: 10428 reserved pages
Sep 19 12:03:53 m2 kernel: 190028 pages shared
Sep 19 12:03:53 m2 kernel: 87 pages swap cached
Comment 11 Dave Anderson 2005-09-19 14:18:01 EDT
I don't know anything about auditd internals, but the message:

Sep 19 09:31:07 m2 auditd[1550]: output error; suspending execution

looks fairly serious.  Is the filesystem containing /var/log/audit.d
full when this occurs?  

There's no low-memory contention indicated by the alt-sysrq-m.

Does CPU1 consistently show the same posix_lock_file trace if you do
several alt-sysrq-w's?  If so, what's running on CPU1?
Comment 12 Michael T. Rampson 2005-09-19 15:04:49 EDT
/var/log/audit.d is not full, but the audit.conf looks like it tells to audit to
shut down when you get to a certain percentage on the disk space.  The CPUs are
constanty spinning with the load average up to between 60 and 80 all the time. 
I even tried to cleanup disk space and restart the audit.  This had some affect
as the load average came down to around 20, but I still couldn't ssh in,
although most of the processes that I couldn't kill previously seemed to go away.
Comment 13 Dave Anderson 2005-09-19 15:13:45 EDT
Well, the alt-sysrq-w output showed at least 2 cpus idling,
and you didn't respond as to whether CPU1 is stuck in the
spinning in the same place.

We'd need to know what processes are in the unkillable state,
and what their traces are.  Maybe an alt-sysrq-t will show
something debuggable for the unkillable processes.
Comment 14 Michael T. Rampson 2005-09-19 15:31:57 EDT
Here is a 't' trace from that machine during that time.  Only 65 K will fit though:

Sep 19 12:07:34 m2 kernel: SysRq : Show State
Sep 19 12:07:34 m2 kernel: 
Sep 19 12:07:34 m2 kernel:                          free                       
sibling
Sep 19 12:07:34 m2 kernel:   task             PC    stack   pid father child
younger older
Sep 19 12:07:34 m2 kernel: init          S 00000000  2604     1      0     6   
   2       (NOTLB)
Sep 19 12:07:34 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc4cd9ea0)
Sep 19 12:07:34 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc4cd9ee4)
Sep 19 12:07:34 m2 kernel: [<c015962c>] __get_free_pages [kernel] 0x1c (0xc4cd9eec)
Sep 19 12:07:34 m2 kernel: [<c01798a1>] __pollwait [kernel] 0x31 (0xc4cd9ef0)
Sep 19 12:07:34 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc4cd9f04)
Sep 19 12:07:34 m2 kernel: [<c0179b6b>] do_select [kernel] 0x13b (0xc4cd9f1c)
Sep 19 12:07:34 m2 kernel: [<c017a00e>] sys_select [kernel] 0x34e (0xc4cd9f60)
Sep 19 12:07:34 m2 kernel: 
Sep 19 12:07:34 m2 kernel: ef261f80)
Sep 19 12:07:34 m2 kernel: 
Sep 19 12:07:34 m2 kernel: ipop3d        D 00000000   256 20304   1652        
20319       (NOTLB)
Sep 19 12:07:34 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xcb6d7e9c)
Sep 19 12:07:34 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xcb6d7ee0)
Sep 19 12:07:34 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xcb6d7ee8)
Sep 19 12:07:34 m2 kernel: [<f8ab7784>] audit_writers [audit] 0x0 (0xcb6d7ef4)
Sep 19 12:07:34 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xcb6d7f10)
Sep 19 12:07:34 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcb6d7f64)
Sep 19 12:07:34 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xcb6d7f74)
Sep 19 12:07:34 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xcb6d7f94)
Sep 19 12:07:34 m2 kernel: 
Sep 19 12:07:34 m2 kernel: ipop3d        D 00000000     0 20319   1652        
20326 20304 (NOTLB)
Sep 19 12:07:34 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf44c7e9c)
Sep 19 12:07:34 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xf44c7ee0)
Sep 19 12:07:34 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xf44c7ee8)
Sep 19 12:07:34 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xf44c7f10)
Sep 19 12:07:34 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf44c7f64)
Sep 19 12:07:34 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xf44c7f74)
Sep 19 12:07:34 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf44c7f94)
Sep 19 12:07:34 m2 kernel: 
Sep 19 12:07:34 m2 kernel: ipop3d        D 00000002  3592 20326   1652        
20428 20319 (NOTLB)
Sep 19 12:07:34 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf2cd1e9c)
Sep 19 12:07:34 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xf2cd1ee0)
Sep 19 12:07:34 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xf2cd1ee8)
Sep 19 12:07:34 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xf2cd1f10)
Sep 19 12:07:34 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf2cd1f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xf2cd1f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf2cd1f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 20428   1652        
20444 20326 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd2f95e9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xd2f95ee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xd2f95ee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xd2f95f10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd2f95f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xd2f95f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd2f95f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     0 20444   1652        
20453 20428 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe56d3e9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xe56d3ee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xe56d3ee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xe56d3f10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe56d3f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xe56d3f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe56d3f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20453   1652        
20507 20444 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc461be9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xc461bee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xc461bee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xc461bf10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc461bf64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xc461bf74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc461bf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20507   1652        
20509 20453 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc8ec3e9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xc8ec3ee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xc8ec3ee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xc8ec3f10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc8ec3f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xc8ec3f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc8ec3f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 20509   1652        
20564 20507 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd721de9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xd721dee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xd721dee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xd721df10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd721df64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xd721df74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd721df94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20564   1652        
20573 20509 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf31dde9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xf31ddee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xf31ddee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xf31ddf10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf31ddf64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xf31ddf74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf31ddf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 20573   1652        
20604 20564 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xca00be9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xca00bee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xca00bee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xca00bf10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xca00bf64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xca00bf74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xca00bf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20604   1652        
20648 20573 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xdc377e9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xdc377ee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xdc377ee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xdc377f10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdc377f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xdc377f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xdc377f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     0 20648   1652        
20694 20604 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc8be1e9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xc8be1ee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xc8be1ee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xc8be1f10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc8be1f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xc8be1f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc8be1f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20694   1652        
20698 20648 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xed5dbe9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xed5dbee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xed5dbee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xed5dbf10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed5dbf64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xed5dbf74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xed5dbf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 20698   1652        
20781 20694 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xedf2fe9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xedf2fee0)
Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xedf2fee8)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xedf2ff10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xedf2ff64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xedf2ff74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xedf2ff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: crond         S 00000002     0 20702   1797 20703   
           (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xdd4afed0)
Sep 19 12:07:35 m2 kernel: [<c0171790>] pipe_wait [kernel] 0x70 (0xdd4aff14)
Sep 19 12:07:35 m2 kernel: [<c0171884>] pipe_read [kernel] 0xc4 (0xdd4aff68)
Sep 19 12:07:35 m2 kernel: [<c0164767>] sys_read [kernel] 0x97 (0xdd4aff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: crond         D 00000002     0 20703  20702         
           (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe16b5e9c)
Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xe16b5ee0)
Sep 19 12:07:35 m2 kernel: [<f8ab7784>] audit_writers [audit] 0x0 (0xe16b5ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xe16b5f10)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe16b5f64)
Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xe16b5f74)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe16b5f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001   288 20781   1652        
20823 20698 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xec37bef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xec37bf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xec37bf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xec37bf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xec37bf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20823   1652        
20850 20781 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd219fef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd219ff34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd219ff48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd219ff6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd219ff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 20850   1652        
20898 20823 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xef637ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xef637f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xef637f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xef637f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xef637f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001  3760 20898   1652        
20934 20850 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe8c75ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe8c75f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe8c75f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe8c75f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe8c75f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 20934   1652        
20960 20898 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd8987ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd8987f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd8987f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd8987f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd8987f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001    20 20960   1652        
21010 20934 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xce9d3ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xce9d3f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xce9d3f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xce9d3f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xce9d3f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 21010   1652        
21061 20960 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf3b19ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf3b19f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf3b19f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf3b19f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf3b19f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003   612 21061   1652        
21075 21010 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd6a11ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd6a11f34)
Sep 19 12:07:35 m2 kernel: [<c0134bf3>] update_process_time_intertick [kernel]
0x53 (0xd6a11f38)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd6a11f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd6a11f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd6a11f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002    76 21075   1652        
21090 21061 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xece33ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xece33f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xece33f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xece33f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xece33f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21090   1652        
21115 21075 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc712fef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc712ff34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc712ff48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc712ff6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc712ff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 21115   1652        
21188 21090 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf7eedef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf7eedf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf7eedf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf7eedf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf7eedf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21188   1652        
21203 21115 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc747def0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc747df34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc747df48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc747df6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc747df94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21203   1652        
21219 21188 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe9311ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe9311f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe9311f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe9311f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe9311f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     0 21219   1652        
21238 21203 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd87fdef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd87fdf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd87fdf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd87fdf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd87fdf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     0 21238   1652        
21336 21219 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xcd731ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xcd731f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xcd731f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xcd731f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xcd731f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     4 21336   1652        
21354 21238 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd001def0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd001df34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd001df48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd001df6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd001df94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21354   1652        
21382 21336 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xdd41fef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xdd41ff34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xdd41ff48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xdd41ff6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xdd41ff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003  3884 21382   1652        
21413 21354 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd923bef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd923bf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd923bf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd923bf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd923bf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21413   1652        
21424 21382 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc390fef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc390ff34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc390ff48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc390ff6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc390ff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21424   1652        
21438 21413 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe198def0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe198df34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe198df48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe198df6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe198df94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21438   1652        
21450 21424 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf106bef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf106bf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf106bf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf106bf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf106bf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001   564 21450   1652        
21470 21438 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc5dabef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc5dabf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc5dabf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc5dabf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc5dabf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 21470   1652        
21539 21450 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf3823ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf3823f34)
Sep 19 12:07:35 m2 kernel: [<c0134bf3>] update_process_time_intertick [kernel]
0x53 (0xf3823f38)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf3823f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf3823f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf3823f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     4 21539   1652        
21558 21470 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf46edef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf46edf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf46edf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf46edf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf46edf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000   596 21558   1652        
21560 21539 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc56abef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc56abf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc56abf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc56abf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc56abf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 21560   1652        
21603 21558 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xcac99ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xcac99f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xcac99f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xcac99f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xcac99f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000  4228 21603   1652        
21604 21560 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xd6367ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xd6367f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xd6367f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd6367f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd6367f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     0 21604   1652        
21641 21603 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc9899ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc9899f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc9899f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc9899f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc9899f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001  1792 21641   1652        
21644 21604 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xed795ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xed795f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xed795f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xed795f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xed795f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 21644   1652        
21658 21641 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe4429ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe4429f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe4429f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe4429f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe4429f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002  2864 21658   1652        
21669 21644 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf3929ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf3929f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf3929f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf3929f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf3929f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21669   1652        
21678 21658 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xdaebfef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xdaebff34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xdaebff48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xdaebff6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xdaebff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000003     0 21678   1652        
21765 21669 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xee6e5ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xee6e5f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xee6e5f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xee6e5f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xee6e5f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002   424 21765   1652        
21777 21678 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe4631ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe4631f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe4631f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe4631f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe4631f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     8 21777   1652        
21816 21765 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe0283ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe0283f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe0283f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe0283f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe0283f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21816   1652        
21852 21777 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xea481ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xea481f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xea481f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xea481f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xea481f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21852   1652        
21858 21816 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe2abfef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe2abff34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe2abff48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe2abff6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe2abff94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 21858   1652        
21868 21852 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe5ce7ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe5ce7f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe5ce7f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe5ce7f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe5ce7f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21868   1652        
21869 21858 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc5165ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xc5165f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xc5165f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc5165f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc5165f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 21869   1652        
21873 21868 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf6821ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf6821f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf6821f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf6821f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf6821f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 21873   1652        
21931 21869 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf68b1ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf68b1f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf68b1f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf68b1f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf68b1f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000002     0 21931   1652        
21950 21873 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe77f7ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xe77f7f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xe77f7f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe77f7f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe77f7f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 21950   1652        
21970 21931 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf5085ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf5085f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf5085f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf5085f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf5085f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 21970   1652        
22006 21950 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf73d5ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf73d5f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf73d5f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf73d5f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf73d5f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000001     0 22006   1652        
22009 21970 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xea2fbef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xea2fbf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xea2fbf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xea2fbf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xea2fbf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: ipop3d        D 00000000     0 22009   1652         
     22006 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xee0cbef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xee0cbf34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xee0cbf48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xee0cbf6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xee0cbf94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: vsftpd        D 00000000    24 22675   1698 22676   
           (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf42b5ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xf42b5f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xf42b5f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf42b5f6c)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf42b5f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: vsftpd        Z 00000002     0 22676  22675         
           (L-TLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xe83bdf4c)
Sep 19 12:07:35 m2 kernel: [<c012d611>] exit_notify [kernel] 0x121 (0xe83bdf70)
Sep 19 12:07:35 m2 kernel: [<c012db10>] do_exit [kernel] 0x370 (0xe83bdf90)
Sep 19 12:07:35 m2 kernel: [<c012dcbb>] do_group_exit [kernel] 0x8b (0xe83bdfac)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: httpd         S 00000003     0  2651   1759         
3194       (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc9905ec0)
Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc9905f04)
Sep 19 12:07:35 m2 kernel: [<c02253cc>] sock_poll [kernel] 0x2c (0xc9905f0c)
Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc9905f24)
Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xc9905f3c)
Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xc9905f70)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: sshd          D 00000001     0  2953   1636  2954   
           (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xed315ef0)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f24)
Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c
(0xed315f34)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f3c)
Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29
(0xed315f48)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f4c)
Sep 19 12:07:35 m2 kernel: [<f8ab76ac>] audit_lock [audit] 0x8 (0xed315f58)
Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f68)
Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xed315f6c)
Sep 19 12:07:35 m2 kernel: [<c01659fb>] __fput [kernel] 0xbb (0xed315f78)
Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xed315f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: sshd          S 00000003     0  2954   2953         
           (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf6929df0)
Sep 19 12:07:35 m2 kernel: [<c013527c>] schedule_timeout [kernel] 0xbc (0xf6929e34)
Sep 19 12:07:35 m2 kernel: [<c029345f>] unix_stream_data_wait [kernel] 0xcf
(0xf6929e6c)
Sep 19 12:07:35 m2 kernel: [<c0228bf4>] alloc_skb [kernel] 0xc4 (0xf6929e88)
Sep 19 12:07:35 m2 kernel: [<c02938a1>] unix_stream_recvmsg [kernel] 0x3f1
(0xf6929ec4)
Sep 19 12:07:35 m2 kernel: [<c0224e68>] sock_recvmsg [kernel] 0x58 (0xf6929f08)
Sep 19 12:07:35 m2 kernel: [<c0224f96>] sock_read [kernel] 0x96 (0xf6929f50)
Sep 19 12:07:35 m2 kernel: [<c0164767>] sys_read [kernel] 0x97 (0xf6929f94)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: httpd         S 00000000  1792  3194   1759  4037   
3220  2651 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0159354>] __alloc_pages [kernel]
0xb4 (0xc8ddfeac)
Sep 19 12:07:35 m2 kernel: [<c0124194>] schedule [kernel] 0x2f4 (0xc8ddfec0)
Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc8ddff04)
Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc8ddff24)
Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xc8ddff3c)
Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xc8ddff70)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: httpd         S 00000000     8  3220   1759         
3239  3194 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xf5159d3c)
Sep 19 12:07:35 m2 kernel: [<c024f13f>] ip_finish_output2 [kernel] 0xcf (0xf5159d4c)
Sep 19 12:07:35 m2 kernel: [<c01ae036>] sys_semtimedop [kernel] 0x626 (0xf5159d80)
Sep 19 12:07:35 m2 kernel: [<c024d2a9>] ip_queue_xmit [kernel] 0x359 (0xf5159e08)
Sep 19 12:07:35 m2 kernel: [<c024f040>] dst_output [kernel] 0x0 (0xf5159e20)
Sep 19 12:07:35 m2 kernel: [<f88a41df>] ext3_do_update_inode [ext3] 0x16f
(0xf5159e40)
Sep 19 12:07:35 m2 kernel: [<c02561c6>] tcp_recvmsg [kernel] 0x456 (0xf5159e90)
Sep 19 12:07:35 m2 kernel: [<c02630fa>] tcp_clear_xmit_timers [kernel] 0x5a
(0xf5159ea4)
Sep 19 12:07:35 m2 kernel: [<c0267f3b>] tcp_v4_destroy_sock [kernel] 0x1b
(0xf5159eb8)
Sep 19 12:07:35 m2 kernel: [<c02569d8>] tcp_destroy_sock [kernel] 0xd8 (0xf5159ecc)
Sep 19 12:07:35 m2 kernel: [<c0256ee9>] tcp_close [kernel] 0x399 (0xf5159ee0)
Sep 19 12:07:35 m2 kernel: [<c0224e68>] sock_recvmsg [kernel] 0x58 (0xf5159f08)
Sep 19 12:07:35 m2 kernel: [<c017df70>] dput [kernel] 0x30 (0xf5159f64)
Sep 19 12:07:35 m2 kernel: [<c01659fb>] __fput [kernel] 0xbb (0xf5159f78)
Sep 19 12:07:35 m2 kernel: [<c01138e9>] sys_ipc [kernel] 0x59 (0xf5159f90)
Sep 19 12:07:35 m2 kernel: [<c0163c46>] sys_close [kernel] 0x66 (0xf5159fb0)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: httpd         S 00000003   836  3239   1759         
3241  3220 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xc4b69ec0)
Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc4b69f04)
Sep 19 12:07:35 m2 kernel: [<c02253cc>] sock_poll [kernel] 0x2c (0xc4b69f0c)
Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc4b69f24)
Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xc4b69f3c)
Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xc4b69f70)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: httpd         S 00000001  1792  3241   1759         
3246  3239 (NOTLB)
Sep 19 12:07:35 m2 kernel: Call Trace:   [<c0124194>] schedule [kernel] 0x2f4
(0xce397ec0)
Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xce397f04)
Sep 19 12:07:35 m2 kernel: [<c02253cc>] sock_poll [kernel] 0x2c (0xce397f0c)
Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xce397f24)
Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xce397f3c)
Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xce397f70)
Sep 19 12:07:35 m2 kernel: [<c014a676>] sys_sendfile [kernel] 0x56 (0xce397f9c)
Sep 19 12:07:35 m2 kernel: 
Sep 19 12:07:35 m2 kernel: httpd         S 00000003     0  3246   1759         
3249  3241 (NOTLB)

Comment 15 Michael T. Rampson 2005-09-19 15:35:06 EDT
see the earlier postings for what isn't killable.  The above was about half of
the dump.
Comment 16 Dave Anderson 2005-09-19 15:53:40 EDT
 
There certainly are a number of tasks that are failing to acquire
the audit_lock semaphore.  I don't see any evidence of who currently
holds it, nor any evidence of what would make the cpus run at 100%.

If you run top, are any of the processes that are seen above as
sleeping on the audit_lock semaphore the ones that are taking up
all of the cpu time?

Comment 17 Michael T. Rampson 2005-09-19 16:06:39 EDT
I haven't been able to run top as it does screen writes and I can't get in to
any of the tty's, only webmin.  So it looks like when audit shuts down, then
these processes try to write but they can't as it has suspended itself.  Things
like ipop, vsftpd, sshd seem to accumulate.  You can't kill them with anything.
 If you clean up disk space and restart audit, then these processes go away. 
Seems to be some issue with audit suspending itself and then processes wait on
trying to log access and get hung up until audit comes back.
Comment 18 Dave Anderson 2005-09-19 16:23:43 EDT
Can you get us a netdump or diskdump?

Comment 19 Michael T. Rampson 2005-09-19 18:10:38 EDT
If you tell me how to configure it to send it to you, I'll set it up the next
time it happens on one of these boxes.
Comment 20 Dave Anderson 2005-09-20 09:16:19 EDT
Install the netdump package on the client (where the hangs occur).
Install the netdump-server package on a machine with enough space 
to capture a full memory dump from the client.  There are README
files that come with the packages for the client and server here:

 /usr/share/doc/netdump-x.x.x/README.client
 /usr/share/doc/netdump-server-x.x.x/README

Further info in netdump(8) and netdump-server(8) man pages.

It will need to be set up prior to the hang occurrance, i.e., you cannot
set it up the next time it happens.  It's a kernel module that is loaded
during boot.  

Anyway, the setup is essentially done like this:

Server side:

After installing the netdump-server package on the server, set up a password
for the netdump user.  To start the service one time, enter:

  $ service netdump-server start

To make it available with every boot:

  $ chkconfig --add netdump-server
  $ chkconfig netdump-server on

To ensure that it's running:

  $ service netdump-server status


Client side:

After installing the netdump package on the client, modify the NETDUMPADDR
field in /etc/sysconfig/netdump to the IP address of the netdump-server.
Then enter:

  $ service netdump propagate

one time.  It will ask for the netdump password that you set up on the server.  
Then for per-boot-session netdump-capturing, enter:

  $ service netdump start

For netdump-capturing all the time, enter: 

  $ chkconfig --add netdump
  $ chkconfig netdump on

To ensure that it's running:

  $ service netdump status

After setting things up both the server and client, it's worth testing it
first with an alt-sysrq-c on the client.
 
Comment 21 Michael T. Rampson 2005-09-27 12:13:45 EDT
as a further update, neither machine showed a problem this weekend as I made
sure there was plenty of disk space on the /var/log partitions to keep audit
from turning off.  I would imagine that you could recreate this on an SMP box
and filling up /var/log till audit turns off, then just keep trying to make
connections via ssh, vsftpd, and/or ipop3.
Comment 22 Dave Anderson 2005-09-27 13:43:14 EDT
Right -- which is also a pretty valid work-around...

It appears to have to do with certain network-related processes
that, in RHEL3, have been made "audit-aware" such that they directly
perform ioctls on the the /dev/audit device, and those seem to be
running into a locking issue when the audit daemon suspends itself.
Comment 23 Olle Liljenzin 2005-12-15 05:27:49 EST
I can just confirm that we have similar problems on a number of RHEL3 machines
that are running auditd.

I know it is a separate issue, but is auditd ever cleaning up old log files with
the default configuration? Or is it just a matter of time before /var/log gets
full and the machine becomes unstable?
Comment 24 Dave Anderson 2005-12-15 08:48:09 EST
I know very little about how auditd cleans up old log files, but the issue
is related to the amount of free space left on their /var partition.  I note
this suggestion in another thread:

> Check to see if they have <=20% left on their /var/ partition. 
> If they do they need to
>
> 1) chkconfig auditd off
> 2) clear out some of the space on /var
> 3) make audit.d less touchy by editing the /etc/audit/auditd.conf file and
>    changing
>
> notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%";
>
> to something like
>
> notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 5%";
Comment 25 Michael T. Rampson 2005-12-15 10:43:45 EST
The only way around this problem is to either not run auditd or make sure there
is always enough space on the device that the log files are on, which can be
problematic, but hopefully, on most machines for most operations, it is not an
issue.  This is still a bug as far as I'm concerned as these processes shouldn't
get hung up (at least not in this way where it is unkillable) because auditd has
turned itself off.
Comment 26 Greg Bailey 2006-08-18 11:34:19 EDT
I also just hit this situation.  Now I understand the audit system a lot better
and understand why it went into suspend mode, but I had to hard reset my server
to get out of this because the shutdown scripts also hung.  Is there anyway to
shutdown cleanly when this occurs?
Comment 27 RHEL Product and Program Management 2007-10-19 14:54:26 EDT
This bug is filed against RHEL 3, which is in maintenance phase.
During the maintenance phase, only security errata and select mission
critical bug fixes will be released for enterprise products. Since
this bug does not meet that criteria, it is now being closed.
 
For more information of the RHEL errata support policy, please visit:
http://www.redhat.com/security/updates/errata/
 
If you feel this bug is indeed mission critical, please contact your
support representative. You may be asked to provide detailed
information on how this bug is affecting you.

Note You need to log in before you can comment on or make changes to this bug.