From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.10) Gecko/20050716 Firefox/1.0.6 Description of problem: machine gets into a state where ipop3d, sshd, CROND become unkillable and a bunch of them have been spawned and cpu seems to go to 100%. You have to reboot the machine to clear everything up. Seems to happen once a week. Version-Release number of selected component (if applicable): kernel-smp-2.4.21-32.0.1.EL How reproducible: Always Steps to Reproduce: 1. In this case, just let the machine run and usually in about a week, it gets into this state. You can't login via ssh, but httpd still alive and serving. Can't get mail, etc 2. 3. Additional info:
Hello, Michael. Thanks for your bug report. Please report this to Customer Support to see if they can help you determine the what's leading to this problem. At the very least, we'll need something more tangible to go on to debug this problem. My RHEL3 machine running that release does not incur the same problem, so we'd need a more reliable way to reproduce it.
This may be related to an sshd issue. In each case, someone was running a program to try to login via ssh. Is it possible that this causes the kernel to run out of some resource that causes ipop3, crond, and vsftpd to wedge? current version of openssh is openssh-3.6.1p2-33.30.4.
Also, this has happened on a 2nd server also running RHEL 3.0, completely different HW. In both cases, they are running SMP.
It's impossible to speculate without evidence. A netdump/diskdump would be best, which you could generate with an alt-sysrq-c on the console when the system gets into that state (presuming it still can receive keyboard interrupts). You might try this sequence of keyboard entries on the system console when it gets into that state: alt-sysrq-w alt-sysrq-m alt-sysrq-t and capture the output using a serial console. The "w" will show a trace of the active task on each cpu, the "m" will show memory statistics, and the "t" will trace all tasks on the system.
Is there a way of generating this from the command-line? I can't get into the box when this happens. I'm able to get in and execute commands via webmin as it is an already running process, but trying to get in any other way seems problematic. You can login in, but the system never comes back with a prompt. I can issue commands via webmin however, although not all commands work or affect anything (for instance restart /etc/init.d/sshd does restart ssh, but the old ssh's don't go away and you can't get in. /etc/init.d/crond fails.)
You can: echo c > /proc/sysrq-trigger or w, m or t. BTW, this all presumes that /proc/sys/kernel/sysrq is equal to 1. If not, you can do either of these for per-session enablement: echo 1 > /proc/sys/kernel/sysrq sysctl -w kernel.sysrq=1 or put this in your /etc/sysctl.conf to have it on permanently: kernel.sysrq = 1
after doing echo 1> /proc/sys/kernel/sysrq and then doing echo c > /proc/sysrq-trigger caused the machine to reboot. This after trying to run the reboot command does nothing. Also, is there a way to direct this output to a file in the filesystem instead of serial port? Also, someone tried to run a hack ipop3 which xinetd limited when it ran out of instances, but the ipop3 continued to hang around.
Netdump and/or diskdump would have to be set up in order for alt-sysrq-c to do anything other than crash/reboot the machine.
Here is an echo w from a machine that exhibits this: Sep 19 09:31:07 m2 audbin[20300]: saving binary audit log /var/log/audit.d/bin.1 Sep 19 09:31:07 m2 audbin[20300]: threshold 20.00 exceeded for filesystem /var/log/audit.d/. - free blocks down to 18.74% Sep 19 09:31:07 m2 auditd[1550]: Notify command /usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% exited with status 1 Sep 19 09:31:07 m2 auditd[1550]: output error Sep 19 09:31:07 m2 auditd[1550]: output error Sep 19 09:31:07 m2 auditd[1550]: output error; suspending execution Sep 19 11:59:39 m2 kernel: SysRq : Show CPUs Sep 19 11:59:39 m2 kernel: CPU3: Sep 19 11:59:39 m2 kernel: edb45f1c 00000003 c01cfeaf 00000000 c03b5df4 00000077 00000006 c01d013a Sep 19 11:59:39 m2 kernel: 00000077 00000000 00000000 00000000 00000000 00000000 00000000 c01d009d Sep 19 11:59:39 m2 kernel: 00000077 00000000 00000000 00000000 00000002 f7066a00 ffffffea 00000002 Sep 19 11:59:39 m2 kernel: Call Trace: [<c01cfeaf>] sysrq_handle_showcpus [kernel] 0xf (0xedb45f24) Sep 19 11:59:39 m2 kernel: [<c01d013a>] __handle_sysrq_nolock [kernel] 0x7a (0xedb45f38) Sep 19 11:59:39 m2 kernel: [<c01d009d>] handle_sysrq [kernel] 0x5d (0xedb45f58) Sep 19 11:59:39 m2 kernel: [<c01994a3>] write_sysrq_trigger [kernel] 0x53 (0xedb45f7c) Sep 19 11:59:39 m2 kernel: [<c01648a7>] sys_write [kernel] 0x97 (0xedb45f94) Sep 19 11:59:39 m2 kernel: Sep 19 11:59:39 m2 kernel: CPU2: Sep 19 11:59:39 m2 kernel: c4eddf64 00000002 c011ca7f 00000000 00001f7c c03f6caa c0109100 00000000 Sep 19 11:59:39 m2 kernel: c4edc000 c4edc000 c4edc000 c0109100 00000000 00000068 00000068 fffffffb Sep 19 11:59:39 m2 kernel: c0109129 00000060 00000246 c01091c2 0602080b 00000000 00000000 00000000 Sep 19 11:59:39 m2 kernel: Call Trace: [<c011ca7f>] smp_call_function_interrupt [kernel] 0x2f (0xc4eddf6c) Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc4eddf7c) Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc4eddf90) Sep 19 11:59:39 m2 kernel: [<c0109129>] default_idle [kernel] 0x29 (0xc4eddfa4) Sep 19 11:59:39 m2 kernel: [<c01091c2>] cpu_idle [kernel] 0x42 (0xc4eddfb0) Sep 19 11:59:39 m2 kernel: [<c0129883>] printk [kernel] 0x153 (0xc4eddfcc) Sep 19 11:59:39 m2 kernel: Sep 19 11:59:39 m2 kernel: CPU0: Sep 19 11:59:39 m2 kernel: c03f5f88 00000000 c011ca7f 00000000 00001fa0 c03f6caa c0109100 00000000 Sep 19 11:59:39 m2 kernel: c03f4000 c03f4000 c03f4000 c0109100 00000000 00000068 00000068 fffffffb Sep 19 11:59:39 m2 kernel: c0109129 00000060 00000246 c01091c2 0002080b 00099800 c0107000 0008e000 Sep 19 11:59:39 m2 kernel: Call Trace: [<c011ca7f>] smp_call_function_interrupt [kernel] 0x2f (0xc03f5f90) Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc03f5fa0) Sep 19 11:59:39 m2 kernel: [<c0109100>] default_idle [kernel] 0x0 (0xc03f5fb4) Sep 19 11:59:39 m2 kernel: [<c0109129>] default_idle [kernel] 0x29 (0xc03f5fc8) Sep 19 11:59:39 m2 kernel: [<c01091c2>] cpu_idle [kernel] 0x42 (0xc03f5fd4) Sep 19 11:59:39 m2 kernel: [<c0107000>] stext [kernel] 0x0 (0xc03f5fe0) Sep 19 11:59:39 m2 kernel: Sep 19 11:59:39 m2 kernel: CPU1: Sep 19 11:59:39 m2 kernel: e9bb5ea0 00000001 c011ca7f 00000000 00001eb8 c03f6caa 00000000 e2c81d3c Sep 19 11:59:39 m2 kernel: e9bb4000 00000000 00000000 00000000 ffffff00 ffff0068 00000068 fffffffb Sep 19 11:59:39 m2 kernel: c017bf2d 00000060 00000286 e2c81d3c e9551520 f5af5380 00000000 00000b81 Sep 19 11:59:39 m2 kernel: Call Trace: [<c011ca7f>] smp_call_function_interrupt [kernel] 0x2f (0xe9bb5ea8) Sep 19 11:59:39 m2 kernel: [<c017bf2d>] posix_lock_file [kernel] 0x42d (0xe9bb5ee0) Sep 19 11:59:39 m2 kernel: [<c017ced4>] fcntl_setlk [kernel] 0x174 (0xe9bb5f4c) Sep 19 11:59:39 m2 kernel: [<c0177e66>] do_fcntl [kernel] 0x1c6 (0xe9bb5f7c) Sep 19 11:59:39 m2 kernel: [<c0178054>] sys_fcntl64 [kernel] 0x54 (0xe9bb5f9c) Sep 19 11:59:39 m2 kernel: Could this be an issue of audit shutting down because of disk space?
Here is an 'm' dump: Sep 19 12:03:53 m2 kernel: SysRq : Show Memory Sep 19 12:03:53 m2 kernel: Mem-info: Sep 19 12:03:53 m2 kernel: Zone:DMA freepages: 2880 min: 0 low: 0 high: 0 Sep 19 12:03:53 m2 kernel: Zone:Normal freepages: 65550 min: 1278 low: 4543 high: 6303 Sep 19 12:03:53 m2 kernel: Zone:HighMem freepages: 15366 min: 255 low: 4606 high: 6909 Sep 19 12:03:53 m2 kernel: Free pages: 83797 ( 15366 HighMem) Sep 19 12:03:53 m2 kernel: ( Active: 273633/67421, inactive_laundry: 13324, inactive_clean: 6930, free: 83797 ) Sep 19 12:03:53 m2 kernel: aa:0 ac:0 id:0 il:0 ic:0 fr:2880 Sep 19 12:03:53 m2 kernel: aa:2912 ac:55430 id:21437 il:3573 ic:2871 fr:65551 Sep 19 12:03:53 m2 kernel: aa:16051 ac:199240 id:45984 il:9751 ic:4059 fr:15366 Sep 19 12:03:53 m2 kernel: 2*4kB 1*8kB 1*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11520kB) Sep 19 12:03:53 m2 kernel: 29360*4kB 12959*8kB 2288*16kB 136*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 262200kB) Sep 19 12:03:53 m2 kernel: 10540*4kB 2017*8kB 136*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 61464kB) Sep 19 12:03:53 m2 kernel: Swap cache: add 4362, delete 4275, find 1067976/1068072, race 0+0 Sep 19 12:03:53 m2 kernel: 60251 pages of slabcache Sep 19 12:03:53 m2 kernel: 392 pages of kernel stacks Sep 19 12:03:53 m2 kernel: 0 lowmem pagetables, 2657 highmem pagetables Sep 19 12:03:53 m2 kernel: 40 bounce buffer pages, 32 are on the emergency list Sep 19 12:03:53 m2 kernel: Free swap: 2047840kB Sep 19 12:03:53 m2 kernel: 524160 pages of RAM Sep 19 12:03:53 m2 kernel: 294768 pages of HIGHMEM Sep 19 12:03:53 m2 kernel: 10428 reserved pages Sep 19 12:03:53 m2 kernel: 190028 pages shared Sep 19 12:03:53 m2 kernel: 87 pages swap cached
I don't know anything about auditd internals, but the message: Sep 19 09:31:07 m2 auditd[1550]: output error; suspending execution looks fairly serious. Is the filesystem containing /var/log/audit.d full when this occurs? There's no low-memory contention indicated by the alt-sysrq-m. Does CPU1 consistently show the same posix_lock_file trace if you do several alt-sysrq-w's? If so, what's running on CPU1?
/var/log/audit.d is not full, but the audit.conf looks like it tells to audit to shut down when you get to a certain percentage on the disk space. The CPUs are constanty spinning with the load average up to between 60 and 80 all the time. I even tried to cleanup disk space and restart the audit. This had some affect as the load average came down to around 20, but I still couldn't ssh in, although most of the processes that I couldn't kill previously seemed to go away.
Well, the alt-sysrq-w output showed at least 2 cpus idling, and you didn't respond as to whether CPU1 is stuck in the spinning in the same place. We'd need to know what processes are in the unkillable state, and what their traces are. Maybe an alt-sysrq-t will show something debuggable for the unkillable processes.
Here is a 't' trace from that machine during that time. Only 65 K will fit though: Sep 19 12:07:34 m2 kernel: SysRq : Show State Sep 19 12:07:34 m2 kernel: Sep 19 12:07:34 m2 kernel: free sibling Sep 19 12:07:34 m2 kernel: task PC stack pid father child younger older Sep 19 12:07:34 m2 kernel: init S 00000000 2604 1 0 6 2 (NOTLB) Sep 19 12:07:34 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc4cd9ea0) Sep 19 12:07:34 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc4cd9ee4) Sep 19 12:07:34 m2 kernel: [<c015962c>] __get_free_pages [kernel] 0x1c (0xc4cd9eec) Sep 19 12:07:34 m2 kernel: [<c01798a1>] __pollwait [kernel] 0x31 (0xc4cd9ef0) Sep 19 12:07:34 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc4cd9f04) Sep 19 12:07:34 m2 kernel: [<c0179b6b>] do_select [kernel] 0x13b (0xc4cd9f1c) Sep 19 12:07:34 m2 kernel: [<c017a00e>] sys_select [kernel] 0x34e (0xc4cd9f60) Sep 19 12:07:34 m2 kernel: Sep 19 12:07:34 m2 kernel: ef261f80) Sep 19 12:07:34 m2 kernel: Sep 19 12:07:34 m2 kernel: ipop3d D 00000000 256 20304 1652 20319 (NOTLB) Sep 19 12:07:34 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xcb6d7e9c) Sep 19 12:07:34 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xcb6d7ee0) Sep 19 12:07:34 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xcb6d7ee8) Sep 19 12:07:34 m2 kernel: [<f8ab7784>] audit_writers [audit] 0x0 (0xcb6d7ef4) Sep 19 12:07:34 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xcb6d7f10) Sep 19 12:07:34 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcb6d7f64) Sep 19 12:07:34 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xcb6d7f74) Sep 19 12:07:34 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xcb6d7f94) Sep 19 12:07:34 m2 kernel: Sep 19 12:07:34 m2 kernel: ipop3d D 00000000 0 20319 1652 20326 20304 (NOTLB) Sep 19 12:07:34 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf44c7e9c) Sep 19 12:07:34 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xf44c7ee0) Sep 19 12:07:34 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xf44c7ee8) Sep 19 12:07:34 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xf44c7f10) Sep 19 12:07:34 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf44c7f64) Sep 19 12:07:34 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xf44c7f74) Sep 19 12:07:34 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf44c7f94) Sep 19 12:07:34 m2 kernel: Sep 19 12:07:34 m2 kernel: ipop3d D 00000002 3592 20326 1652 20428 20319 (NOTLB) Sep 19 12:07:34 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf2cd1e9c) Sep 19 12:07:34 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xf2cd1ee0) Sep 19 12:07:34 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xf2cd1ee8) Sep 19 12:07:34 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xf2cd1f10) Sep 19 12:07:34 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf2cd1f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xf2cd1f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf2cd1f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 20428 1652 20444 20326 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd2f95e9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xd2f95ee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xd2f95ee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xd2f95f10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd2f95f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xd2f95f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd2f95f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 0 20444 1652 20453 20428 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe56d3e9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xe56d3ee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xe56d3ee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xe56d3f10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe56d3f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xe56d3f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe56d3f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20453 1652 20507 20444 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc461be9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xc461bee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xc461bee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xc461bf10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc461bf64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xc461bf74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc461bf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20507 1652 20509 20453 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc8ec3e9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xc8ec3ee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xc8ec3ee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xc8ec3f10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc8ec3f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xc8ec3f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc8ec3f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 20509 1652 20564 20507 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd721de9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xd721dee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xd721dee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xd721df10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd721df64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xd721df74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd721df94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20564 1652 20573 20509 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf31dde9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xf31ddee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xf31ddee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xf31ddf10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf31ddf64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xf31ddf74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf31ddf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 20573 1652 20604 20564 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xca00be9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xca00bee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xca00bee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xca00bf10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xca00bf64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xca00bf74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xca00bf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20604 1652 20648 20573 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xdc377e9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xdc377ee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xdc377ee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xdc377f10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdc377f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xdc377f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xdc377f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 0 20648 1652 20694 20604 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc8be1e9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xc8be1ee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xc8be1ee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xc8be1f10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc8be1f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xc8be1f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc8be1f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20694 1652 20698 20648 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xed5dbe9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xed5dbee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xed5dbee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xed5dbf10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed5dbf64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xed5dbf74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xed5dbf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 20698 1652 20781 20694 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xedf2fe9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xedf2fee0) Sep 19 12:07:35 m2 kernel: [<c01ba867>] misc_open [kernel] 0xd7 (0xedf2fee8) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xedf2ff10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xedf2ff64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xedf2ff74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xedf2ff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: crond S 00000002 0 20702 1797 20703 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xdd4afed0) Sep 19 12:07:35 m2 kernel: [<c0171790>] pipe_wait [kernel] 0x70 (0xdd4aff14) Sep 19 12:07:35 m2 kernel: [<c0171884>] pipe_read [kernel] 0xc4 (0xdd4aff68) Sep 19 12:07:35 m2 kernel: [<c0164767>] sys_read [kernel] 0x97 (0xdd4aff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: crond D 00000002 0 20703 20702 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe16b5e9c) Sep 19 12:07:35 m2 kernel: [<f8ab0b56>] audit_msg_new [audit] 0x1e6 (0xe16b5ee0) Sep 19 12:07:35 m2 kernel: [<f8ab7784>] audit_writers [audit] 0x0 (0xe16b5ef0) Sep 19 12:07:35 m2 kernel: [<f8ab05ed>] audit_user_message [audit] 0xdd (0xe16b5f10) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe16b5f64) Sep 19 12:07:35 m2 kernel: [<f8aaf410>] auditf_ioctl [audit] 0x240 (0xe16b5f74) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe16b5f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 288 20781 1652 20823 20698 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xec37bef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xec37bf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xec37bf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xec37bf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xec37bf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xec37bf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20823 1652 20850 20781 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd219fef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd219ff34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd219ff48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd219ff68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd219ff6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd219ff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 20850 1652 20898 20823 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xef637ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xef637f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xef637f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xef637f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xef637f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xef637f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 3760 20898 1652 20934 20850 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe8c75ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe8c75f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe8c75f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe8c75f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe8c75f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe8c75f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 20934 1652 20960 20898 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd8987ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd8987f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd8987f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd8987f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd8987f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd8987f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 20 20960 1652 21010 20934 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xce9d3ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xce9d3f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xce9d3f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xce9d3f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xce9d3f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xce9d3f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 21010 1652 21061 20960 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf3b19ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf3b19f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf3b19f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3b19f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf3b19f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf3b19f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 612 21061 1652 21075 21010 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd6a11ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd6a11f34) Sep 19 12:07:35 m2 kernel: [<c0134bf3>] update_process_time_intertick [kernel] 0x53 (0xd6a11f38) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd6a11f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6a11f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd6a11f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd6a11f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 76 21075 1652 21090 21061 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xece33ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xece33f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xece33f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xece33f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xece33f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xece33f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21090 1652 21115 21075 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc712fef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc712ff34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc712ff48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc712ff68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc712ff6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc712ff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 21115 1652 21188 21090 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf7eedef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf7eedf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf7eedf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf7eedf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf7eedf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf7eedf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21188 1652 21203 21115 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc747def0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc747df34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc747df48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc747df68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc747df6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc747df94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21203 1652 21219 21188 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe9311ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe9311f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe9311f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe9311f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe9311f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe9311f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 0 21219 1652 21238 21203 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd87fdef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd87fdf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd87fdf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd87fdf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd87fdf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd87fdf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 0 21238 1652 21336 21219 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xcd731ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xcd731f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xcd731f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcd731f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xcd731f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xcd731f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 4 21336 1652 21354 21238 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd001def0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd001df34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd001df48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd001df68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd001df6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd001df94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21354 1652 21382 21336 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xdd41fef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xdd41ff34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xdd41ff48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdd41ff68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xdd41ff6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xdd41ff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 3884 21382 1652 21413 21354 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd923bef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd923bf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd923bf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd923bf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd923bf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd923bf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21413 1652 21424 21382 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc390fef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc390ff34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc390ff48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc390ff68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc390ff6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc390ff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21424 1652 21438 21413 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe198def0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe198df34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe198df48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe198df68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe198df6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe198df94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21438 1652 21450 21424 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf106bef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf106bf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf106bf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf106bf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf106bf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf106bf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 564 21450 1652 21470 21438 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc5dabef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc5dabf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc5dabf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5dabf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc5dabf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc5dabf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 21470 1652 21539 21450 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf3823ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf3823f34) Sep 19 12:07:35 m2 kernel: [<c0134bf3>] update_process_time_intertick [kernel] 0x53 (0xf3823f38) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf3823f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3823f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf3823f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf3823f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 4 21539 1652 21558 21470 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf46edef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf46edf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf46edf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf46edf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf46edf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf46edf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 596 21558 1652 21560 21539 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc56abef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc56abf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc56abf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc56abf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc56abf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc56abf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 21560 1652 21603 21558 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xcac99ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xcac99f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xcac99f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xcac99f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xcac99f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xcac99f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 4228 21603 1652 21604 21560 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xd6367ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xd6367f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xd6367f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xd6367f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xd6367f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xd6367f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 0 21604 1652 21641 21603 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc9899ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc9899f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc9899f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc9899f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc9899f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc9899f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 1792 21641 1652 21644 21604 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xed795ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xed795f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xed795f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed795f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xed795f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xed795f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 21644 1652 21658 21641 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe4429ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe4429f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe4429f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4429f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe4429f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe4429f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 2864 21658 1652 21669 21644 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf3929ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf3929f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf3929f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf3929f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf3929f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf3929f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21669 1652 21678 21658 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xdaebfef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xdaebff34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xdaebff48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xdaebff68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xdaebff6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xdaebff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000003 0 21678 1652 21765 21669 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xee6e5ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xee6e5f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xee6e5f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee6e5f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xee6e5f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xee6e5f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 424 21765 1652 21777 21678 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe4631ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe4631f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe4631f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe4631f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe4631f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe4631f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 8 21777 1652 21816 21765 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe0283ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe0283f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe0283f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe0283f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe0283f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe0283f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21816 1652 21852 21777 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xea481ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xea481f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xea481f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea481f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xea481f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xea481f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21852 1652 21858 21816 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe2abfef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe2abff34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe2abff48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe2abff68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe2abff6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe2abff94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 21858 1652 21868 21852 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe5ce7ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe5ce7f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe5ce7f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe5ce7f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe5ce7f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe5ce7f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21868 1652 21869 21858 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc5165ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xc5165f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xc5165f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xc5165f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xc5165f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xc5165f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 21869 1652 21873 21868 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf6821ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf6821f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf6821f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf6821f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf6821f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf6821f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 21873 1652 21931 21869 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf68b1ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf68b1f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf68b1f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf68b1f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf68b1f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf68b1f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000002 0 21931 1652 21950 21873 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe77f7ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xe77f7f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xe77f7f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xe77f7f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xe77f7f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xe77f7f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 21950 1652 21970 21931 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf5085ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf5085f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf5085f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf5085f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf5085f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf5085f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 21970 1652 22006 21950 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf73d5ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf73d5f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf73d5f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf73d5f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf73d5f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf73d5f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000001 0 22006 1652 22009 21970 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xea2fbef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xea2fbf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xea2fbf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xea2fbf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xea2fbf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xea2fbf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: ipop3d D 00000000 0 22009 1652 22006 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xee0cbef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xee0cbf34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xee0cbf48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xee0cbf68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xee0cbf6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xee0cbf94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: vsftpd D 00000000 24 22675 1698 22676 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf42b5ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xf42b5f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xf42b5f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xf42b5f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xf42b5f6c) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xf42b5f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: vsftpd Z 00000002 0 22676 22675 (L-TLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xe83bdf4c) Sep 19 12:07:35 m2 kernel: [<c012d611>] exit_notify [kernel] 0x121 (0xe83bdf70) Sep 19 12:07:35 m2 kernel: [<c012db10>] do_exit [kernel] 0x370 (0xe83bdf90) Sep 19 12:07:35 m2 kernel: [<c012dcbb>] do_group_exit [kernel] 0x8b (0xe83bdfac) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: httpd S 00000003 0 2651 1759 3194 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc9905ec0) Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc9905f04) Sep 19 12:07:35 m2 kernel: [<c02253cc>] sock_poll [kernel] 0x2c (0xc9905f0c) Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc9905f24) Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xc9905f3c) Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xc9905f70) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: sshd D 00000001 0 2953 1636 2954 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xed315ef0) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f24) Sep 19 12:07:35 m2 kernel: [<c02a2dec>] rwsem_down_failed_common [kernel] 0x5c (0xed315f34) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f3c) Sep 19 12:07:35 m2 kernel: [<c02a2bf9>] rwsem_down_read_failed [kernel] 0x29 (0xed315f48) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f4c) Sep 19 12:07:35 m2 kernel: [<f8ab76ac>] audit_lock [audit] 0x8 (0xed315f58) Sep 19 12:07:35 m2 kernel: [<f8ab76a4>] audit_lock [audit] 0x0 (0xed315f68) Sep 19 12:07:35 m2 kernel: [<f8ab07ea>] .text.lock.control [audit] 0x6f (0xed315f6c) Sep 19 12:07:35 m2 kernel: [<c01659fb>] __fput [kernel] 0xbb (0xed315f78) Sep 19 12:07:35 m2 kernel: [<c0178756>] sys_ioctl [kernel] 0xf6 (0xed315f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: sshd S 00000003 0 2954 2953 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf6929df0) Sep 19 12:07:35 m2 kernel: [<c013527c>] schedule_timeout [kernel] 0xbc (0xf6929e34) Sep 19 12:07:35 m2 kernel: [<c029345f>] unix_stream_data_wait [kernel] 0xcf (0xf6929e6c) Sep 19 12:07:35 m2 kernel: [<c0228bf4>] alloc_skb [kernel] 0xc4 (0xf6929e88) Sep 19 12:07:35 m2 kernel: [<c02938a1>] unix_stream_recvmsg [kernel] 0x3f1 (0xf6929ec4) Sep 19 12:07:35 m2 kernel: [<c0224e68>] sock_recvmsg [kernel] 0x58 (0xf6929f08) Sep 19 12:07:35 m2 kernel: [<c0224f96>] sock_read [kernel] 0x96 (0xf6929f50) Sep 19 12:07:35 m2 kernel: [<c0164767>] sys_read [kernel] 0x97 (0xf6929f94) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: httpd S 00000000 1792 3194 1759 4037 3220 2651 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0159354>] __alloc_pages [kernel] 0xb4 (0xc8ddfeac) Sep 19 12:07:35 m2 kernel: [<c0124194>] schedule [kernel] 0x2f4 (0xc8ddfec0) Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc8ddff04) Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc8ddff24) Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xc8ddff3c) Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xc8ddff70) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: httpd S 00000000 8 3220 1759 3239 3194 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xf5159d3c) Sep 19 12:07:35 m2 kernel: [<c024f13f>] ip_finish_output2 [kernel] 0xcf (0xf5159d4c) Sep 19 12:07:35 m2 kernel: [<c01ae036>] sys_semtimedop [kernel] 0x626 (0xf5159d80) Sep 19 12:07:35 m2 kernel: [<c024d2a9>] ip_queue_xmit [kernel] 0x359 (0xf5159e08) Sep 19 12:07:35 m2 kernel: [<c024f040>] dst_output [kernel] 0x0 (0xf5159e20) Sep 19 12:07:35 m2 kernel: [<f88a41df>] ext3_do_update_inode [ext3] 0x16f (0xf5159e40) Sep 19 12:07:35 m2 kernel: [<c02561c6>] tcp_recvmsg [kernel] 0x456 (0xf5159e90) Sep 19 12:07:35 m2 kernel: [<c02630fa>] tcp_clear_xmit_timers [kernel] 0x5a (0xf5159ea4) Sep 19 12:07:35 m2 kernel: [<c0267f3b>] tcp_v4_destroy_sock [kernel] 0x1b (0xf5159eb8) Sep 19 12:07:35 m2 kernel: [<c02569d8>] tcp_destroy_sock [kernel] 0xd8 (0xf5159ecc) Sep 19 12:07:35 m2 kernel: [<c0256ee9>] tcp_close [kernel] 0x399 (0xf5159ee0) Sep 19 12:07:35 m2 kernel: [<c0224e68>] sock_recvmsg [kernel] 0x58 (0xf5159f08) Sep 19 12:07:35 m2 kernel: [<c017df70>] dput [kernel] 0x30 (0xf5159f64) Sep 19 12:07:35 m2 kernel: [<c01659fb>] __fput [kernel] 0xbb (0xf5159f78) Sep 19 12:07:35 m2 kernel: [<c01138e9>] sys_ipc [kernel] 0x59 (0xf5159f90) Sep 19 12:07:35 m2 kernel: [<c0163c46>] sys_close [kernel] 0x66 (0xf5159fb0) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: httpd S 00000003 836 3239 1759 3241 3220 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xc4b69ec0) Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xc4b69f04) Sep 19 12:07:35 m2 kernel: [<c02253cc>] sock_poll [kernel] 0x2c (0xc4b69f0c) Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xc4b69f24) Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xc4b69f3c) Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xc4b69f70) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: httpd S 00000001 1792 3241 1759 3246 3239 (NOTLB) Sep 19 12:07:35 m2 kernel: Call Trace: [<c0124194>] schedule [kernel] 0x2f4 (0xce397ec0) Sep 19 12:07:35 m2 kernel: [<c0135225>] schedule_timeout [kernel] 0x65 (0xce397f04) Sep 19 12:07:35 m2 kernel: [<c02253cc>] sock_poll [kernel] 0x2c (0xce397f0c) Sep 19 12:07:35 m2 kernel: [<c01351b0>] process_timeout [kernel] 0x0 (0xce397f24) Sep 19 12:07:35 m2 kernel: [<c017a306>] do_pollfd [kernel] 0x166 (0xce397f3c) Sep 19 12:07:35 m2 kernel: [<c017a4e3>] sys_poll [kernel] 0x183 (0xce397f70) Sep 19 12:07:35 m2 kernel: [<c014a676>] sys_sendfile [kernel] 0x56 (0xce397f9c) Sep 19 12:07:35 m2 kernel: Sep 19 12:07:35 m2 kernel: httpd S 00000003 0 3246 1759 3249 3241 (NOTLB)
see the earlier postings for what isn't killable. The above was about half of the dump.
There certainly are a number of tasks that are failing to acquire the audit_lock semaphore. I don't see any evidence of who currently holds it, nor any evidence of what would make the cpus run at 100%. If you run top, are any of the processes that are seen above as sleeping on the audit_lock semaphore the ones that are taking up all of the cpu time?
I haven't been able to run top as it does screen writes and I can't get in to any of the tty's, only webmin. So it looks like when audit shuts down, then these processes try to write but they can't as it has suspended itself. Things like ipop, vsftpd, sshd seem to accumulate. You can't kill them with anything. If you clean up disk space and restart audit, then these processes go away. Seems to be some issue with audit suspending itself and then processes wait on trying to log access and get hung up until audit comes back.
Can you get us a netdump or diskdump?
If you tell me how to configure it to send it to you, I'll set it up the next time it happens on one of these boxes.
Install the netdump package on the client (where the hangs occur). Install the netdump-server package on a machine with enough space to capture a full memory dump from the client. There are README files that come with the packages for the client and server here: /usr/share/doc/netdump-x.x.x/README.client /usr/share/doc/netdump-server-x.x.x/README Further info in netdump(8) and netdump-server(8) man pages. It will need to be set up prior to the hang occurrance, i.e., you cannot set it up the next time it happens. It's a kernel module that is loaded during boot. Anyway, the setup is essentially done like this: Server side: After installing the netdump-server package on the server, set up a password for the netdump user. To start the service one time, enter: $ service netdump-server start To make it available with every boot: $ chkconfig --add netdump-server $ chkconfig netdump-server on To ensure that it's running: $ service netdump-server status Client side: After installing the netdump package on the client, modify the NETDUMPADDR field in /etc/sysconfig/netdump to the IP address of the netdump-server. Then enter: $ service netdump propagate one time. It will ask for the netdump password that you set up on the server. Then for per-boot-session netdump-capturing, enter: $ service netdump start For netdump-capturing all the time, enter: $ chkconfig --add netdump $ chkconfig netdump on To ensure that it's running: $ service netdump status After setting things up both the server and client, it's worth testing it first with an alt-sysrq-c on the client.
as a further update, neither machine showed a problem this weekend as I made sure there was plenty of disk space on the /var/log partitions to keep audit from turning off. I would imagine that you could recreate this on an SMP box and filling up /var/log till audit turns off, then just keep trying to make connections via ssh, vsftpd, and/or ipop3.
Right -- which is also a pretty valid work-around... It appears to have to do with certain network-related processes that, in RHEL3, have been made "audit-aware" such that they directly perform ioctls on the the /dev/audit device, and those seem to be running into a locking issue when the audit daemon suspends itself.
I can just confirm that we have similar problems on a number of RHEL3 machines that are running auditd. I know it is a separate issue, but is auditd ever cleaning up old log files with the default configuration? Or is it just a matter of time before /var/log gets full and the machine becomes unstable?
I know very little about how auditd cleans up old log files, but the issue is related to the amount of free space left on their /var partition. I note this suggestion in another thread: > Check to see if they have <=20% left on their /var/ partition. > If they do they need to > > 1) chkconfig auditd off > 2) clear out some of the space on /var > 3) make audit.d less touchy by editing the /etc/audit/auditd.conf file and > changing > > notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%"; > > to something like > > notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 5%";
The only way around this problem is to either not run auditd or make sure there is always enough space on the device that the log files are on, which can be problematic, but hopefully, on most machines for most operations, it is not an issue. This is still a bug as far as I'm concerned as these processes shouldn't get hung up (at least not in this way where it is unkillable) because auditd has turned itself off.
I also just hit this situation. Now I understand the audit system a lot better and understand why it went into suspend mode, but I had to hard reset my server to get out of this because the shutdown scripts also hung. Is there anyway to shutdown cleanly when this occurs?
This bug is filed against RHEL 3, which is in maintenance phase. During the maintenance phase, only security errata and select mission critical bug fixes will be released for enterprise products. Since this bug does not meet that criteria, it is now being closed. For more information of the RHEL errata support policy, please visit: http://www.redhat.com/security/updates/errata/ If you feel this bug is indeed mission critical, please contact your support representative. You may be asked to provide detailed information on how this bug is affecting you.