Red Hat Bugzilla – Bug 445755
audispd spews syslog with "queue is full - dropping event" - 21000 times in 12 seconds
Last modified: 2010-08-26 16:00:35 EDT
Description of problem: audispd prints lots and lots of these messages: ,----[ /var/log/messages ]---- | May 8 22:32:39 nan audispd: queue is full - dropping event | May 8 22:32:39 nan audispd: queue is full - dropping event | May 8 22:32:39 nan audispd: queue is full - dropping event `---- In this specific case, I count more than 20500 such lines in less than 13 seconds. Version-Release number of selected component (if applicable): audit-1.7.2-6.fc9.i386 kernel-PAE-2.6.25-14.fc9.i686 How reproducible: Sporadically, specific cause unknown. Steps to Reproduce: ??? Actual results: 1730 messages per second. Expected results: Something on the order of ONE message per second, or less. Additional info: Even if the sporadic occurrent of this problem makes it difficult to find the underlying problem, rate-limiting this overflow message would still make sense.
This is a problem I've been looking at for a few weeks. I added separate priority boost controls between audispd and auditd in audit-1.7.3-1. It can be found here: http://koji.fedoraproject.org/koji/buildinfo?buildID=48645 if you wanted to do any early testing. The basic problem is that depending on your audit rules you may need to increase the priority of audispd so that its gets more run time and increase the internal queue of audispd.
Changing version to '9' as part of upcoming Fedora 9 GA. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
I have just found that my logs have the "audispd: queue is full - dropping event" flood with whatever audit and kernel version was current on 2008-05-25.
We are still looking at this. Something changed wrt the scheduler and we have not pinpointed the best way forward. It may be a combination of changing the qos to be lossless, increasing the priority boost, and lengthening the audispd internal queue.
Does the current audit package (1.7.5) solve the problems? It interact with the scheduler differently than earlier versions.
I cannot see the "audispd: queue is full - dropping event" message in my /var/log after audit was updated to audit-1.7.5-1.fc9.i386 on 2008-09-17. However, that does not necessarily mean much: I have the "queue is full" message in the logs a few hours before audit-1.7.5-1.fc9.i386 was installed, but there is a period of a few weeks before that during which there were no "queue is full" messages even with pre-1.7.5 audit. I'll keep my eyes open. If I do not see the "queue is full" during the next week or such and you say this is supposed be fixed, I'd say we close the bug, and I'll just reopen it in case the "queue is full" spewing happens again.
Thanks for the feedback. Closing bug.
For what it's worth, I'm seeing this in Fedora 12, with audit-2.0.4-1.fc12.i686 This system is pretty much just running as my asterisk server.
When you get this: 1) boost the priority of audispd and possibly auditd 2) See if you are running afoul of SE Linux AVCs.
I tried boosting it, but then I took a look at the system as a whole and noticed I had a spinning emacs and setroubleshootd process taking my system up load a load of 4.5. I killed both of those off and then the system came back down to a reasonable state and the log messages disappeared. The spinning emacs was due to a network failure that caused my session to break mid-stream. Sorry for the noise.
I'm on a just-upgraded-to-F13 system and am seeing this, too. Over 100 AVCs per minute. I've filed a couple of BZs with the bigger culprits: https://bugzilla.redhat.com/show_bug.cgi?id=571880 https://bugzilla.redhat.com/show_bug.cgi?id=571939
I experienced this on Fedora 13. Over 120000 lines were written to messages in conjunction with an AVC denial of spamassassin. Fixed it by enabling boolean spamassassin_can_network.