Bug 445755 - audispd spews syslog with "queue is full - dropping event" - 21000 times in 12 seconds
audispd spews syslog with "queue is full - dropping event" - 21000 times in 1...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: audit (Show other bugs)
9
All Linux
low Severity low
: ---
: ---
Assigned To: Steve Grubb
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-08 16:55 EDT by Hans Ulrich Niedermann
Modified: 2010-08-26 16:00 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-09-25 08:34:47 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Hans Ulrich Niedermann 2008-05-08 16:55:37 EDT
Description of problem:

  audispd prints lots and lots of these messages:

  ,----[ /var/log/messages ]----
  | May  8 22:32:39 nan audispd: queue is full - dropping event
  | May  8 22:32:39 nan audispd: queue is full - dropping event
  | May  8 22:32:39 nan audispd: queue is full - dropping event
  `----

  In this specific case, I count more than 20500 such lines in
  less than 13 seconds.

Version-Release number of selected component (if applicable):

  audit-1.7.2-6.fc9.i386
  kernel-PAE-2.6.25-14.fc9.i686

How reproducible:

  Sporadically, specific cause unknown.

Steps to Reproduce:

  ???
  
Actual results:

  1730 messages per second.

Expected results:

  Something on the order of ONE message per second, or less.

Additional info:

  Even if the sporadic occurrent of this problem makes it difficult to find the
underlying problem, rate-limiting this overflow message would still make sense.
Comment 1 Steve Grubb 2008-05-12 13:04:58 EDT
This is a problem I've been looking at for a few weeks. I added separate
priority boost controls between audispd and auditd in audit-1.7.3-1. It can be
found here:

http://koji.fedoraproject.org/koji/buildinfo?buildID=48645

if you wanted to do any early testing. The basic problem is that depending on
your audit rules you may need to increase the priority of audispd so that its
gets more run time and increase the internal queue of audispd.
Comment 2 Bug Zapper 2008-05-14 06:51:46 EDT
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 3 Hans Ulrich Niedermann 2008-06-06 06:48:36 EDT
I have just found that my logs have the "audispd: queue is full - dropping
event" flood with whatever audit and kernel version was current on 2008-05-25.
Comment 4 Steve Grubb 2008-06-23 17:06:45 EDT
We are still looking at this. Something changed wrt the scheduler and we have
not pinpointed the best way forward. It may be a combination of changing the qos
to be lossless, increasing the priority boost, and lengthening the audispd
internal queue.
Comment 5 Steve Grubb 2008-09-19 16:31:39 EDT
Does the current audit package (1.7.5) solve the problems? It interact with the scheduler differently than earlier versions.
Comment 6 Hans Ulrich Niedermann 2008-09-20 19:56:25 EDT
I cannot see the "audispd: queue is full - dropping
event" message in my /var/log after audit was updated to audit-1.7.5-1.fc9.i386 on 2008-09-17.

However, that does not necessarily mean much: I have the "queue is full" message in the logs a few hours before audit-1.7.5-1.fc9.i386 was installed, but there is a period of a few weeks before that during which there were no "queue is full" messages even with pre-1.7.5 audit.

I'll keep my eyes open. If I do not see the "queue is full" during the next week or such and you say this is supposed be fixed, I'd say we close the bug, and I'll just reopen it in case the "queue is full" spewing happens again.
Comment 7 Steve Grubb 2008-09-25 08:34:47 EDT
Thanks for the feedback. Closing bug.
Comment 8 Derek Atkins 2010-03-08 10:21:37 EST
For what it's worth, I'm seeing this in Fedora 12, with audit-2.0.4-1.fc12.i686

This system is pretty much just running as my asterisk server.
Comment 9 Steve Grubb 2010-03-08 10:33:17 EST
When you get this: 1) boost the priority of audispd and possibly auditd
2) See if you are running afoul of SE Linux AVCs.
Comment 10 Derek Atkins 2010-03-08 10:39:31 EST
I tried boosting it, but then I took a look at the system as a whole and noticed I had a spinning emacs and setroubleshootd process taking my system up load a load of 4.5.  I killed both of those off and then the system came back down to a reasonable state and the log messages disappeared.  The spinning emacs was due to a network failure that caused my session to break mid-stream.

Sorry for the noise.
Comment 11 Jim Meyering 2010-03-16 15:38:48 EDT
I'm on a just-upgraded-to-F13 system and am seeing this, too.  Over 100 AVCs per minute. I've filed a couple of BZs with the bigger culprits:

https://bugzilla.redhat.com/show_bug.cgi?id=571880
https://bugzilla.redhat.com/show_bug.cgi?id=571939
Comment 12 aergistal 2010-08-26 16:00:35 EDT
I experienced this on Fedora 13. Over 120000 lines were written to messages in conjunction with an AVC denial of spamassassin. Fixed it by enabling boolean spamassassin_can_network.

Note You need to log in before you can comment on or make changes to this bug.