Bug 158546 - audit fills /var
Summary: audit fills /var
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: laus
Version: 3.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Adrian Likins
QA Contact: Fanny Augustin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2005-05-23 14:27 UTC by Elliot Peele
Modified: 2007-11-30 22:07 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-05-24 20:25:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Elliot Peele 2005-05-23 14:27:17 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:
Most of the time

Steps to Reproduce:
1. install rhel3
2. wait awhile
3. /var will fill up
  
Actual results:
U5 update broke on many machines because audit was using several hundred MB,
filling /var to the point that all of the packages in the update wouldn't fit in
/var. 

Expected results:
Audit logs shouldn't fill up /var to the point of breaking updates.

Additional info:
Also had problems with audit filling /var and breaking other services because
they were trying to write to /var/log, /var/cache, or /var/spool.

Comment 1 Jason Vas Dias 2005-05-24 16:05:06 UTC
Did this problem occur before, after or during upgrade to RHEL-3-U5 ?

You say:
"
U5 update broke on many machines because audit was using several
hundred MB,
filling /var to the point that all of the packages in the update
wouldn't fit in
/var.
"

Do you mean you had a pre-U5 RHEL-3 installed, and then U5 update
failed because of lack of space in /var ?

U5 update does address this problem - the 'audbin' program that
rotates the audit log is now given a threshold parameter of the
amount of free space it must maintain (bug #130071) .

But it sounds like you couldn't install U5 because of the /var
space problems ?



Comment 2 Elliot Peele 2005-05-24 18:11:07 UTC
Yes these were pre U5 machines that broke when trying to update to U5 because of
a lack of space in /var.

Looking over the posted fix, that doesn't seem to be the correct way to go about
fixing this. Writing a logrotate script that make audit rotate like any other
log would make more sense, unless there is a good reason not to.

Also if I am reading that correctly audit stops loging things once it has
reached the threashold and by default you are depending on all of the users to
deal with rotating logs themselves.


An example of why just saying use this percentage is bad:

At NCSU we have around 1000 machines running Red Hat Enterprise Linux 3 in
production. On almost all of them /var is a 1GB partition. Before audit came
along there hadn't been any problems with /var being 1GB.

If audit had been running along and filling up all but 20% if the partition that
would leave 200MB free. This would have broken the update to U5 on just about
all of them because U5 was well over 200MB.


Comment 3 Jason Vas Dias 2005-05-24 18:50:18 UTC
I'm sorry that the audit logs are causing you problems.

If you do not require the audit feature, perhaps the simplest solution
is to turn it off:
# service del audit; rm -f /var/log/audit.d/*
Performance of the system as a whole will increase.
On a clean install of U5, the audit service is no longer enabled by default.
 
Audit logging is wholly user configurable through /etc/audit/audit.conf .

You said:
> Writing a logrotate script that make audit rotate like any other
> log would make more sense, unless there is a good reason not to.

By default, laus auditd is configured to use BINARY log files, and
it has a built in automatic log rotation mechanism: audbin , which
is because of this line in audit.conf:
     notify = "/usr/bin/audbin -S /var/log/audit/save.%u -T 20%";

audbin takes the full binary "bin.%d" audit log, saves the records in it to 
a "save.%u" file, and clears it out. 

You can also configure audit to log to a text file, which has different
threshold and rotation mechanisms, or to syslog - see "man audit.conf" -
but performance would be adversely impacted, so binary logging is the
default.

In U5, audbin now accepts the -T <threshold> and -N <notify command>
options. 
When the percentage of free disk space on the partition to which audbin
is trying to create the 'save.%u' saved log file is less than the 
-T <threshold> parameter (by default 20%) , audbin will execute 
the -N command for each oldest  save.%u file, until there is at least
<threshold> free space left . 
So you could specify a -N command of 'mv %f /backup' to move the oldest
save.%u file to a /backup partition, or 'rm %f' to remove it - see 
"man audbin" .

So we are NOT 
"depending on all of the users to deal with rotating logs themselves" -
but administrators are enabled to configure audit logging and automatic
audit log rotation to suit their audit requirements and disk usage policies .

Really the root cause of this bug is nothing to do with audit, but with
up2date's lack of checking the available disk space under /var/spool/up2date ;
if up2date does not have enough space under this directory to download all its
RPMs, it should refuse to perform the upgrade with an informative error message.
Perhaps you might consider removing /var/spool/up2date and making it a link
to /tmp . 

So I'm moving this bug to up2date, which should check that there is enough
disk space under /var/spool/up2date before allowing the update to proceed .

Comment 4 Jason Vas Dias 2005-05-24 18:52:13 UTC
Correction the above line:
# service del audit; rm -f /var/log/audit.d/*
should be:
# chkconfig --del audit; rm -f /var/log/audit.d/*


Comment 5 Elliot Peele 2005-05-24 19:57:49 UTC
This isn't a bug in up2date. Up2date does the right thing and checks for
available space and returns an errors saying that it doesn't have enough space
to download the rpms. The point being that audit filling up /var breaks the
update, which tends to leave a system in a vulnerable state without any good way
to fix it.

Comment 6 Jason Vas Dias 2005-05-24 20:25:09 UTC
If, as you say:
> Up2date does the right thing and checks for available space and 
> returns an errors saying that it doesn't have enough space to 
> download the rpms

Then the update is not "broken", as it was never initiated, and the
system is no more "vulnerable" than it was before the update was
attempted .

There is a "good way to fix it" : remove the /var/log/audit.d/save.* 
files that you no longer require before doing the upgrade, or move
/var/spool/up2date onto a partition with more free space.

As I said in Comment #3 above, in U5 laus has mechanisms to control
how much free space to maintain on the saved audit log partition,
and gives you control over how old saved audit logs are rotated -
see "man 1 audbin" .

The LAuS audit system gives you plenty of ways to configure
the size and type of audit logs, audit log rotation, and free disk
space thresholds, and auditing itself is optional .

It is up to users to configure LAuS to suit their needs and systems.

The default configuration of LAuS is suitable for most needs, but
may require customization by users .

In short, this problem is not due to a problem with LAuS, but in 
the way it was configured on systems before RHEL-3-U5. In RHEL-3-U5,
LAuS gives you more ways to prevent this type of problem occuring.

Hence, this bug is being closed as "NOT A BUG".



Note You need to log in before you can comment on or make changes to this bug.