Bug 170472

Summary: /var fills up when IBM Informix IDS makes lots of semop() calls.
Product: Red Hat Enterprise Linux 3 Reporter: Ronald Cole <ronald>
Component: lausAssignee: Steve Grubb <sgrubb>
Status: CLOSED WONTFIX QA Contact: Jay Turner <jturner>
Severity: high Docs Contact:
Priority: high    
Version: 3.0CC: marcobillpeter, pgraner, poelstra, srevivo, tom.obrien
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-04-10 02:03:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 190430    

Description Ronald Cole 2005-10-12 00:30:13 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050921 Red Hat/1.7.12-1.1.3.2

Description of problem:
A RHEL 3WS Update 5 system serving as a production server for IBM Informix IDS 9.40.UC5 simply fills up /var/log/audit.d by generating over 100mb of log entries for "semop" a day!!

On a recent Friday evening, /var reached 80% and LAuS suspended.  By Monday morning, the kernel was blocking sshd from trying to enqueue new audit events for delivery to the audit daemon, preventing anyone from logging in remotely to liberate some log files from the filesystem.



Version-Release number of selected component (if applicable):
laus-0.1-70RHEL3

How reproducible:
Always

Steps to Reproduce:
1. Install RHEL3WS
2. Install IBM Informix IDS 9.40
3. Run lots and lots of 4GL programs to crunch lots and lots of data.
  

Actual Results:  LAuS causes /var to eventually fill to 80% much, much faster than a system without Informix IDS installed.  Hilarity ensues.

Expected Results:  /var/log/audit.d to not fill up so much faster than a system without Informix IDS installed.

Additional info:

After a moderate amount of investigation, I found this LAuS white paper:
http://ftp.isr.ist.utl.pt/pub/MIRRORS/ftp.suse.com/projects/security/laus/laus/doc/LAuS-Design.pdf>
and determined that RedHat just used the example configuration file for filter.conf which logs ALL ipc syscalls.

RedHat should probably go over the filter.conf configuration file and limit it strictly to just the logging the calls required for CAPP compliance.

In my immediate case, I applied the following patch (but I'm not sure that it's 100% correct):

--- /etc/audit/filter.conf~     2005-04-07 11:24:53.000000000 +0900
+++ /etc/audit/filter.conf      2005-10-12 08:54:42.000000000 +0900
@@ -417,8 +417,19 @@
 syscall @timechange-ops = always;
  
  
-# not required by CAPP
-syscall ipc = always;
+## not required by CAPP
+#syscall ipc = always;
+#
+# only what is required by CAPP,
+# otherwise IBM Informix IDS 9.40.UC5 will fill /var relatively quickly with
+# a simply unbelievable number of calls to "semop" that we don't care about.
+#
+set    ipc-ops = {
+                       "msgctl", "msgget",
+                       "semctl", "semget",
+                       "shmat", "shmctl", "shmget",
+};
+syscall @ipc-ops = always;
  
 syscall socket = is-af-packet(arg0) || is-sock-raw(arg1);
 syscall ioctl = is-sysconf-ioctl(arg1);

Comment 1 Jason Vas Dias 2005-10-12 16:09:23 UTC
By default, on a clean installation, auditing is not enabled in RHEL-3-U5+ 
( the 'audit' SysVinit service is not enabled in any runlevel by default ).

You should only enable auditing if you actually require the audit log records.

If you enable auditing, you need to configure a mechanism for handling 
audit log rotation and the case when there is insufficient disk space
to rotate an audit log - there are several mechanisms for doing so, as
documented in /etc/audit/audit.conf:

notify		= "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%";
#  AUDBIN THRESHOLDS:
#       The above notify will cause auditd to enter 'suspend' mode when 
#       free space on the /var/ filesystem falls below 20%. 
#       To take remedial action, eg. moving the oldest save file to /backup, use:
#	notify		= "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'mv -f %f
/backup'";
#       or even
#	notify		= "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'rm -f %f'";
#       This will free space by removing the oldest "save." files first from /var,
#       returning 0 to auditd and allowing it to continue.


See 'man audbin' for details on the -T and -N options.

By default, audit will put the system into suspend mode when insufficient 
disk space exists to save an audit log and maintain the audbin -T threshold.
This must be the default action to take, as any other action could result
in loss of audit data. 

You can either delete saved audit logs, move them to a different filesystem,
or not create 'save.*' saved audit log files at all, by making the audit.conf
notify option something like:
   notify =  "my_audit_log_analyser";
where "my_audit_log_analyser" is a script that uses the data in the binary 
audit log given its first argument, perhaps saving audit records of interest
to some database, and then calls 'audbin -C $audit_log' .

Many other users may depend on the filter.conf shipped by default in the laus
package, and the default filters cannot be changed to accomodate the needs of
Informix users.

Everything about the LAuS audit package is meant to be configured to suit 
individual user requirements, as you have done. 

No other RHEL-3 software depends on the audit system being enabled, so if do
not actually require the audit log data, the best solution is to disable
the audit service:
  # chkconfig --level=0123456 audit off
which is the state in which LAuS is installed on a clean install of RHEL-3-U5+. 

Comment 2 Ronald Cole 2005-10-12 17:49:30 UTC
I, personally, can't think of any valid reason for any admin to depend on the
logging of semop() calls.  It's just noise and apparently not required by CAPP.

Nor can I think of any valid reason for a user to depend on a direct lift of the
*EXAMPLE* filter.conf from a white paper.  It would be much better to simply
expend the effort to create an useful filter.conf and restrict it to logging
what's required by CAPP and then letting the security admin add to it if their
needs require it.

Finally, I didn't install RHEL3 at Update 5, I installed it much earlier when
audit WAS enabled by default.  There are probably lots of RHEL3 systems
installed with audit enabled and in it's default configuration that are
vulnerable to a DOS attack by a user with a program calling semop() in an
infinite loop.  But if Red Hat would rather expend the effort to rationalize
this than to fix it and issue an errata, that's it's business I, suppose.  *sigh*

Comment 3 Jason Vas Dias 2005-10-12 17:59:47 UTC
OK, I will remove the semop* filters from the default configuration. 
But this will not be fixed during upgrade, as /etc/audit/filter.conf is a 
%config(noreplace) file - users are expected to customize it to suit their
requirements, and an RPM upgrade will not replace this file - it will be
created as /etc/audit/filter.conf.rpmnew . I've committed this change to CVS
and it will be in laus-0.1-74RHEL3+ .  Also, we cannot disable the laus service
in an RPM upgrade - this must be left to the local administrator's decision, as
there is no way of determining if it was enabled only because it was installed
pre-RHEL-3-U5 or if the administrator enabled it after clean installing U5.
  



Comment 4 Ronald Cole 2005-10-12 18:59:18 UTC
Hmmm, since RHBA-2005:219-07 neglected to mention that audit was previously
installed enabled by default and should be disabled until it's properly
configured, perhaps the new errata should include all these caveats and warnings
for sys admins who have never heard of CAPP or EAL3.

Comment 12 Ronald Cole 2006-03-22 00:14:11 UTC
Wow... you've completely missed getting laus-0.1-74RHEL3+ into Update 6 or
Update 7!  I keep getting email, as recently as today, that you're making other
bugs depend on this one, but no errata activity for RHEL3 whatsoever on this
one.  Is Red Hat foot dragging until April 30, 2006?  Paying customers want to know.