Bug 191735

Summary: Logins hang after auditd messages are thrown in syslog..
Product: Red Hat Enterprise Linux 3 Reporter: Michael Romero <mike.l.romero>
Component: lausAssignee: Jason Vas Dias <jvdias>
Status: CLOSED NOTABUG QA Contact: Jay Turner <jturner>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.0CC: srevivo
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-05-22 16:51:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michael Romero 2006-05-15 15:13:27 UTC
Description of problem:
I'm experiencing an issue on one of my servers where SSH and Console logins 
will hang.  I've noticed that each time this starts to happen, there is a 
corresponding entry in /var/log/messages thrown by auditd.. 

May 13 12:10:02 vq2xr001 audbin[27818]: saving binary audit 
log /var/log/audit.d/bin.1
May 13 12:10:02 vq2xr001 audbin[27818]: threshold 20.00 exceeded for 
filesystem /var/log/audit.d/. - free blocks down to 19.8
2%
May 13 12:10:02 vq2xr001 auditd[840]: Notify command /usr/sbin/audbin -
S /var/log/audit.d/save.%u -C -T 20% exited with statu
s 1
May 13 12:10:02 vq2xr001 auditd[840]: output error
May 13 12:10:02 vq2xr001 auditd[840]: output error
May 13 12:10:02 vq2xr001 auditd[840]: output error; suspending execution

I checked /etc/audit/audit.conf to compare it against my other systems and they 
are identical.  

Version-Release number of selected component (if applicable):
laus-0.1-70RHEL3

How reproducible:
This will happen every few days once the file-size parameter 
in /etc/audit/audit.log is reached.  

Steps to Reproduce:
1.  Reboot Server once issue arises
2.  Wait for a few days for the audit log to fill up? (not sure what fills it 
up)
3.  Experience Issue
  
Actual results:
Cannot log into server using SSH or Console.  A reboot is required in order for 
the system to be accessible at this point.

Expected results:
Normal operation.

Additional info:

Comment 1 Jason Vas Dias 2006-05-22 16:51:06 UTC
The Linux AUdit Subsystem (LAuS) was enabled by default for RHEL-3 systems pre-U5,
which in retrospect appears to have been a mistake - systems clean-installed from
a RHEL-3-U5 or later do not have the audit system enabled by default.

The 'audit' service will maintain logs of all auditable events and system calls
occuring on the system in /var/log/audit.d/* . 

Once the free disk space under /var/log/audit.d/ falls below a threshold 
configurable in /etc/audit/audit.conf:
"
  notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%";
"
audit will be unable to create further disk logs and will enter the state
configurable in /etc/audit.conf, which is by default:
"
       error {
                action {
                        type = suspend;
                };
"
You can stop LAuS putting the system into suspend mode when it is unable 
to write audit logs by removing the 'action { type=suspend; }' clause 
above. 

If you do not want to enable auditing on your system, then simply:
  # chkconfig --level=0123456 audit off
and reboot; no more logs will be created and nothing else depends 
on audit being enabled - this is the default for post-U5 clean-installed
systems.

Or, if you want to continue running with audit enabled, you can adjust the
threshold in /etc/audit/audit.conf:
'
  notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 1%";
'
or make audbin delete the old log files by default so that the default is
never reached:
'
  notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 10% \
                     -N '/bin/rm -f %f'
' 
or move old log files to a different location:
'
   notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 10% \
                     -N '/bin/mv -f %f /another_partition/'
 
'
or process old log files with some script:
'
   notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 10% \
                     -N '/bin/my_audit_log_rotation_script %f'
 
'

Systems which are to be audited must not allow the system to be used if 
auditing is not possible - this is demanded by the audited system 
certification requirements, and is the way LAuS is designed to operate.

You do not need to enable audit by default - if you don't require auditing,
just disable the 'audit' service as described above.

If you do enable auditing, you need to configure a mechanism for rotation
of log files so that the audit logs do not exhaust the free space on the
audit log directory partition. As shown above, LAuS provides many flexible
mechanisms with which to do this, which are fully documented in the man-pages
: audit.conf(5), audit(4), laus(7), audbin(1) .

Hence, this is 'NOTABUG' - the audit daemon was only doing what it was
told to do by the site configuration and was behaving as designed.