Hide Forgot
Description of problem: I'm experiencing an issue on one of my servers where SSH and Console logins will hang. I've noticed that each time this starts to happen, there is a corresponding entry in /var/log/messages thrown by auditd.. May 13 12:10:02 vq2xr001 audbin[27818]: saving binary audit log /var/log/audit.d/bin.1 May 13 12:10:02 vq2xr001 audbin[27818]: threshold 20.00 exceeded for filesystem /var/log/audit.d/. - free blocks down to 19.8 2% May 13 12:10:02 vq2xr001 auditd[840]: Notify command /usr/sbin/audbin - S /var/log/audit.d/save.%u -C -T 20% exited with statu s 1 May 13 12:10:02 vq2xr001 auditd[840]: output error May 13 12:10:02 vq2xr001 auditd[840]: output error May 13 12:10:02 vq2xr001 auditd[840]: output error; suspending execution I checked /etc/audit/audit.conf to compare it against my other systems and they are identical. Version-Release number of selected component (if applicable): laus-0.1-70RHEL3 How reproducible: This will happen every few days once the file-size parameter in /etc/audit/audit.log is reached. Steps to Reproduce: 1. Reboot Server once issue arises 2. Wait for a few days for the audit log to fill up? (not sure what fills it up) 3. Experience Issue Actual results: Cannot log into server using SSH or Console. A reboot is required in order for the system to be accessible at this point. Expected results: Normal operation. Additional info:
The Linux AUdit Subsystem (LAuS) was enabled by default for RHEL-3 systems pre-U5, which in retrospect appears to have been a mistake - systems clean-installed from a RHEL-3-U5 or later do not have the audit system enabled by default. The 'audit' service will maintain logs of all auditable events and system calls occuring on the system in /var/log/audit.d/* . Once the free disk space under /var/log/audit.d/ falls below a threshold configurable in /etc/audit/audit.conf: " notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%"; " audit will be unable to create further disk logs and will enter the state configurable in /etc/audit.conf, which is by default: " error { action { type = suspend; }; " You can stop LAuS putting the system into suspend mode when it is unable to write audit logs by removing the 'action { type=suspend; }' clause above. If you do not want to enable auditing on your system, then simply: # chkconfig --level=0123456 audit off and reboot; no more logs will be created and nothing else depends on audit being enabled - this is the default for post-U5 clean-installed systems. Or, if you want to continue running with audit enabled, you can adjust the threshold in /etc/audit/audit.conf: ' notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 1%"; ' or make audbin delete the old log files by default so that the default is never reached: ' notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 10% \ -N '/bin/rm -f %f' ' or move old log files to a different location: ' notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 10% \ -N '/bin/mv -f %f /another_partition/' ' or process old log files with some script: ' notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 10% \ -N '/bin/my_audit_log_rotation_script %f' ' Systems which are to be audited must not allow the system to be used if auditing is not possible - this is demanded by the audited system certification requirements, and is the way LAuS is designed to operate. You do not need to enable audit by default - if you don't require auditing, just disable the 'audit' service as described above. If you do enable auditing, you need to configure a mechanism for rotation of log files so that the audit logs do not exhaust the free space on the audit log directory partition. As shown above, LAuS provides many flexible mechanisms with which to do this, which are fully documented in the man-pages : audit.conf(5), audit(4), laus(7), audbin(1) . Hence, this is 'NOTABUG' - the audit daemon was only doing what it was told to do by the site configuration and was behaving as designed.