Bug 1380084

This is caused by fixing the flush mode for sync and data to actually do what they claimed to do. Of course this yields low performance but they get the guarantee that they are after. What they were getting before in spite of requesting data was "none". This was documented in the 6.8 release notes.

Bug 1369249 will create a new flush mode that gives higher performance while allowing the freq setting to be relatively low like 50. This gives a good balance between speed and keeping events synced.

In the mean time, they can use incremental flush mode and set freq to something like 200 or a little higher. Or they can set flush = none which is what they had before.

Steps to reproduce are incorrect.  
The issue is with RHEL 6.8 version of audit packages (2.4.5-3) NOT RHEL 6.7 audit packages (2.3.7-5 - RHEL 6.7 pkgs work as expected)!

Again steps to reproduce:
1. UPGRADE to RHEL 6.8 - or fresh RHEL 6.8 install with audit pkgs version 2.4.5-3 installed.

2. Configure auditd.conf as noted above.

3. Set audit.rules as follows to create many entries in audit.log during delete:
## This file contains the auditctl rules that are loaded
## whenever the audit daemon is started via the initscripts.
## The rules are simply the parameters that would be passed
## to auditctl.
##
## First rule - delete all
-D

# Enable system call auditing, disabled in leue of specific system call auditing
#-e 1

## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 131072

## Set failure mode to print failure messages (1) vs. panic (2)
-f 2



## Note: if this is being used on a 32 bit machine, comment out the b64 lines
## These rules assume that login under the root account is not allowed.
## It is also assumed that 500 represents the first usable user account
##
## (GEN002880: CAT II) The IAO will ensure the auditing software can
## record the following for each audit event:
##- Date and time of the event
##- Userid that initiated the event
##- Type of event
##- Success or failure of the event
##- For I&A events, the origin of the request (e.g., terminal ID)
##- For events that introduce an object into a userâaddress space, and
##  for object deletion events, the name of the object, and in MLS
##  systems, the objectâsecurity level.
##
## Things that could affect time
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -k time-change
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change
#-a always,exit -F arch=b32 -S clock_settime -k time-change
#-a always,exit -F arch=b64 -S clock_settime -k time-change
-w /etc/localtime -p wa -k time-change

## Things that affect identity
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity

## Things that could affect system locale
-a exit,always -F arch=b32 -S sethostname -k system-locale
-a exit,always -F arch=b64 -S sethostname -k system-locale
-w /etc/issue -p wa -k system-locale
-w /etc/issue.net -p wa -k system-locale
-w /etc/hosts -p wa -k system-locale
-w /etc/sysconfig/network -p wa -k system-locale

## Things that could affect MAC policy
-w /etc/selinux/ -p wa -k MAC-policy


## (GEN002900: CAT III) The IAO will ensure audit files are retained at
## least one year; systems containing SAMI will be retained for five years.
##
## Site action - no action in config files

## (GEN002920: CAT III) The IAO will ensure audit files are backed up
## no less than weekly onto a different system than the system being
## audited or backup media.
##
## Can be done with cron script

## (GEN002700: CAT I) (Previously â095) The SA will ensure audit data
## files have permissions of 640, or more restrictive.
##
## Done automatically by auditd

## (GEN002720-GEN002840: CAT II) (Previously â100-G106) The SA will
## configure the auditing system to audit the following events for all
## users and root:
##
## - Logon (unsuccessful and successful) and logout (successful)
##
## Handled by pam, sshd, login, and gdm
## Might also want to watch these files if needing extra information
#-w /var/log/faillog -p wa -k logins
#-w /var/log/lastlog -p wa -k logins


##- Process and session initiation (unsuccessful and successful)
##
## The session initiation is audited by pam without any rules needed.
## Might also want to watch this file if needing extra information
#-w /var/run/utmp -p wa -k session
#-w /var/log/btmp -p wa -k session
#-w /var/log/wtmp -p wa -k session

##- Discretionary access control permission modification (unsuccessful
## and successful use of chown/chmod)
-a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod

##- Unauthorized access attempts to files (unsuccessful)
-a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-13 -F auid>=500 -F auid!=4294967295 -k access
-a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-1 -F auid>=500 -F auid!=4294967295 -k access
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-13 -F auid>=500 -F auid!=4294967295 -k access
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-1 -F auid>=500 -F auid!=4294967295 -k access

##- Use of privileged commands (unsuccessful and successful)
## use find /bin -type f -perm -04000 2>/dev/null and put all those files in a rule like this
-a always,exit -F arch=b32 -S execve -F path=/bin/ping -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b64 -S execve -F path=/bin/ping -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b32 -S execve -F path=/bin/ping6 -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b64 -S execve -F path=/bin/ping6 -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b32 -S execve -F path=/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b64 -S execve -F path=/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b32 -S execve -F path=/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b64 -S execve -F path=/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b32 -S execve -F path=/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F arch=b64 -S execve -F path=/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged

##- Use of print command (unsuccessful and successful)

##- Export to media (successful)
## You have to mount media before using it. You must disable all automounting
## so that its done manually in order to get the correct user requesting the export
-a always,exit -F arch=b32 -S mount -F auid>=500 -F auid!=4294967295 -k export
-a always,exit -F arch=b64 -S mount -F auid>=500 -F auid!=4294967295 -k export

##- System startup and shutdown (unsuccessful and successful)

##- Files and programs deleted by the user (successful and unsuccessful)
-a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete

##- All system administration actions
##- All security personnel actions
##
## This should be recorded by sudo. Do not allow unrestricted root shells
-w /etc/sudoers -p wa -k actions

## (GEN002860: CAT II) (Previously â674) The SA and/or IAO will
##ensure old audit logs are closed and new audit logs are started daily.
##
## Site action. Can be assisted by a cron job

## Not specifically required by the STIG; but common sense items
## Optional - could indicate someone trying to do something bad or
## just debugging
#-a entry,always -F arch=b32 -S ptrace -k tracing
#-a entry,always -F arch=b64 -S ptrace -k tracing

## Optional - could be an attempt to bypass audit or simply legacy program
#-a always,exit -F arch=b32 -S personality -k bypass
#-a always,exit -F arch=b64 -S personality -k bypass

## Put your own watches after this point
# -w /your-file -p rwxa -k mykey
# These rules are from lspp.rules (Labeled Security Protection Profile)
##
## FAU_SAR.1, FAU_SAR.2, FMT_MTD.1
## successful and unsuccessful attempts to read information from the
## audit records; all modifications to the audit trail
##
#-w /var/log/audit/ -k LOG_audit
-w /var/log/audit/audit.log -k LOG_audit.log

##
## FAU_SEL.1, FMT_MTD.1
## modifications to audit configuration that occur while the audit
## collection functions are operating; all modications to the set of
## audited events
##
-w /etc/audit/auditd.conf -p wa -k CFG_auditd.conf
-w /etc/sysconfig/auditd  -p wa -k CFG_auditd.conf
-w /etc/audit/audit.rules -p wa -k CFG_audit.rules
-w /etc/libaudit.conf -p wa -k CFG_libaudit.conf

##
## FDP_ETC.2
## Export of Labeled User Data
##
## Printing
-w /etc/cups/ -p wa -k CFG_cups
-w /etc/init.d/cups -p wa -k CFG_initd_cups

##
## FDP_IFC.1
## Mandatory Access Control Policy
##
-w /etc/selinux/config -p wa -k CFG_selinux_config
-w /etc/selinux/semanage.conf -p wa -k CFG_MAC_policy
-w /etc/selinux/restorecond.conf -p wa -k CFG_MAC_policy

##
## FPT_TST.1 Self Test
## aide is used to verify integrity of data and executables
##
-w /etc/security/rbac-self-test.conf -p wa -k CFG_RBAC_self_test
-w /etc/aide.conf -p wa -k CFG_aide.conf
-w /var/lib/aide/aide.db.gz -k CFG_aide.db
-w /var/lib/aide/aide.db.new.gz -k CFG_aide.db
-w /var/log/aide -p wa -k CFG_aide.log
-w /var/log/aide/aide.log -p wa -k CFG_aide.log

##
## Security Databases
##

## cron configuration & scheduled jobs
-w /etc/cron.allow -p wa -k CFG_cron.allow
-w /etc/cron.deny -p wa -k CFG_cron.deny
-w /etc/cron.d/ -p wa -k CFG_cron.d
-w /etc/cron.daily/ -p wa -k CFG_cron.daily
-w /etc/cron.hourly/ -p wa -k CFG_cron.hourly
-w /etc/cron.monthly/ -p wa -k CFG_cron.monthly
-w /etc/cron.weekly/ -p wa -k CFG_cron.weekly
-w /etc/crontab -p wa -k CFG_crontab
-w /var/spool/cron/root -k CFG_crontab_root

## user, group, password databases
-w /etc/group -p wa -k CFG_group
-w /etc/passwd -p wa -k CFG_passwd
-w /etc/gshadow -k CFG_gshadow
-w /etc/shadow -k CFG_shadow
-w /etc/security/opasswd -k CFG_opasswd

## login configuration and information
-w /etc/login.defs -p wa -k CFG_login.defs
-w /etc/securetty -p wa -k CFG_securetty
-w /var/log/faillog -p wa -k LOG_faillog
-w /var/log/lastlog -p wa -k LOG_lastlog
-w /var/log/tallylog -p wa -k LOG_tallylog

## network configuration
-w /etc/hosts -p wa -k CFG_hosts
-w /etc/sysconfig/ -p wa

## system startup scripts
-w /etc/inittab -p wa -k CFG_inittab
-w /etc/rc.d/init.d/ -p wa
-w /etc/rc.d/init.d/auditd -p wa -k CFG_initd_auditd

## library search paths
-w /etc/ld.so.conf -p wa -k CFG_ld.so.conf

## local time zone
-w /etc/localtime -p wa -k CFG_localtime

## kernel parameters
-w /etc/sysctl.conf -p wa -k CFG_sysctl.conf

## modprobe configuration
-w /etc/modprobe.conf -p wa -k CFG_modprobe.conf

## pam configuration
-w /etc/pam.d/ -p wa -k CFG_pam
-w /etc/security/limits.conf -p wa  -k CFG_pam
-w /etc/security/pam_env.conf -p wa -k CFG_pam
-w /etc/security/namespace.conf -p wa -k CFG_pam
-w /etc/security/namespace.init -p wa -k CFG_pam

## postfix configuration
-w /etc/aliases -p wa -k CFG_aliases
-w /etc/postfix/ -p wa -k CFG_postfix

## ssh configuration
-w /etc/ssh/sshd_config -k CFG_sshd_config

## vsftpd configuration

-w /etc/vsftpd.ftpusers -k CFG_vsftpd.ftpusers
-w /etc/vsftpd/vsftpd.conf -k CFG_vsftpd.conf



4. Create a directory with many files ~70K files / 5GB total dir size, and delete the directory with '/bin/rm -rf' command (or a mv command) to generate enough audit.log entries to cause a logrotate at the specified max_log_file=36 in /etc/audit/auditd.conf = 37748736 bytes.   System will hang/halt or audit panic.

This is working exactly as it should. Its set to -f 2 which say panic if we overflow the backlog, the backlog is overflowed by the move, the system panics. Working per design.

What changed between 6.7 and 6.8 was that data or sync flush modes are actually honored rather than giving you the "none" mode. This was documented in the release notes.

To not overflow the backlog, you need to change the flush mode to something else because its too slow for this work load.

Steve, thanks for the feedback.  I'm testing now with flush = none in the auditd.conf file (did a 'service auditd reload' to activate).

To summarize - Setting flush=data never really worked before version 2.4.5, it was really acting like flush=none.  
Your recommendation if to set flush to ~200+ (which may be slow??) until bug 1369249 is resolved - or we can/should? just leave flush=none until fix and re-evaluate at that time?

Thanks again for quick reply.

(In reply to John G from comment #4)
> To summarize - Setting flush=data never really worked before version 2.4.5,
> it was really acting like flush=none.

Correct.

> Your recommendation if to set flush to ~200+ (which may be slow??) until bug
> 1369249 is resolved - or we can/should?

That would be flush = incremental and freq = 200.

> just leave flush=none until fix and
> re-evaluate at that time?

Its up to you. Either should work. When the "none" mode is given, the OS figures out when it can flush audit records to disk. If the kernel oopses, there's no telling how many audit records were in flight. By setting incremental, then freq determines the upper limit to how many records could be lost on an oops. Obviously it needs to be high enough that it can handle the workload given to it, but low enough to bound any risk.

The new mode scheduled for rhel 7.3 and 6.9 incremental_async give you far higher throughput while lowering the freq number to minimize risk.

Steve,
I confirmed flush=none resolved issue.  Thanks again.

PS - Is this issue also present in RHEL 7.2?

This issue is not present in 7.2. However, audit is being updated on 7.3 to 2.6.5 which will have the issue. But it also has the incremental_async mode which when chosen should let you drop freq to about 50 and handle large volumes of data. This limits risk better than incremental.

Closing this bug. You can monitor Bug 1369249 for delivery of incremental_async on RHEL 6.