Description of problem:
In case the remote service (e.g. SIEM) becomes unavailable for just a few seconds, 'audisp-remote' process consumes 100% of one CPU. On a system with very few audit events it takes 'audisp-remote' several minutes (or much longer) to get restarted. On single CPU systems this has a high impact.
Version-Release number of selected component:
- RHEL 8.3
Steps to Reproduce:
- Install 'audispd-plugins'
remote_server = rhel-01.rr-int1.net
format = ascii
active = yes
- On remote host (rhel-01) listen on port
# nc --listen --keep-open 60
- service auditd restart
- On remote host (rhel-01) stop netcat command and start it again
- 'audisp-remote' is utilizing 100% of one CPU
- Additional CPUs don't seem to be affected
- 'audisp-remote' doesn't utilize any of the CPUs with 100%
- This is especially an issue on single CPU systems
- On systems with few audit events, it can take several minutes (or much longer) until the process is restarted
- auditd: plugin /sbin/audisp-remote terminated unexpectedly
auditd: plugin /sbin/audisp-remote was restarted
- This can also be observed, if the remote server was only unavailable for a few seconds
- Seems to affect RHEL 7 and RHEL 8
This is also a severe issue in virtual Environments. On hosts with many VMs sending their audit messages to the same destination, this saturates CPU on the virhost, when destination becomes unavailable.
Although it doesn't seem to be documented, it looks like audisp-remote in "format=ascii" mode is nothing more then a program that relays stdin to a remote tcp port. On a test system I was able to use a socat wrapper as a custom audisp plugin and it looks like it's a perfect drop-in relacement (without the bug and the added feature that we could even use TLS as transport encryption).
Could you please confirm that my findings are correct as I think this could be a feasible solution for us.
We already evaluated other solutions like shipping the contents of /var/log/audit/audit.log. As we need a quick and lightweight solution (because of regulatory reasons) using the auditd multiplexer was (besides this bug) the perfect way. In the long run we are planing to switch to auditbeat anyway.
Upstream commit 3e45aa9 should fix this.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (audit bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.