.`audisp-remote` now correctly detects the availability of the remote locations
Previously, the `audisp-remote` plugin did not detect that remote services became unavailable. As a consequence, the `audisp-remote` process would enter a state with high CPU usage. With this update, `audisp-remote` can properly detect remote services becoming unavailable. As a result, the process no longer enters a high-CPU-usage state.
Description of problem:
In case the remote service (e.g. SIEM) becomes unavailable for just a few seconds, 'audisp-remote' process consumes 100% of one CPU. On a system with very few audit events it takes 'audisp-remote' several minutes (or much longer) to get restarted. On single CPU systems this has a high impact.
Version-Release number of selected component:
- RHEL 8.3
- audit-3.0-0.17.20191104git1c2f876.el8.x86_64
- audispd-plugins-3.0-0.17.20191104git1c2f876.el8.x86_64
Steps to Reproduce:
- Install 'audispd-plugins'
- /etc/audit/audisp-remote.conf
remote_server = rhel-01.rr-int1.net
format = ascii
- /etc/audit/plugins.d/au-remote.conf
active = yes
- On remote host (rhel-01) listen on port
# nc --listen --keep-open 60
- service auditd restart
- On remote host (rhel-01) stop netcat command and start it again
Actual results:
- 'audisp-remote' is utilizing 100% of one CPU
- Additional CPUs don't seem to be affected
Expected results:
- 'audisp-remote' doesn't utilize any of the CPUs with 100%
Additional info:
- This is especially an issue on single CPU systems
- On systems with few audit events, it can take several minutes (or much longer) until the process is restarted
- auditd[12345]: plugin /sbin/audisp-remote terminated unexpectedly
auditd[12345]: plugin /sbin/audisp-remote was restarted
- This can also be observed, if the remote server was only unavailable for a few seconds
- Seems to affect RHEL 7 and RHEL 8
- https://github.com/linux-audit/audit-userspace/blob/master/audisp/plugins/remote/audisp-remote.c#L509
This is also a severe issue in virtual Environments. On hosts with many VMs sending their audit messages to the same destination, this saturates CPU on the virhost, when destination becomes unavailable.
Although it doesn't seem to be documented, it looks like audisp-remote in "format=ascii" mode is nothing more then a program that relays stdin to a remote tcp port. On a test system I was able to use a socat wrapper as a custom audisp plugin and it looks like it's a perfect drop-in relacement (without the bug and the added feature that we could even use TLS as transport encryption).
Could you please confirm that my findings are correct as I think this could be a feasible solution for us.
We already evaluated other solutions like shipping the contents of /var/log/audit/audit.log. As we need a quick and lightweight solution (because of regulatory reasons) using the auditd multiplexer was (besides this bug) the perfect way. In the long run we are planing to switch to auditbeat anyway.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (audit bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2022:2096
Description of problem: In case the remote service (e.g. SIEM) becomes unavailable for just a few seconds, 'audisp-remote' process consumes 100% of one CPU. On a system with very few audit events it takes 'audisp-remote' several minutes (or much longer) to get restarted. On single CPU systems this has a high impact. Version-Release number of selected component: - RHEL 8.3 - audit-3.0-0.17.20191104git1c2f876.el8.x86_64 - audispd-plugins-3.0-0.17.20191104git1c2f876.el8.x86_64 Steps to Reproduce: - Install 'audispd-plugins' - /etc/audit/audisp-remote.conf remote_server = rhel-01.rr-int1.net format = ascii - /etc/audit/plugins.d/au-remote.conf active = yes - On remote host (rhel-01) listen on port # nc --listen --keep-open 60 - service auditd restart - On remote host (rhel-01) stop netcat command and start it again Actual results: - 'audisp-remote' is utilizing 100% of one CPU - Additional CPUs don't seem to be affected Expected results: - 'audisp-remote' doesn't utilize any of the CPUs with 100% Additional info: - This is especially an issue on single CPU systems - On systems with few audit events, it can take several minutes (or much longer) until the process is restarted - auditd[12345]: plugin /sbin/audisp-remote terminated unexpectedly auditd[12345]: plugin /sbin/audisp-remote was restarted - This can also be observed, if the remote server was only unavailable for a few seconds - Seems to affect RHEL 7 and RHEL 8 - https://github.com/linux-audit/audit-userspace/blob/master/audisp/plugins/remote/audisp-remote.c#L509