RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1906065 - 'audisp-remote' process consumes 100% of one CPU, if remote location is not available
Summary: 'audisp-remote' process consumes 100% of one CPU, if remote location is not a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: audit
Version: 8.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.0
Assignee: Sergio Correia
QA Contact: Martin Zelený
Khushbu Borole
URL:
Whiteboard:
Depends On: 1939406
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-09 15:52 UTC by Rainer Beyel
Modified: 2022-05-10 16:54 UTC (History)
5 users (show)

Fixed In Version: audit-3.0.5-1.el8
Doc Type: Bug Fix
Doc Text:
.`audisp-remote` now correctly detects the availability of the remote locations Previously, the `audisp-remote` plugin did not detect that remote services became unavailable. As a consequence, the `audisp-remote` process would enter a state with high CPU usage. With this update, `audisp-remote` can properly detect remote services becoming unavailable. As a result, the process no longer enters a high-CPU-usage state.
Clone Of:
Environment:
Last Closed: 2022-05-10 15:30:12 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2022:2096 0 None None None 2022-05-10 15:30:38 UTC

Description Rainer Beyel 2020-12-09 15:52:29 UTC 
Description of problem:
  In case the remote service (e.g. SIEM) becomes unavailable for just a few seconds, 'audisp-remote' process consumes 100% of one CPU. On a system with very few audit events it takes 'audisp-remote' several minutes (or much longer) to get restarted. On single CPU systems this has a high impact.

Version-Release number of selected component:
  - RHEL 8.3
  - audit-3.0-0.17.20191104git1c2f876.el8.x86_64
  - audispd-plugins-3.0-0.17.20191104git1c2f876.el8.x86_64

Steps to Reproduce:
  - Install 'audispd-plugins'

  - /etc/audit/audisp-remote.conf
    remote_server = rhel-01.rr-int1.net
    format = ascii

  - /etc/audit/plugins.d/au-remote.conf
    active = yes

  - On remote host (rhel-01) listen on port
    # nc --listen --keep-open 60

  - service auditd restart

  - On remote host (rhel-01) stop netcat command and start it again

Actual results:
  - 'audisp-remote' is utilizing 100% of one CPU
  - Additional CPUs don't seem to be affected

Expected results:
  - 'audisp-remote' doesn't utilize any of the CPUs with 100%

Additional info:
  - This is especially an issue on single CPU systems
  - On systems with few audit events, it can take several minutes (or much longer) until the process is restarted
    - auditd[12345]: plugin /sbin/audisp-remote terminated unexpectedly
      auditd[12345]: plugin /sbin/audisp-remote was restarted
    - This can also be observed, if the remote server was only unavailable for a few seconds
  - Seems to affect RHEL 7 and RHEL 8
  - https://github.com/linux-audit/audit-userspace/blob/master/audisp/plugins/remote/audisp-remote.c#L509
Comment 2 Matthias Strelow 2020-12-10 10:50:28 UTC 
This is also a severe issue in virtual Environments. On hosts with many VMs sending their audit messages to the same destination, this saturates CPU on the virhost, when destination becomes unavailable.
Comment 3 Marco Reichwald 2020-12-11 12:22:46 UTC 
Although it doesn't seem to be documented, it looks like audisp-remote in "format=ascii" mode is nothing more then a program that relays stdin to a remote tcp port. On a test system I was able to use a socat wrapper as a custom audisp plugin and it looks like it's a perfect drop-in relacement (without the bug and the added feature that we could even use TLS as transport encryption).
Could you please confirm that my findings are correct as I think this could be a feasible solution for us.

We already evaluated other solutions like shipping the contents of /var/log/audit/audit.log. As we need a quick and lightweight solution (because of regulatory reasons) using the auditd multiplexer was (besides this bug) the perfect way. In the long run we are planing to switch to auditbeat anyway.
Comment 4 Steve Grubb 2020-12-15 19:46:24 UTC 
Upstream commit 3e45aa9 should fix this.
Comment 18 errata-xmlrpc 2022-05-10 15:30:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (audit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2096
Note You need to log in before you can comment on or make changes to this bug.