This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 2207869 - When audit has "admin_space_left_action = halt", the system is halted with no visible message on the console
Summary: When audit has "admin_space_left_action = halt", the system is halted with no...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: audit
Version: 9.2
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Sergio Correia
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-05-17 07:28 UTC by Renaud Métrich
Modified: 2023-08-08 11:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-08 11:14:22 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-990 0 None None None 2023-08-08 11:14:21 UTC
Red Hat Issue Tracker RHELPLAN-157451 0 None None None 2023-05-17 07:28:42 UTC
Red Hat Issue Tracker SECENGSP-5193 0 None None None 2023-05-17 07:28:50 UTC
Red Hat Knowledge Base (Solution) 3352261 0 None None None 2023-05-17 07:31:55 UTC

Description Renaud Métrich 2023-05-17 07:28:20 UTC
Description of problem:

CIS level 2 hardening requires having "admin_space_left_action = halt" in the configuration.
If for some reason the space gets low on /var/log/audit, it becomes impossible to boot the system, since it goes down immediately during the boot.

Checking the console output, nowhere we can see an explicit message stating auditd is halting the system.
Only a message is recorded in the journal, *assuming the journal is persistent, which is far from being the usual case*:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
May 17 08:58:23 vm-audit9 auditd[682]: Audit daemon is low on disk space for logging
May 17 08:58:23 vm-audit9 auditd[682]: The audit daemon is now halting the system 
May 17 08:58:23 vm-audit9 auditd[682]: No plugins found, not dispatching events 
May 17 08:58:23 vm-audit9 systemd[1]: Started RPC Bind.
May 17 08:58:23 vm-audit9 auditd[682]: Init complete, auditd 3.0.7 listening for events (startup state enable)
 :
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

But definitely this message is not visible to admins on the console:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
         Mounting /var/log/audit...
[    3.249105] XFS (vdb): Mounting V5 Filesystem
[    3.274115] XFS (vdb): Ending clean mount
[  OK  ] Mounted /var/log/audit.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
 :
[  OK  ] Removed slice Slice /system/modprobe.
[  OK  ] Removed slice Slice /system/systemd-hibernate-resume.
[  OK  ] Stopped target RPC Port Mapper.
[  OK  ] Closed LVM2 poll daemon socket.
[  OK  ] Closed Process Core Dump Socket.
[  OK  ] Started /usr/sbin/lvm vgch…y --autoactivation event rhel.
         Stopping Load/Save Random Seed...
[  OK  ] Removed slice Slice /system/getty.
[  OK  ] Removed slice Slice /system/serial-getty.
[  OK  ] Removed slice Slice /system/sshd-keygen.
[  OK  ] Stopped Generate network units from Kernel command line.
[  OK  ] Stopped target Path Units.
[  OK  ] Stopped target Slice Units.
 :
[  OK  ] Stopped target Local File Systems.
         Unmounting /boot...
         Unmounting /run/credentials/systemd-sysctl.service...
         Unmounting /run/credential…-tmpfiles-setup-dev.service...
[    3.471501] XFS (vda1): Unmounting Filesystem
         Unmounting /var/log/audit...
 :
[    3.641752] reboot: Power down
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

All this makes it *really hard* to troubleshoot, and usually requires Red Hat Support to enter the game, *and them to struggle a lot*.

We need to highlight the operation on the console and probably log the message in the journal as "CRIT" as well.


Version-Release number of selected component (if applicable):

All audit releases

How reproducible:

Always

Steps to Reproduce:
1. Configure auditd to halt the system

   # sed -i "s/admin_space_left_action = .*/admin_space_left_action = halt/" /etc/audit/auditd.conf

2. Set up a dedicated mount for /var/log/audit (1GB)

   # mkfs.xfs /dev/vdb
   # echo "/dev/vdb /var/log/audit xfs defaults 0 0" >> /etc/fstab
   # service auditd stop
   # systemctl daemon-reload
   # mount /var/log/audit
   # restorecon -Frv /var/log/audit
   # service auditd start

3. Fill the mount point with wreck

   # dd if=/dev/random of=/var/log/audit/filler bs=1M count=970

4. Reboot

Actual results:

System goes down during boot

Expected results:

System goes down after printing a *large visible red message* on the console

Comment 1 Steve Grubb 2023-05-17 17:13:48 UTC
The audit event is sent to syslog with a priority of ALERT which is higher than CRIT. In testing on F38, logging with LOG_CONS is not honored. I suppose journald does not have ALERT or CONS capabilities? Even as root, no console messages in case capabilities were needed. Writing to /dev/console also does not send a message to the console. The wall command also seems non-functional. Not sure what else can be done. I did make a patch (upstream commit 0b118c7) to shutdown auditd when processing any runlevel change. Maybe that is noticeable?


Note You need to log in before you can comment on or make changes to this bug.