Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
CIS level 2 hardening requires having "admin_space_left_action = halt" in the configuration.
If for some reason the space gets low on /var/log/audit, it becomes impossible to boot the system, since it goes down immediately during the boot.
Checking the console output, nowhere we can see an explicit message stating auditd is halting the system.
Only a message is recorded in the journal, *assuming the journal is persistent, which is far from being the usual case*:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
May 17 08:58:23 vm-audit9 auditd[682]: Audit daemon is low on disk space for logging
May 17 08:58:23 vm-audit9 auditd[682]: The audit daemon is now halting the system
May 17 08:58:23 vm-audit9 auditd[682]: No plugins found, not dispatching events
May 17 08:58:23 vm-audit9 systemd[1]: Started RPC Bind.
May 17 08:58:23 vm-audit9 auditd[682]: Init complete, auditd 3.0.7 listening for events (startup state enable)
:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
But definitely this message is not visible to admins on the console:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Mounting /var/log/audit...
[ 3.249105] XFS (vdb): Mounting V5 Filesystem
[ 3.274115] XFS (vdb): Ending clean mount
[ OK ] Mounted /var/log/audit.
[ OK ] Reached target Local File Systems.
Starting Create Volatile Files and Directories...
:
[ OK ] Removed slice Slice /system/modprobe.
[ OK ] Removed slice Slice /system/systemd-hibernate-resume.
[ OK ] Stopped target RPC Port Mapper.
[ OK ] Closed LVM2 poll daemon socket.
[ OK ] Closed Process Core Dump Socket.
[ OK ] Started /usr/sbin/lvm vgch…y --autoactivation event rhel.
Stopping Load/Save Random Seed...
[ OK ] Removed slice Slice /system/getty.
[ OK ] Removed slice Slice /system/serial-getty.
[ OK ] Removed slice Slice /system/sshd-keygen.
[ OK ] Stopped Generate network units from Kernel command line.
[ OK ] Stopped target Path Units.
[ OK ] Stopped target Slice Units.
:
[ OK ] Stopped target Local File Systems.
Unmounting /boot...
Unmounting /run/credentials/systemd-sysctl.service...
Unmounting /run/credential…-tmpfiles-setup-dev.service...
[ 3.471501] XFS (vda1): Unmounting Filesystem
Unmounting /var/log/audit...
:
[ 3.641752] reboot: Power down
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
All this makes it *really hard* to troubleshoot, and usually requires Red Hat Support to enter the game, *and them to struggle a lot*.
We need to highlight the operation on the console and probably log the message in the journal as "CRIT" as well.
Version-Release number of selected component (if applicable):
All audit releases
How reproducible:
Always
Steps to Reproduce:
1. Configure auditd to halt the system
# sed -i "s/admin_space_left_action = .*/admin_space_left_action = halt/" /etc/audit/auditd.conf
2. Set up a dedicated mount for /var/log/audit (1GB)
# mkfs.xfs /dev/vdb
# echo "/dev/vdb /var/log/audit xfs defaults 0 0" >> /etc/fstab
# service auditd stop
# systemctl daemon-reload
# mount /var/log/audit
# restorecon -Frv /var/log/audit
# service auditd start
3. Fill the mount point with wreck
# dd if=/dev/random of=/var/log/audit/filler bs=1M count=970
4. Reboot
Actual results:
System goes down during boot
Expected results:
System goes down after printing a *large visible red message* on the console
The audit event is sent to syslog with a priority of ALERT which is higher than CRIT. In testing on F38, logging with LOG_CONS is not honored. I suppose journald does not have ALERT or CONS capabilities? Even as root, no console messages in case capabilities were needed. Writing to /dev/console also does not send a message to the console. The wall command also seems non-functional. Not sure what else can be done. I did make a patch (upstream commit 0b118c7) to shutdown auditd when processing any runlevel change. Maybe that is noticeable?
Description of problem: CIS level 2 hardening requires having "admin_space_left_action = halt" in the configuration. If for some reason the space gets low on /var/log/audit, it becomes impossible to boot the system, since it goes down immediately during the boot. Checking the console output, nowhere we can see an explicit message stating auditd is halting the system. Only a message is recorded in the journal, *assuming the journal is persistent, which is far from being the usual case*: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- May 17 08:58:23 vm-audit9 auditd[682]: Audit daemon is low on disk space for logging May 17 08:58:23 vm-audit9 auditd[682]: The audit daemon is now halting the system May 17 08:58:23 vm-audit9 auditd[682]: No plugins found, not dispatching events May 17 08:58:23 vm-audit9 systemd[1]: Started RPC Bind. May 17 08:58:23 vm-audit9 auditd[682]: Init complete, auditd 3.0.7 listening for events (startup state enable) : -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- But definitely this message is not visible to admins on the console: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Mounting /var/log/audit... [ 3.249105] XFS (vdb): Mounting V5 Filesystem [ 3.274115] XFS (vdb): Ending clean mount [ OK ] Mounted /var/log/audit. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... : [ OK ] Removed slice Slice /system/modprobe. [ OK ] Removed slice Slice /system/systemd-hibernate-resume. [ OK ] Stopped target RPC Port Mapper. [ OK ] Closed LVM2 poll daemon socket. [ OK ] Closed Process Core Dump Socket. [ OK ] Started /usr/sbin/lvm vgch…y --autoactivation event rhel. Stopping Load/Save Random Seed... [ OK ] Removed slice Slice /system/getty. [ OK ] Removed slice Slice /system/serial-getty. [ OK ] Removed slice Slice /system/sshd-keygen. [ OK ] Stopped Generate network units from Kernel command line. [ OK ] Stopped target Path Units. [ OK ] Stopped target Slice Units. : [ OK ] Stopped target Local File Systems. Unmounting /boot... Unmounting /run/credentials/systemd-sysctl.service... Unmounting /run/credential…-tmpfiles-setup-dev.service... [ 3.471501] XFS (vda1): Unmounting Filesystem Unmounting /var/log/audit... : [ 3.641752] reboot: Power down -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- All this makes it *really hard* to troubleshoot, and usually requires Red Hat Support to enter the game, *and them to struggle a lot*. We need to highlight the operation on the console and probably log the message in the journal as "CRIT" as well. Version-Release number of selected component (if applicable): All audit releases How reproducible: Always Steps to Reproduce: 1. Configure auditd to halt the system # sed -i "s/admin_space_left_action = .*/admin_space_left_action = halt/" /etc/audit/auditd.conf 2. Set up a dedicated mount for /var/log/audit (1GB) # mkfs.xfs /dev/vdb # echo "/dev/vdb /var/log/audit xfs defaults 0 0" >> /etc/fstab # service auditd stop # systemctl daemon-reload # mount /var/log/audit # restorecon -Frv /var/log/audit # service auditd start 3. Fill the mount point with wreck # dd if=/dev/random of=/var/log/audit/filler bs=1M count=970 4. Reboot Actual results: System goes down during boot Expected results: System goes down after printing a *large visible red message* on the console