Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Rsyslog must keep track of which parts of the monitored files are already processed. This is done in so-called "state files" that are created in the rsyslog working directory and are read on startup to resume monitoring after a shutdown. The current design does not delete state files when logs are rotated because some users of rsyslog prefer to move them back later to the same location, and after that these files are not treated as new but processing continues from where it previously ended. On the other hand, others would like to get rid of the state files when rotation takes place. The outcome is going to be a configurable parameter for imfile instances to chose one of the two mentioned options. Due to listed reasons I do not intend to change the default behavior of this module; rather introduce an optional imfile module parameter, deleteStateOnFileMove. If it's set to "on", no redundant state file remains in the system at all.
Description of problem:
imfile plugin doesn't seem to clean up older state files in /var/lib/rsyslog, particularly when the monitored file rotates rapidly (e.g. /var/log/audit/audit.log with many rules).
This pollutes the file system, and, very likely, with time, can be a real issue when rsyslog detects a rotation and searches whether a state file still applies.
Version-Release number of selected component (if applicable):
rsyslog-8.1911.0-6.el8.x86_64
How reproducible:
Always
Steps to Reproduce:
1. Set up tons of audit rules (see "audit.rules" in attachment) and keep 20 audit logs
# cp audit.rules > /etc/audit/rules.d/audit.rules
# sed -i "s/^num_logs = .*/num_logs = 20/" /etc/audit/auditd.conf
2. Monitor audit.log using imfile (see "audit.conf" in attachment)
# cp audit.conf /etc/rsyslog.d/
3. Restart the services (move system to permissive because rsyslog cannot read audit logs by default)
# setenforce 0
# service auditd restart
# systemctl restart rsyslog
4. Start a program generating tons of audit logs
# while :; do find /usr -type f -exec hexdump -n 16 {} \; >/dev/null; done
Actual results:
After 10 minutes or so, we see tons of files in /var/lib/rsyslog:
# while sleep 60; do echo; date; ll /var/lib/rsyslog/ /var/log/audit/; stat -c "%n: %i" /var/log/audit/*; done | tee watch.log
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Mon Dec 21 03:54:36 EST 2020
/var/lib/rsyslog/:
total 1288
-rw-------. 1 root root 108 Dec 21 03:48 imfile-state:134741388:07c931b2f49822db
-rw-------. 1 root root 108 Dec 21 03:44 imfile-state:134741388:07e03d7e9dbdc574
-rw-------. 1 root root 108 Dec 21 03:47 imfile-state:134741388:1806251035a10b80
-rw-------. 1 root root 108 Dec 21 03:50 imfile-state:134741388:1e9276f3285db21f
...
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Expected results:
Only last state file kept for every inode of /var/log/audit/audit.log*, since others do not match (checksum is different and file doesn't exist anymore)
Additional info:
On RHEL7, with rsyslog-8.24.0-57.el7_9, same behaviour is observed but there are far less remaining state files.
Created attachment 1745563[details]
Very simple reproducer
1. Create a dedicated mount point for the test (formatted with XFS preferably, so that inodes get reused)
# mkdir -p /disk2/rsyslog
2. Execute rsyslog with "rsyslog.conf" configuration in ZIP file
3. Execute "logger.sh" generator in ZIP file
4. Wait for 20/30 rotations
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (rsyslog bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2022:7460
Comment 35oralie
2023-01-13 08:38:45 UTC
Comment hidden (spam)
I give this article 5 stars overall and heartily [url=https://slimerancher2.co]slime rancher 2[/url] recommend it. Actually, I'm thinking the same thing. I really like this topic, so I hope you write more about it. I'm grateful.
Description of problem: imfile plugin doesn't seem to clean up older state files in /var/lib/rsyslog, particularly when the monitored file rotates rapidly (e.g. /var/log/audit/audit.log with many rules). This pollutes the file system, and, very likely, with time, can be a real issue when rsyslog detects a rotation and searches whether a state file still applies. Version-Release number of selected component (if applicable): rsyslog-8.1911.0-6.el8.x86_64 How reproducible: Always Steps to Reproduce: 1. Set up tons of audit rules (see "audit.rules" in attachment) and keep 20 audit logs # cp audit.rules > /etc/audit/rules.d/audit.rules # sed -i "s/^num_logs = .*/num_logs = 20/" /etc/audit/auditd.conf 2. Monitor audit.log using imfile (see "audit.conf" in attachment) # cp audit.conf /etc/rsyslog.d/ 3. Restart the services (move system to permissive because rsyslog cannot read audit logs by default) # setenforce 0 # service auditd restart # systemctl restart rsyslog 4. Start a program generating tons of audit logs # while :; do find /usr -type f -exec hexdump -n 16 {} \; >/dev/null; done Actual results: After 10 minutes or so, we see tons of files in /var/lib/rsyslog: # while sleep 60; do echo; date; ll /var/lib/rsyslog/ /var/log/audit/; stat -c "%n: %i" /var/log/audit/*; done | tee watch.log -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Mon Dec 21 03:54:36 EST 2020 /var/lib/rsyslog/: total 1288 -rw-------. 1 root root 108 Dec 21 03:48 imfile-state:134741388:07c931b2f49822db -rw-------. 1 root root 108 Dec 21 03:44 imfile-state:134741388:07e03d7e9dbdc574 -rw-------. 1 root root 108 Dec 21 03:47 imfile-state:134741388:1806251035a10b80 -rw-------. 1 root root 108 Dec 21 03:50 imfile-state:134741388:1e9276f3285db21f ... -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Expected results: Only last state file kept for every inode of /var/log/audit/audit.log*, since others do not match (checksum is different and file doesn't exist anymore) Additional info: On RHEL7, with rsyslog-8.24.0-57.el7_9, same behaviour is observed but there are far less remaining state files.