RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1649250 - rsyslog-8.24.0-34.el7.x86_64 imfile broken with logrotate and tomcat rotated logs
Summary: rsyslog-8.24.0-34.el7.x86_64 imfile broken with logrotate and tomcat rotated ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rsyslog
Version: 7.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Jiří Vymazal
QA Contact: Radovan Sroka
URL:
Whiteboard:
: 1649491 (view as bug list)
Depends On:
Blocks: 1648377 1649308 1656860
TreeView+ depends on / blocked
 
Reported: 2018-11-13 09:07 UTC by Jarno Huuskonen
Modified: 2022-03-13 16:03 UTC (History)
12 users (show)

Fixed In Version: rsyslog-8.24.0-35.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-06 12:48:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github rsyslog rsyslog pull 3036 0 'None' closed imfile: improve truncation detection 2021-01-25 06:15:26 UTC
Github rsyslog rsyslog pull 3051 0 'None' closed bugfix imfile: file change was not reliably detected 2021-01-25 06:15:26 UTC
Github rsyslog rsyslog pull 3307 0 'None' closed disable file vs directory error on symlinks 2021-01-25 06:15:26 UTC
Red Hat Bugzilla 1133687 0 high CLOSED File Input Module(imfile) does not continue logging when service is logrotated with the 'copytruncate' option 2023-09-28 04:03:14 UTC
Red Hat Bugzilla 1531295 0 urgent CLOSED [RFE]: rsyslog imfile support for symlink wildcard combination 2021-06-10 14:06:10 UTC
Red Hat Bugzilla 1674471 0 medium CLOSED rsyslog imfile fails to detect log rotation 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHSA-2019:2110 0 None None None 2019-08-06 12:48:36 UTC

Internal Links: 1531295 1674471 1762035

Description Jarno Huuskonen 2018-11-13 09:07:03 UTC
Description of problem:
After rhel7 update 6 (rsyslog-8.24.0-34.el7.x86_64) rsyslog imfile stopped working:
- when tomcat rotates for example access_log
(        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="access_log" rotatable="true" renameOnRotate="true"
               requestAttributesEnabled="true" buffered="true"
               fileDateFormat=".yyyyMMdd" 
               pattern="%h %l %u %t &quot;%r&quot; %s %b %D JSESSIONID=%{JSESSIONID}c" />)

- sometimes when logrotate rotates apache access_log (doesn't happen always)

Imfile example config for tomcat:
module(load="imfile" mode="inotify" PollingInterval="20")
input(type="imfile" ruleset="uef_imfile_fwd"
        File="/var/log/tomcat/access_log"
        PersistStateInterval="300"
        Tag="access-hostnamehere"
        Severity="info"
        Facility="local7"
)


Imfile example for apache:
module(load="imfile" mode="inotify" PollingInterval="20")
input(type="imfile" ruleset="uef_imfile_fwd"
        File="/var/log/httpd/access_log"
        PersistStateInterval="300"
        Tag="access-hostnamehere"
        Severity="info"
        Facility="local7"
)

(uef_imfile_fwd ruleset forwards logs to remote server).

Version-Release number of selected component (if applicable):
syslog-8.24.0-34.el7.x86_64

How reproducible:
- Tomcat/java rotated logs: seems to happen after every rotate
- logrotate/apache: happens sometimes

Steps to Reproduce:
1. use rsyslog imfile to monitor tomcat access_log 
2. let tomcat rotate logfile
3. rsyslog stops forwarding logs after 

Actual results:
For example after after apache logrotate, rsyslog was monitoring wrong file:
these are the files from /var/lib/rsyslog/

file: imfile-state:4237564
content:
{ "filename": "\/var\/log\/httpd\/access_log", "prev_was_nl": 0, "curr_offs": 2295391, "strt_offs": 2295391 }
file: imfile-state:4331555
content: { "filename": "\/var\/log\/httpd\/error_log", "prev_was_nl": 0, "curr_offs": 1242, "strt_offs": 1242 }

Actual /var/log/httpd/access_log was already using different inode:
 File: ‘/var/log/httpd/access_log’
  Size: 255753          Blocks: 512        IO Block: 4096   regular file
Device: fd03h/64771d    Inode: 4237513     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:httpd_log_t:s0
Access: 2018-11-11 04:02:02.936520161 +0200
Modify: 2018-11-12 10:09:36.723540441 +0200
Change: 2018-11-12 10:09:36.723540441 +0200

So AFAIK rsyslog(imfile) didn't recognise rotation.

Similar situation with tomcat access_log: /var/lib/rsyslog/imfile-state:164
({ "filename": "\/var\/log\/tomcat\/access_log", "prev_was_nl": 0, "curr_offs": 83250, "strt_offs": 83250 }) while the actual logfile is already using different inode:
  File: ‘/var/log/tomcat/access_log’
  Size: 38850           Blocks: 80         IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 190         Links: 1
Access: (0644/-rw-r--r--)  Uid: (   91/  tomcat)   Gid: (   91/  tomcat)
Context: system_u:object_r:tomcat_log_t:s0
Access: 2018-11-13 10:15:54.450811462 +0200
Modify: 2018-11-13 10:28:30.741826028 +0200
Change: 2018-11-13 10:28:30.741826028 +0200


Expected results:
rsyslog(imfile) continues to monitor/forward logs after logrotation.

Additional info:
Workaround for tomcat access logs:
When using logrotate+copytruncate for tomcat accesslog(+rotatable="false")
(and reloadontruncate="on" with imfile) seems to work.

This sounds like the problem described in rsyslog bug 3051 (https://github.com/rsyslog/rsyslog/pull/3051 / https://github.com/rsyslog/rsyslog/commit/e929ee7220de8a81a8669573ac44526397bb3158)

AFAIK rsyslog-8.24.0-34.el7.x86_64 imfile is missing the above commit.

Comment 4 Jiří Vymazal 2018-11-14 08:00:56 UTC
*** Bug 1649491 has been marked as a duplicate of this bug. ***

Comment 5 Jarno Huuskonen 2018-11-14 08:32:37 UTC
I built rsyslog-8.24.0-34.el7.x86_64 with this quick hack patch (manually copied/edited from https://github.com/rsyslog/rsyslog/commit/e929ee7220de8a81a8669573ac44526397bb3158).
I'm testing this with tomcat access_log and it looks like rsyslog continues to monitor the access_log after inode change(rotation).

So I think backporting the rsyslog pull/3051 should fix this problem.
 
diff -urN rsyslog-8.24.0.orig/plugins/imfile/imfile.c rsyslog-8.24.0/plugins/imfile/imfile.c
--- rsyslog-8.24.0.orig/plugins/imfile/imfile.c 2018-11-13 11:36:57.862001818 +0200
+++ rsyslog-8.24.0/plugins/imfile/imfile.c      2018-11-13 11:36:26.706026617 +0200
@@ -629,8 +629,11 @@
                        act_obj_unlink(act);
                        restart = 1;
                        break;
+               } else if(fileInfo.st_ino != act->ino) {
+                       act_obj_unlink(act);
+                       restart = 1;
+                       break;
                }
-               // TODO: add inode check for change notification!
 
                /* Note: active nodes may get deleted, so we need to do the
                 * pointer advancement at the end of the for loop!

Comment 17 Pavel Znamensky 2019-05-23 13:26:11 UTC
Any news on this?

Comment 18 Doug Wussler 2019-06-06 14:15:55 UTC
Just looking to have my expectations managed.  The Priority and Severity of this issue are both HIGH.  We are approaching 7 months since this issue was opened.  Is the release of the fix imminent?

Comment 19 Dalibor Pospíšil 2019-06-07 13:24:13 UTC
The fix is targetting RHEL-7.7.

Comment 20 Jarno Huuskonen 2019-06-07 15:43:54 UTC
What about RHEL8 rsyslog-8.37.0-9.el8 ? AFAIK 1674471 is tracking RHEL8 rsyslog, any update when rsyslog-8.37.0-10.el8 (for RHEL8) is going to be available ?

Comment 22 errata-xmlrpc 2019-08-06 12:48:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2110


Note You need to log in before you can comment on or make changes to this bug.