Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1649250

Summary: rsyslog-8.24.0-34.el7.x86_64 imfile broken with logrotate and tomcat rotated logs
Product: Red Hat Enterprise Linux 7 Reporter: Jarno Huuskonen <jarno.huuskonen>
Component: rsyslogAssignee: Jiří Vymazal <jvymazal>
Status: CLOSED ERRATA QA Contact: Radovan Sroka <rsroka>
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: adam.winberg, dapospis, doug.wussler, fkrska, jvymazal, kompastver, pasik, rmeggins, rsroka, s.buesing, sjayapra, stturner
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rsyslog-8.24.0-35.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 12:48:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1648377, 1649308, 1656860    

Description Jarno Huuskonen 2018-11-13 09:07:03 UTC
Description of problem:
After rhel7 update 6 (rsyslog-8.24.0-34.el7.x86_64) rsyslog imfile stopped working:
- when tomcat rotates for example access_log
(        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="access_log" rotatable="true" renameOnRotate="true"
               requestAttributesEnabled="true" buffered="true"
               fileDateFormat=".yyyyMMdd" 
               pattern="%h %l %u %t &quot;%r&quot; %s %b %D JSESSIONID=%{JSESSIONID}c" />)

- sometimes when logrotate rotates apache access_log (doesn't happen always)

Imfile example config for tomcat:
module(load="imfile" mode="inotify" PollingInterval="20")
input(type="imfile" ruleset="uef_imfile_fwd"
        File="/var/log/tomcat/access_log"
        PersistStateInterval="300"
        Tag="access-hostnamehere"
        Severity="info"
        Facility="local7"
)


Imfile example for apache:
module(load="imfile" mode="inotify" PollingInterval="20")
input(type="imfile" ruleset="uef_imfile_fwd"
        File="/var/log/httpd/access_log"
        PersistStateInterval="300"
        Tag="access-hostnamehere"
        Severity="info"
        Facility="local7"
)

(uef_imfile_fwd ruleset forwards logs to remote server).

Version-Release number of selected component (if applicable):
syslog-8.24.0-34.el7.x86_64

How reproducible:
- Tomcat/java rotated logs: seems to happen after every rotate
- logrotate/apache: happens sometimes

Steps to Reproduce:
1. use rsyslog imfile to monitor tomcat access_log 
2. let tomcat rotate logfile
3. rsyslog stops forwarding logs after 

Actual results:
For example after after apache logrotate, rsyslog was monitoring wrong file:
these are the files from /var/lib/rsyslog/

file: imfile-state:4237564
content:
{ "filename": "\/var\/log\/httpd\/access_log", "prev_was_nl": 0, "curr_offs": 2295391, "strt_offs": 2295391 }
file: imfile-state:4331555
content: { "filename": "\/var\/log\/httpd\/error_log", "prev_was_nl": 0, "curr_offs": 1242, "strt_offs": 1242 }

Actual /var/log/httpd/access_log was already using different inode:
 File: ‘/var/log/httpd/access_log’
  Size: 255753          Blocks: 512        IO Block: 4096   regular file
Device: fd03h/64771d    Inode: 4237513     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:httpd_log_t:s0
Access: 2018-11-11 04:02:02.936520161 +0200
Modify: 2018-11-12 10:09:36.723540441 +0200
Change: 2018-11-12 10:09:36.723540441 +0200

So AFAIK rsyslog(imfile) didn't recognise rotation.

Similar situation with tomcat access_log: /var/lib/rsyslog/imfile-state:164
({ "filename": "\/var\/log\/tomcat\/access_log", "prev_was_nl": 0, "curr_offs": 83250, "strt_offs": 83250 }) while the actual logfile is already using different inode:
  File: ‘/var/log/tomcat/access_log’
  Size: 38850           Blocks: 80         IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 190         Links: 1
Access: (0644/-rw-r--r--)  Uid: (   91/  tomcat)   Gid: (   91/  tomcat)
Context: system_u:object_r:tomcat_log_t:s0
Access: 2018-11-13 10:15:54.450811462 +0200
Modify: 2018-11-13 10:28:30.741826028 +0200
Change: 2018-11-13 10:28:30.741826028 +0200


Expected results:
rsyslog(imfile) continues to monitor/forward logs after logrotation.

Additional info:
Workaround for tomcat access logs:
When using logrotate+copytruncate for tomcat accesslog(+rotatable="false")
(and reloadontruncate="on" with imfile) seems to work.

This sounds like the problem described in rsyslog bug 3051 (https://github.com/rsyslog/rsyslog/pull/3051 / https://github.com/rsyslog/rsyslog/commit/e929ee7220de8a81a8669573ac44526397bb3158)

AFAIK rsyslog-8.24.0-34.el7.x86_64 imfile is missing the above commit.

Comment 4 Jiří Vymazal 2018-11-14 08:00:56 UTC
*** Bug 1649491 has been marked as a duplicate of this bug. ***

Comment 5 Jarno Huuskonen 2018-11-14 08:32:37 UTC
I built rsyslog-8.24.0-34.el7.x86_64 with this quick hack patch (manually copied/edited from https://github.com/rsyslog/rsyslog/commit/e929ee7220de8a81a8669573ac44526397bb3158).
I'm testing this with tomcat access_log and it looks like rsyslog continues to monitor the access_log after inode change(rotation).

So I think backporting the rsyslog pull/3051 should fix this problem.
 
diff -urN rsyslog-8.24.0.orig/plugins/imfile/imfile.c rsyslog-8.24.0/plugins/imfile/imfile.c
--- rsyslog-8.24.0.orig/plugins/imfile/imfile.c 2018-11-13 11:36:57.862001818 +0200
+++ rsyslog-8.24.0/plugins/imfile/imfile.c      2018-11-13 11:36:26.706026617 +0200
@@ -629,8 +629,11 @@
                        act_obj_unlink(act);
                        restart = 1;
                        break;
+               } else if(fileInfo.st_ino != act->ino) {
+                       act_obj_unlink(act);
+                       restart = 1;
+                       break;
                }
-               // TODO: add inode check for change notification!
 
                /* Note: active nodes may get deleted, so we need to do the
                 * pointer advancement at the end of the for loop!

Comment 17 Pavel Znamensky 2019-05-23 13:26:11 UTC
Any news on this?

Comment 18 Doug Wussler 2019-06-06 14:15:55 UTC
Just looking to have my expectations managed.  The Priority and Severity of this issue are both HIGH.  We are approaching 7 months since this issue was opened.  Is the release of the fix imminent?

Comment 19 Dalibor Pospíšil 2019-06-07 13:24:13 UTC
The fix is targetting RHEL-7.7.

Comment 20 Jarno Huuskonen 2019-06-07 15:43:54 UTC
What about RHEL8 rsyslog-8.37.0-9.el8 ? AFAIK 1674471 is tracking RHEL8 rsyslog, any update when rsyslog-8.37.0-10.el8 (for RHEL8) is going to be available ?

Comment 22 errata-xmlrpc 2019-08-06 12:48:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2110