Bug 1303617 - [RFE] wildcard filename support [NEEDINFO]
Summary: [RFE] wildcard filename support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rsyslog
Version: 7.2
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Peter Vrabec
QA Contact: Stefan Dordevic
Mirek Jahoda
URL:
Whiteboard:
Depends On:
Blocks: 1203710 1296594 1313485
TreeView+ depends on / blocked
 
Reported: 2016-02-01 13:11 UTC by Rich Megginson
Modified: 2016-11-04 05:37 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
_rsyslog_ imfile module now supports a wildcard file name The _rsyslog_ packages provide an enhanced, multi-threaded syslog daemon. With this update, the _rsyslog_ imfile module supports using wildcards inside file names and adding the actual file name to the message's metadata. This is useful, when _rsyslog_ needs to read logs under a directory and does not know the names of files in advance.
Clone Of:
Environment:
Last Closed: 2016-11-04 05:37:23 UTC
ebarrera: needinfo? (theinric)


Attachments (Terms of Use)
Backport of wildcard support (71.68 KB, patch)
2016-05-03 14:50 UTC, Radovan Sroka
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2401 normal SHIPPED_LIVE rsyslog bug fix and enhancement update 2016-11-03 13:55:50 UTC

Description Rich Megginson 2016-02-01 13:11:36 UTC
Description of problem:

openshift needs to read log files under /var/log/containers - one log per container.  This is difficult to do with rsyslog v7 imfile because you don't know the names of the files in advance, and even if you did, there could be thousands of them that change frequently.  It would be much easier if you could just configure imfile to read from /var/log/containers/*, and put the filename in the message metadata.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html?highlight=wildcard#

v8 (version 8.5.0 and later) with inotify support can have imfile use a wildcard filename and add the actual filename to the metadata.  I don't know if it will be possible to backport this functionality to v7.

Comment 2 Rich Megginson 2016-02-03 14:54:13 UTC
After speaking with Tomas, it may be possible to backport this feature from rsyslog v8 to rsyslog v7 in RHEL 7.3.  I am requesting acks for 7.3.

Comment 8 Radovan Sroka 2016-05-03 14:50:25 UTC
Created attachment 1153459 [details]
Backport of wildcard support

This backport introduce some new features:

Module:
        - option "mode" which can be polling|inotify and it's not required
        - !BUT! there is difference between us and upstream
        - we have as default option "polling" and upstream has "inotify"
        - because we need to preserve backward compatibility

Input:
        - new options: escapeLF, removestateondelete works as upstream 
        - new features: dynamic statefiles, wildcards(inotify mode)
        - statefile is not required now, if it's not given then dynamic statefile is created

Refactoring of internal structures:
        - active file table: from fixed array to double-linked list 
        - using listners as upstream

doc/imfile.html is updated as well.

This patch was derived from upstream commits:
        ea9c81ce
        7ca5b1e5
        c9c6bd47
        68ac87cb
        25baf0e1
        dcd19358
        36f373be
        d11858de
        fd7ab1ee
        f63d5ed4
        e73b8413
        7cf1349a
        e53d6114
        ef9fad05
        e5e7836b
        385fde74
        0f164bf7

---------------------------------------------------------------------

And here's some test cases:
---

#default is polling mode so it's ok

module(load="imfile")
input(type="imfile" tag="tagggg1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" file="/tmp/input_data")

------------------------------------------------------------------
#default is polling so wildcards are not allowed

module(load="imfile")
input(type="imfile" tag="tagggg1" file="/tmp/input_*")

-----------------------------------------------------------------
#it's monitoring all matched files

module(load="imfile" mode="inotify")
input(type="imfile" tag="tagggg1" file="/tmp/input_*")

-----------------------------------------------------------------
#test of polling mode

module(load="imfile" mode="polling")
input(type="imfile" tag="tagggg1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" file="/tmp/input_data")

-----------------------------------------------------------------
#in polling mode wildcards are not allowed

module(load="imfile" mode="polling")
input(type="imfile" tag="tagggg1" file="/tmp/input_*")

-----------------------------------------------------------------
#statefile in inotify mode without wildcards is ok 

module(load="imfile" mode="inotify")
input(type="imfile" tag="tagggg1" statefile="st1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" statefile="st2" file="/tmp/input_data")

-----------------------------------------------------------------
#statefile in polling mode is ok

module(load="imfile" mode="polling")
input(type="imfile" tag="tagggg1" statefile="st1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" statefile="st2" file="/tmp/input_data")

-----------------------------------------------------------------
#statefiles with wildcards are not allowed

module(load="imfile" mode="inotify")
input(type="imfile" tag="tagggg1" statefile="st1" file="/tmp/input_*")

Comment 16 errata-xmlrpc 2016-11-04 05:37:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2401.html


Note You need to log in before you can comment on or make changes to this bug.