Bug 1303617

Summary: [RFE] wildcard filename support
Product: Red Hat Enterprise Linux 7 Reporter: Rich Megginson <rmeggins>
Component: rsyslogAssignee: Peter Vrabec <pvrabec>
Status: CLOSED ERRATA QA Contact: Stefan Dordevic <sdordevi>
Severity: unspecified Docs Contact: Mirek Jahoda <mjahoda>
Priority: high    
Version: 7.2CC: bressers, ebarrera, jsantos, jvymazal, lmiksik, pvrabec, rsawhill, rsroka, santony, sdordevi, vanhoof
Target Milestone: rcKeywords: FutureFeature, Patch
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
_rsyslog_ imfile module now supports a wildcard file name The _rsyslog_ packages provide an enhanced, multi-threaded syslog daemon. With this update, the _rsyslog_ imfile module supports using wildcards inside file names and adding the actual file name to the message's metadata. This is useful, when _rsyslog_ needs to read logs under a directory and does not know the names of files in advance.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 05:37:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1203710, 1296594, 1313485    
Attachments:
Description Flags
Backport of wildcard support none

Description Rich Megginson 2016-02-01 13:11:36 UTC 
Description of problem:

openshift needs to read log files under /var/log/containers - one log per container.  This is difficult to do with rsyslog v7 imfile because you don't know the names of the files in advance, and even if you did, there could be thousands of them that change frequently.  It would be much easier if you could just configure imfile to read from /var/log/containers/*, and put the filename in the message metadata.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html?highlight=wildcard#

v8 (version 8.5.0 and later) with inotify support can have imfile use a wildcard filename and add the actual filename to the metadata.  I don't know if it will be possible to backport this functionality to v7.
Comment 2 Rich Megginson 2016-02-03 14:54:13 UTC 
After speaking with Tomas, it may be possible to backport this feature from rsyslog v8 to rsyslog v7 in RHEL 7.3.  I am requesting acks for 7.3.
Comment 8 Radovan Sroka 2016-05-03 14:50:25 UTC 
Created attachment 1153459 [details]
Backport of wildcard support

This backport introduce some new features:

Module:
        - option "mode" which can be polling|inotify and it's not required
        - !BUT! there is difference between us and upstream
        - we have as default option "polling" and upstream has "inotify"
        - because we need to preserve backward compatibility

Input:
        - new options: escapeLF, removestateondelete works as upstream 
        - new features: dynamic statefiles, wildcards(inotify mode)
        - statefile is not required now, if it's not given then dynamic statefile is created

Refactoring of internal structures:
        - active file table: from fixed array to double-linked list 
        - using listners as upstream

doc/imfile.html is updated as well.

This patch was derived from upstream commits:
        ea9c81ce
        7ca5b1e5
        c9c6bd47
        68ac87cb
        25baf0e1
        dcd19358
        36f373be
        d11858de
        fd7ab1ee
        f63d5ed4
        e73b8413
        7cf1349a
        e53d6114
        ef9fad05
        e5e7836b
        385fde74
        0f164bf7

---------------------------------------------------------------------

And here's some test cases:
---

#default is polling mode so it's ok

module(load="imfile")
input(type="imfile" tag="tagggg1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" file="/tmp/input_data")

------------------------------------------------------------------
#default is polling so wildcards are not allowed

module(load="imfile")
input(type="imfile" tag="tagggg1" file="/tmp/input_*")

-----------------------------------------------------------------
#it's monitoring all matched files

module(load="imfile" mode="inotify")
input(type="imfile" tag="tagggg1" file="/tmp/input_*")

-----------------------------------------------------------------
#test of polling mode

module(load="imfile" mode="polling")
input(type="imfile" tag="tagggg1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" file="/tmp/input_data")

-----------------------------------------------------------------
#in polling mode wildcards are not allowed

module(load="imfile" mode="polling")
input(type="imfile" tag="tagggg1" file="/tmp/input_*")

-----------------------------------------------------------------
#statefile in inotify mode without wildcards is ok 

module(load="imfile" mode="inotify")
input(type="imfile" tag="tagggg1" statefile="st1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" statefile="st2" file="/tmp/input_data")

-----------------------------------------------------------------
#statefile in polling mode is ok

module(load="imfile" mode="polling")
input(type="imfile" tag="tagggg1" statefile="st1" file="/tmp/input_file")
input(type="imfile" tag="tagggg2" statefile="st2" file="/tmp/input_data")

-----------------------------------------------------------------
#statefiles with wildcards are not allowed

module(load="imfile" mode="inotify")
input(type="imfile" tag="tagggg1" statefile="st1" file="/tmp/input_*")
Comment 16 errata-xmlrpc 2016-11-04 05:37:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2401.html