Bug 1313490 - (rsyslog-rhel74-rebase) rebase to rsyslog v8
rebase to rsyslog v8
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rsyslog (Show other bugs)
All Linux
medium Severity medium
: rc
: ---
Assigned To: Radovan Sroka
Stefan Dordevic
Vladimír Slávik
: Rebase, Triaged
Depends On: 1433913 1395145 1399555 1399562 1399569 1399652 1400594 1401870 1403815 1403831 1403858 1403874 1403888 1403907 1410508 1422395 1422414 1431615
Blocks: 1377248 1053641 1056548 1133687 1174345 1326216 1369462 1404805 1410630 1411695 1411798 1412395
  Show dependency treegraph
Reported: 2016-03-01 11:59 EST by Rich Megginson
Modified: 2017-08-02 03:19 EDT (History)
26 users (show)

See Also:
Fixed In Version: rsyslog-8.24.0-6.el7
Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
*rsyslog* rebased to version 8.24.0 The *rsyslog* utility has been rebased to upstream version 8.24.0, which includes numerous enhancements, new features and bug fixes. Notable improvements include: * A new core engine has been implemented, offering faster message processing. * Speed and stability when handling data in the JSON format have been improved. * The RainerScript configuration format has been selected as default and improved with more options. * A new *mmexternal* module for manipulation of messages inside *rsyslog* using external applications has been added. * The *omprog* module has received improvements for better communication with external binaries. * Modules *imrelp* and *omrelp* now support encrypted transmission using the TLS protocol. * The *imuxsock* module now supports rule sets for individual sockets, which override the global rule set. * When the *imuxsock* module is used, rate limiting messages now include PID of the process that causes the rate limiting. * The TCP server error messages now include the IP address of the remote host. * The *imjournal* module no longer stops receiving logs after switching to the persistent *journald* configuration. * Logging to the runtime journal no longer completely stops after a reboot when the machine's clock was set to an earlier time. * Previously, when the *logrotate* utility with "copytruncate" option was rotating a log file, the *imfile* module might not have read all of the log messages from the file being rotated. As a consequence, these log messages were lost. The *imfile* module has been extended to handle this situation. As a consquence, messages are no longer lost when *logrotate* "copytruncate" is used on log files. Customers using custom modules are advised to update their modules for the current rsyslog version. See also the Deprecated Functionality chapter for information about deprecated *rsyslog* options.
Story Points: ---
Clone Of:
Last Closed: 2017-08-01 05:53:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Rich Megginson 2016-03-01 11:59:35 EST
Description of problem:
There are a few issues open against rsyslog in rhel7 that are fixed upstream in rsyslog v8.  In addition, there are a number of features in rsyslog v8 that common logging would like to take advantage of, and help with migrations from fluentd to rsyslog.  Perhaps it would be better to just rebase rsyslog to v8 for RHEL 7.3.

Peter Portante has been running rsyslog v8 ported to run on EL7 (centos + epel7).  His build is based on the rsyslog v8 package in fedora 23.  He is also adding a couple of plugins from upstream rsyslog.

However, there are some other dependencies (packages) that we will need to add to RHEL7 (or include privately in the rsyslogv8 package).  Here is the result of building the fedora 24 package on rhel 7.3 candidate: https://brewweb.devel.redhat.com/taskinfo?taskID=10581812

We are missing the following packages:liblogging-stdlog-devel liblognorm-devel librabbitmq-devel

Here is the result of building Peter P.'s latest srpm on rhel 7.3 candidate: https://brewweb.devel.redhat.com/taskinfo?taskID=10581448

There are other dependencies in addition to the above: adiscon-librdkafka-devel grok grok-devel libgt-devel libksi-devel
These are due to the additional rsyslog v8 modules/plugins: mmfields pmciscoios mmutf8fix rsgtutil kafka ksi mmgrok

The following are available in EPEL7: librabbitmq-devel 

Note that Peter P.'s copr repo has all of the dependencies not already in EPEL7, already ported to build on EL7: https://copr.fedorainfracloud.org/coprs/portante/rsyslog-v8.15/packages/
Comment 1 Rich Megginson 2016-03-01 12:04:34 EST
In addition, we will need the omamqp1 (and imamqp1, tbd) plugins available.  These also have a dependency on qpid-proton-c which is in EPEL7. The current proposal is that the amqp1 plugins for rhel7 will include qpid-proton in a private location unavailable for use outside of the plugin (e.g. compiled and linked as a static library, or shared libraries in a private location).

Note that the rabbit module is problematic since librabbitmq is in EPEL7, so we can't just put it into RHEL7 without figuring out how to deprecate it in EPEL7.  We could also include it in the rsyslog package and hide it as we are doing with qpid-proton.
Comment 2 Rich Megginson 2016-03-01 12:06:16 EST
The amqp1 plugins will be in a separate package, so we don't have to worry about those in the base package.
Comment 3 Radovan Sroka 2016-03-01 13:15:52 EST

I've created copr repo with rsyslog8:

Comment 4 Radovan Sroka 2016-03-01 13:33:58 EST

I've just tested rsyslog8
and here is difference between rsyslog7 and rsyslog8

RHEL7 Rsyslog7:

RHEL8 Rsyslog8:
Comment 5 Radovan Sroka 2016-03-01 13:35:28 EST
(In reply to Radovan Sroka from comment #4)
> Hi, 
> I've just tested rsyslog8
> and here is difference between rsyslog7 and rsyslog8
> RHEL7 Rsyslog7:
> https://beaker.engineering.redhat.com/jobs/1244931
> RHEL8 Rsyslog8:
> https://beaker.engineering.redhat.com/jobs/1242551

RHEL7 --^

Comment 6 Peter Portante 2016-03-01 15:03:41 EST
And please consider that one of the more recent changes for 8.16 and now required in 8.17 of rsyslog is the use of the libfastjson library [1].

The json-c library was found to be too buggy when rsyslog was used at scale in production.  The libfastjson library fixes the problems with json-c.

[1] https://github.com/rsyslog/libfastjson
Comment 7 Radovan Sroka 2016-03-01 15:38:52 EST
Hi Peter,

I understand what json-c does and I will rebase rsyslog8 in copr repo but we must wait until new version of rsyslog will be released (2016-03-08), then I will rebase fedora repo too.
Comment 10 Peter Portante 2016-03-02 06:22:36 EST
There is a leap year bug with v8.16+: http://lists.adiscon.net/pipermail/rsyslog/2016-March/042159.html
Comment 11 Peter Portante 2016-03-02 06:36:54 EST
I filed https://bugzilla.redhat.com/show_bug.cgi?id=1313804
Comment 12 Rich Megginson 2016-03-29 13:22:57 EDT
Peter Portante is now using rsyslog 8.17: https://copr.fedorainfracloud.org/coprs/portante/rsyslog-8.17/

I've ported the omamqp1 to this version (basically, just use upstream rsyslog 8.17 since it is included) and added support for the kubernetes work: https://github.com/richm/docker-rsyslog-collector/tree/rsyslog-v8/rpms

I've created a copr with the omamqp1 and mmkubernetes plugins: https://copr.fedorainfracloud.org/coprs/rmeggins/rsyslogv8-plugins/
Comment 14 Sreejith Anujan 2016-08-17 14:13:18 EDT
With the current rsyslog container, if one needs to do granular log analysis, the file input module in polling mode gives only 10sec granulariy.
With rsyslog version 8.2 and above, inotify mode could be used with file
input module, which can feed the messages immediately into syslog.
Comment 37 errata-xmlrpc 2017-08-01 05:53:51 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.