Bug 1551160 - Aggregated logging stops running in Elastic stack and Kibana stops showing new records
Summary: Aggregated logging stops running in Elastic stack and Kibana stops showing ne...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.z
Assignee: Rich Megginson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-02 23:48 UTC by Greg Rodriguez II
Modified: 2021-06-10 15:02 UTC (History)
4 users (show)

Fixed In Version: logging-fluentd-docker-v3.7.28-1
Doc Type: Bug Fix
Doc Text:
Cause: The Fluentd Kubernetes plugin was using a local variable that was replacing the value of the global "log" object of the same name. Consequence: Strange errors such as - undefined method `debug' for "{\"outcome\" => \"success\"}" - when the plugin would attempt to log an event. Fix: Use a variable with a different name other than "log". Result: No more strange errors.
Clone Of:
Environment:
Last Closed: 2018-04-12 06:43:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1107 0 None None None 2018-04-12 06:43:28 UTC

Description Greg Rodriguez II 2018-03-02 23:48:24 UTC
Description of problem:
Per Rich Megginson, the following issue has been fixed but no errata has been released for 3.7 that has this fix in it.  He requested I create a bug to address.
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/commit/229630925031a521bb07432c279dbf9f996732d3

Version-Release number of selected component (if applicable):
3.7

How reproducible:
Customer Verified

Steps to Reproduce:
Issue was reported to openshift-sme team and Rich Megginson identified the issue to be the github link located in "Additional info" which has been resolved but no errata for 3.7 has been released with this fix.  He advised I create a bug reporting this issue.
Per Rich:
The last released errata was https://errata.devel.redhat.com/advisory/31889
Which had logging-fluentd-docker-v3.7.23-3 <https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=646027>
which was built with http://download-node-02.eng.bos.redhat.com/brewroot/packages/logging-fluentd-docker/v3.7.23/3/data/logs/x86_64-build.log

---> Package rubygem-fluent-plugin-kubernetes_metadata_filter.noarch 0:0.33.0-1.el7 will be installed - which has the bug.

There have been subsequent 3.7 image updates but none of them had the fluentd image


Actual results:
The following error appears in FluentD repeatedly while issue occurs
[error]: Exception emitting record: undefined method `debug' for "{\"outcome\" => \"success\"}":String

Expected results:
Customer expects Kibana to show records without issue and aggregated logging to run consistently

Additional info:
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/commit/229630925031a521bb07432c279dbf9f996732d3

Comment 2 Jeff Cantrill 2018-03-06 14:18:12 UTC
See my comments here: https://bugzilla.redhat.com/show_bug.cgi?id=1513408#c4

Comment 3 Steven Walter 2018-03-12 16:19:33 UTC
Are there any other workarounds for the issue? Customer tried workaround listed in 1513408's original description, to no avail. If there is no other workaround we will wait for fix.

Comment 4 Anping Li 2018-03-28 05:40:31 UTC
The fix is in openshift3/logging-fluentd/images/v3.7.40-1. And no error message are reported.

Comment 7 errata-xmlrpc 2018-04-12 06:43:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1107


Note You need to log in before you can comment on or make changes to this bug.