Bug 1551160

Summary:	Aggregated logging stops running in Elastic stack and Kibana stops showing new records
Product:	OpenShift Container Platform	Reporter:	Greg Rodriguez II <grodrigu>
Component:	Logging	Assignee:	Rich Megginson <rmeggins>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.7.0	CC:	aos-bugs, jcantril, rmeggins, stwalter
Target Milestone:	---
Target Release:	3.7.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	logging-fluentd-docker-v3.7.28-1	Doc Type:	Bug Fix
Doc Text:	Cause: The Fluentd Kubernetes plugin was using a local variable that was replacing the value of the global "log" object of the same name. Consequence: Strange errors such as - undefined method `debug' for "{\"outcome\" => \"success\"}" - when the plugin would attempt to log an event. Fix: Use a variable with a different name other than "log". Result: No more strange errors.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-12 06:43:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Greg Rodriguez II 2018-03-02 23:48:24 UTC

Description of problem:
Per Rich Megginson, the following issue has been fixed but no errata has been released for 3.7 that has this fix in it.  He requested I create a bug to address.
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/commit/229630925031a521bb07432c279dbf9f996732d3

Version-Release number of selected component (if applicable):
3.7

How reproducible:
Customer Verified

Steps to Reproduce:
Issue was reported to openshift-sme team and Rich Megginson identified the issue to be the github link located in "Additional info" which has been resolved but no errata for 3.7 has been released with this fix.  He advised I create a bug reporting this issue.
Per Rich:
The last released errata was https://errata.devel.redhat.com/advisory/31889
Which had logging-fluentd-docker-v3.7.23-3 <https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=646027>
which was built with http://download-node-02.eng.bos.redhat.com/brewroot/packages/logging-fluentd-docker/v3.7.23/3/data/logs/x86_64-build.log

---> Package rubygem-fluent-plugin-kubernetes_metadata_filter.noarch 0:0.33.0-1.el7 will be installed - which has the bug.

There have been subsequent 3.7 image updates but none of them had the fluentd image


Actual results:
The following error appears in FluentD repeatedly while issue occurs
[error]: Exception emitting record: undefined method `debug' for "{\"outcome\" => \"success\"}":String

Expected results:
Customer expects Kibana to show records without issue and aggregated logging to run consistently

Additional info:
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/commit/229630925031a521bb07432c279dbf9f996732d3

Comment 2 Jeff Cantrill 2018-03-06 14:18:12 UTC

See my comments here: https://bugzilla.redhat.com/show_bug.cgi?id=1513408#c4

Comment 3 Steven Walter 2018-03-12 16:19:33 UTC

Are there any other workarounds for the issue? Customer tried workaround listed in 1513408's original description, to no avail. If there is no other workaround we will wait for fix.

Comment 4 Anping Li 2018-03-28 05:40:31 UTC

The fix is in openshift3/logging-fluentd/images/v3.7.40-1. And no error message are reported.

Comment 7 errata-xmlrpc 2018-04-12 06:43:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1107