Bug 1554885

Summary:	Fluentd logs entire response when unexpected error from Elasticsearch
Product:	OpenShift Container Platform	Reporter:	Rich Megginson <rmeggins>
Component:	Logging	Assignee:	Jeff Cantrill <jcantril>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, pportant, rmeggins
Target Milestone:	---
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Plugin logs the entire response on failure Consequence: Fills up the on disk logs Fix: Only log entire response when in debug mode Result: On disk logs no logger consume the disk	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-05-17 06:42:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rich Megginson 2018-03-13 14:13:59 UTC

Description of problem:
When Fluentd receives an unexpected error from Elasticsearch (as opposed to ElasticsearchOutOfMemory or BulkIndexQueueFull which are "expected"), Fluentd will log the entire response.  This is unhelpful.  All errors should be handled like ElasticsearchOutOfMemory and BulkIndexQueueFull errors.  The user can always turn on debug logging to get more detailed information about errors if the problem is persistent.

Version-Release number of selected component (if applicable):
fluent-plugin-elasticsearch-1.13.2 (and .3)

How reproducible:
When the bulk index request times out or returns some error other than ElasticsearchOutOfMemory or BulkIndexQueueFull.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Peter Portante 2018-03-13 17:17:50 UTC

We saw 23,000+ character log lines in fluentd pods which contain the full dump of the error response from Elasticsearch when there is a timeout creating an index.

If the fluentd plugin only emitted the JSON blobs that actually contain an error from the response payload, that would be helpful. But agree the full payload is pretty useless.

If the fluentd plugin could recognize an index creation timeout, and handle it just like it does bulk request rejected errors, that would also be helpful.

Comment 2 Jeff Cantrill 2018-04-18 15:30:56 UTC

fixed with merge of https://github.com/uken/fluent-plugin-elasticsearch/pull/399 and release of v1.15.0

Comment 6 Anping Li 2018-05-07 06:02:35 UTC

The fluentd works well with the fix with logging:v3.9.27. No regression error found, so move bug to verified.

Comment 9 errata-xmlrpc 2018-05-17 06:42:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566