Bug 1554885 - Fluentd logs entire response when unexpected error from Elasticsearch
Summary: Fluentd logs entire response when unexpected error from Elasticsearch
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
Depends On:
TreeView+ depends on / blocked
Reported: 2018-03-13 14:13 UTC by Rich Megginson
Modified: 2018-05-17 06:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Plugin logs the entire response on failure Consequence: Fills up the on disk logs Fix: Only log entire response when in debug mode Result: On disk logs no logger consume the disk
Clone Of:
Last Closed: 2018-05-17 06:42:42 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Github uken fluent-plugin-elasticsearch pull 399 None None None 2018-04-18 15:30:55 UTC
Red Hat Product Errata RHBA-2018:1566 None None None 2018-05-17 06:43:12 UTC

Description Rich Megginson 2018-03-13 14:13:59 UTC
Description of problem:
When Fluentd receives an unexpected error from Elasticsearch (as opposed to ElasticsearchOutOfMemory or BulkIndexQueueFull which are "expected"), Fluentd will log the entire response.  This is unhelpful.  All errors should be handled like ElasticsearchOutOfMemory and BulkIndexQueueFull errors.  The user can always turn on debug logging to get more detailed information about errors if the problem is persistent.

Version-Release number of selected component (if applicable):
fluent-plugin-elasticsearch-1.13.2 (and .3)

How reproducible:
When the bulk index request times out or returns some error other than ElasticsearchOutOfMemory or BulkIndexQueueFull.

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 1 Peter Portante 2018-03-13 17:17:50 UTC
We saw 23,000+ character log lines in fluentd pods which contain the full dump of the error response from Elasticsearch when there is a timeout creating an index.

If the fluentd plugin only emitted the JSON blobs that actually contain an error from the response payload, that would be helpful. But agree the full payload is pretty useless.

If the fluentd plugin could recognize an index creation timeout, and handle it just like it does bulk request rejected errors, that would also be helpful.

Comment 2 Jeff Cantrill 2018-04-18 15:30:56 UTC
fixed with merge of https://github.com/uken/fluent-plugin-elasticsearch/pull/399 and release of v1.15.0

Comment 6 Anping Li 2018-05-07 06:02:35 UTC
The fluentd works well with the fix with logging:v3.9.27. No regression error found, so move bug to verified.

Comment 9 errata-xmlrpc 2018-05-17 06:42:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.