Bug 1556897 - Duplicate elasticsearch entries increase as namespaces increase (constant message rate)
Summary: Duplicate elasticsearch entries increase as namespaces increase (constant mes...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.6.z
Assignee: Jeff Cantrill
QA Contact: Mike Fiedler
URL:
Whiteboard: aos-scalability-39
Depends On: 1548104
Blocks: 1556896
TreeView+ depends on / blocked
 
Reported: 2018-03-15 13:12 UTC by Jeff Cantrill
Modified: 2018-04-12 06:06 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd inserts documents (logs) into Elasticsearch using the bulk insert API but relies upon Elasticsearch to generate UUIDs for each document. It does not remove successfully indexed documents from the bulk payload when the bulk operation fails. Consequence: The initial payload is resubmitted and documents that were successfully indexed are submitted again which will result in duplicate documents with different uuids. Fix: Generate document ids before submitting bulk insert requests. Result: Elasticsearch will disregard insert of documents that already exist in the data store and insert documents that do not.
Clone Of: 1548104
Environment:
Last Closed: 2018-04-12 06:05:33 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 992 0 None None None 2018-03-15 13:13:57 UTC
Red Hat Product Errata RHBA-2018:1106 0 None None None 2018-04-12 06:06:03 UTC

Comment 1 Jeff Cantrill 2018-03-15 13:13:57 UTC 
https://github.com/openshift/origin-aggregated-logging/pull/992
Comment 3 Mike Fiedler 2018-04-10 15:49:46 UTC 
Verified on logging-fluentd v3.6.0.73.112.

250 and 500 1K messages/second across 100 namespaces per node.   Successful retries occurred regularly during the run, but no duplicate index entries were found at the end of the run.  No duplicates and no missing messages

2018-04-10 10:32:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-04-10 10:32:59 -0400 error_class="Fluent::ElasticsearchErrorHandler::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:a139d4"
  2018-04-10 10:32:58 -0400 [warn]: suppressed same stacktrace
2018-04-10 10:32:59 -0400 [warn]: retry succeeded. plugin_id="object:a139d4"
2018-04-10 10:33:20 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-04-10 10:33:21 -0400 error_class="Fluent::ElasticsearchErrorHandler::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:a139d4"
  2018-04-10 10:33:20 -0400 [warn]: suppressed same stacktrace
2018-04-10 10:33:21 -0400 [warn]: retry succeeded. plugin_id="object:a139d4"
"
Comment 6 errata-xmlrpc 2018-04-12 06:05:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1106
Note You need to log in before you can comment on or make changes to this bug.