Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1556897 - Duplicate elasticsearch entries increase as namespaces increase (constant message rate)
Duplicate elasticsearch entries increase as namespaces increase (constant mes...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.6.1
x86_64 Linux
unspecified Severity high
: ---
: 3.6.z
Assigned To: Jeff Cantrill
Mike Fiedler
aos-scalability-39
:
Depends On: 1548104
Blocks: 1556896
  Show dependency tree / graph
 
Reported: 2018-03-15 09:12 EDT by Jeff Cantrill
Modified: 2018-04-12 02:06 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd inserts documents (logs) into Elasticsearch using the bulk insert API but relies upon Elasticsearch to generate UUIDs for each document. It does not remove successfully indexed documents from the bulk payload when the bulk operation fails. Consequence: The initial payload is resubmitted and documents that were successfully indexed are submitted again which will result in duplicate documents with different uuids. Fix: Generate document ids before submitting bulk insert requests. Result: Elasticsearch will disregard insert of documents that already exist in the data store and insert documents that do not.
Story Points: ---
Clone Of: 1548104
Environment:
Last Closed: 2018-04-12 02:05:33 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Github openshift/origin-aggregated-logging/pull/992 None None None 2018-03-15 09:13 EDT
Red Hat Product Errata RHBA-2018:1106 None None None 2018-04-12 02:06 EDT

  None (edit)
Comment 3 Mike Fiedler 2018-04-10 11:49:46 EDT 
Verified on logging-fluentd v3.6.0.73.112.

250 and 500 1K messages/second across 100 namespaces per node.   Successful retries occurred regularly during the run, but no duplicate index entries were found at the end of the run.  No duplicates and no missing messages

2018-04-10 10:32:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-04-10 10:32:59 -0400 error_class="Fluent::ElasticsearchErrorHandler::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:a139d4"
  2018-04-10 10:32:58 -0400 [warn]: suppressed same stacktrace
2018-04-10 10:32:59 -0400 [warn]: retry succeeded. plugin_id="object:a139d4"
2018-04-10 10:33:20 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-04-10 10:33:21 -0400 error_class="Fluent::ElasticsearchErrorHandler::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:a139d4"
  2018-04-10 10:33:20 -0400 [warn]: suppressed same stacktrace
2018-04-10 10:33:21 -0400 [warn]: retry succeeded. plugin_id="object:a139d4"
"
Comment 6 errata-xmlrpc 2018-04-12 02:05:33 EDT 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1106
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.