Bug 1556897
| Summary: | Duplicate elasticsearch entries increase as namespaces increase (constant message rate) | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jeff Cantrill <jcantril> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED ERRATA | QA Contact: | Mike Fiedler <mifiedle> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.6.1 | CC: | aos-bugs, juzhao, lvlcek, mifiedle, pportant, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 3.6.z | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | aos-scalability-39 | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Fluentd inserts documents (logs) into Elasticsearch using the bulk insert API but relies upon Elasticsearch to generate UUIDs for each document. It does not remove successfully indexed documents from the bulk payload when the bulk operation fails.
Consequence: The initial payload is resubmitted and documents that were successfully indexed are submitted again which will result in duplicate documents with different uuids.
Fix: Generate document ids before submitting bulk insert requests.
Result: Elasticsearch will disregard insert of documents that already exist in the data store and insert documents that do not.
|
Story Points: | --- |
| Clone Of: | 1548104 | Environment: | |
| Last Closed: | 2018-04-12 06:05:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1548104 | ||
| Bug Blocks: | 1556896 | ||
|
Comment 1
Jeff Cantrill
2018-03-15 13:13:57 UTC
Verified on logging-fluentd v3.6.0.73.112. 250 and 500 1K messages/second across 100 namespaces per node. Successful retries occurred regularly during the run, but no duplicate index entries were found at the end of the run. No duplicates and no missing messages 2018-04-10 10:32:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-04-10 10:32:59 -0400 error_class="Fluent::ElasticsearchErrorHandler::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:a139d4" 2018-04-10 10:32:58 -0400 [warn]: suppressed same stacktrace 2018-04-10 10:32:59 -0400 [warn]: retry succeeded. plugin_id="object:a139d4" 2018-04-10 10:33:20 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-04-10 10:33:21 -0400 error_class="Fluent::ElasticsearchErrorHandler::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:a139d4" 2018-04-10 10:33:20 -0400 [warn]: suppressed same stacktrace 2018-04-10 10:33:21 -0400 [warn]: retry succeeded. plugin_id="object:a139d4" " Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1106 |