Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1568415

Summary: prevent log record update which causes deletions
Product: OpenShift Container Platform Reporter: Jeff Cantrill <jcantril>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.1CC: anli, aos-bugs, jcantril, pdwyer, pportant, rmeggins
Target Milestone: ---Keywords: OpsBlocker
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The default write operation for fluentd to elasticsearch is index Consequence: Writes can trigger unnecessary 'delete' operations for elasticsearch causing extra load that affects performance Fix: Utilize the 'create' operation Result: Writes to elasticsearch will only c
 Story Points: ---
Clone Of: 1565909 Environment:
Last Closed: 2018-05-18 03:54:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1565909    
Bug Blocks:    

Description Jeff Cantrill 2018-04-17 13:07:17 UTC 
+++ This bug was initially created as a clone of Bug #1565909 +++

Description of problem:
Now that we've switched to adding a unique id to each record, if we do retry, we update the document with that id.  So we don't get duplicates, but instead, we get a lot of deletions.  A document deletion is an expensive operation.

We must change https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/output-es-config.conf and https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/output-es-ops-config.conf to add the parameter `write_operation` to use `create` instead of the default `index`: https://github.com/uken/fluent-plugin-elasticsearch/blob/v0.12/lib/fluent/plugin/out_elasticsearch.rb#L42

We must also have a CI test that can verify
- no duplicates
- no deletions
when the es plugin has to retry.

Version-Release number of selected component (if applicable):
3.10 and backports

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Peter Portante on 2018-04-10 23:43:00 EDT ---

See also https://www.elastic.co/guide/en/elasticsearch/reference/2.0/docs-bulk.html#docs-bulk, which describes in the behavior in the fourth paragraph:

   The possible actions are index, create, delete and update.
   "index" and "create" expect a source on the next line, and
   have the same semantics as the op_type parameter to the
   standard index API (i.e. "create" will fail if a document
   with the same index and type exists already, whereas index
   will add or replace a document as necessary).

--- Additional comment from Jeff Cantrill on 2018-04-11 12:33:10 EDT ---

https://github.com/openshift/origin-aggregated-logging/pull/1076
Comment 2 Anping Li 2018-05-09 10:34:06 UTC 
The fix is in openshift3/logging-fluentd/images/v3.7.46-1. No regression error found. so move bug verified.
Comment 5 errata-xmlrpc 2018-05-18 03:54:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1576