Bug 1565909

Summary:	prevent log record update which causes deletions
Product:	OpenShift Container Platform	Reporter:	Rich Megginson <rmeggins>
Component:	Logging	Assignee:	Jeff Cantrill <jcantril>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, jcantril, nhosoi, pportant, rmeggins
Target Milestone:	---	Keywords:	OpsBlocker
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	logging-fluentd-docker-v3.9.22-2	Doc Type:	Bug Fix
Doc Text:	Cause: The default write operation for fluentd to elasticsearch is index Consequence: Writes can trigger unnecessary 'delete' operations for elasticsearch causing extra load that affects performance Fix: Utilize the 'create' operation Result: Writes to elasticsearch will only create records or skip updates if the records are duplicates reducing the load on the server	Story Points:	---
Clone Of:
Clones:	1568415 (view as bug list)		Environment:
Last Closed:	2018-05-17 06:43:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1568415

Description Rich Megginson 2018-04-11 03:36:04 UTC

Description of problem:
Now that we've switched to adding a unique id to each record, if we do retry, we update the document with that id.  So we don't get duplicates, but instead, we get a lot of deletions.  A document deletion is an expensive operation.

We must change https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/output-es-config.conf and https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/output-es-ops-config.conf to add the parameter `write_operation` to use `create` instead of the default `index`: https://github.com/uken/fluent-plugin-elasticsearch/blob/v0.12/lib/fluent/plugin/out_elasticsearch.rb#L42

We must also have a CI test that can verify
- no duplicates
- no deletions
when the es plugin has to retry.

Version-Release number of selected component (if applicable):
3.10 and backports

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Peter Portante 2018-04-11 03:43:00 UTC

See also https://www.elastic.co/guide/en/elasticsearch/reference/2.0/docs-bulk.html#docs-bulk, which describes in the behavior in the fourth paragraph:

   The possible actions are index, create, delete and update.
   "index" and "create" expect a source on the next line, and
   have the same semantics as the op_type parameter to the
   standard index API (i.e. "create" will fail if a document
   with the same index and type exists already, whereas index
   will add or replace a document as necessary).

Comment 2 Jeff Cantrill 2018-04-11 16:33:10 UTC

https://github.com/openshift/origin-aggregated-logging/pull/1076

Comment 3 openshift-github-bot 2018-04-17 15:34:36 UTC

Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/f2585e6bc099cfcd3a7b3f2325740bd35cd1e22b
bug 1565909. Modify the default operation to be create to avoid deletions

(cherry picked from commit e5ddcdf665bad46dc4a6310a98e9b76aa00fd404)

https://github.com/openshift/origin-aggregated-logging/commit/b1805d3a6fcf49b9a0e237fa6cfcabf3be823c4c
Merge pull request #1077 from jcantrill/1565909_on_master

bug 1565909. Modify the default operation to be create to avoid delet…

Comment 7 Anping Li 2018-05-08 05:24:42 UTC

Regression pass with the fix with 3.7.27. so move to verified.

Comment 10 errata-xmlrpc 2018-05-17 06:43:35 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566

Comment 11 openshift-github-bot 2018-06-20 15:06:15 UTC

Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/a999f873217b16f539d09410684104aa6dca4015
bug 1565909. Modify the default operation to be create to avoid deletions

(cherry picked from commit e5ddcdf665bad46dc4a6310a98e9b76aa00fd404)

https://github.com/openshift/origin-aggregated-logging/commit/6743a2254d1e09d9a3c30f50dbf4de5b7643afc7
Merge pull request #1087 from openshift-cherrypick-robot/cherry-pick-1077-to-es5.x

[es5.x] bug 1565909. Modify the default operation to be create to avoid delet…