Description of problem: Now that we've switched to adding a unique id to each record, if we do retry, we update the document with that id. So we don't get duplicates, but instead, we get a lot of deletions. A document deletion is an expensive operation. We must change https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/output-es-config.conf and https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/output-es-ops-config.conf to add the parameter `write_operation` to use `create` instead of the default `index`: https://github.com/uken/fluent-plugin-elasticsearch/blob/v0.12/lib/fluent/plugin/out_elasticsearch.rb#L42 We must also have a CI test that can verify - no duplicates - no deletions when the es plugin has to retry. Version-Release number of selected component (if applicable): 3.10 and backports How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
See also https://www.elastic.co/guide/en/elasticsearch/reference/2.0/docs-bulk.html#docs-bulk, which describes in the behavior in the fourth paragraph: The possible actions are index, create, delete and update. "index" and "create" expect a source on the next line, and have the same semantics as the op_type parameter to the standard index API (i.e. "create" will fail if a document with the same index and type exists already, whereas index will add or replace a document as necessary).
https://github.com/openshift/origin-aggregated-logging/pull/1076
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/f2585e6bc099cfcd3a7b3f2325740bd35cd1e22b bug 1565909. Modify the default operation to be create to avoid deletions (cherry picked from commit e5ddcdf665bad46dc4a6310a98e9b76aa00fd404) https://github.com/openshift/origin-aggregated-logging/commit/b1805d3a6fcf49b9a0e237fa6cfcabf3be823c4c Merge pull request #1077 from jcantrill/1565909_on_master bug 1565909. Modify the default operation to be create to avoid delet…
Regression pass with the fix with 3.7.27. so move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1566
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/a999f873217b16f539d09410684104aa6dca4015 bug 1565909. Modify the default operation to be create to avoid deletions (cherry picked from commit e5ddcdf665bad46dc4a6310a98e9b76aa00fd404) https://github.com/openshift/origin-aggregated-logging/commit/6743a2254d1e09d9a3c30f50dbf4de5b7643afc7 Merge pull request #1087 from openshift-cherrypick-robot/cherry-pick-1077-to-es5.x [es5.x] bug 1565909. Modify the default operation to be create to avoid delet…