Description of problem: This looks the same as: https://bugzilla.redhat.com/show_bug.cgi?id=1548104 It looks like the elasticsearch_genid work is not present in running pods. See below for the configmap and directory listing for the fluentd pod Version-Release number of selected component (if applicable): 3.10.0-0.32.0 How reproducible: Always in the 3.10 scale lab. 700 pods logging 1 message/second each results in many duplicates @jcantril is aware and is working to synch the OCP images Actual results: Duplicate index entries when bulk index full retries occur Expected results: Successful retries of bulk index full errors with no duplicates Additional info: listing of /etc/fluent/configs.d/openshift: README filter-exclude-journal-debug.conf filter-k8s-meta.conf filter-k8s-record-transform.conf filter-kibana-transform.conf filter-post-z-retag-one.conf filter-pre-a-audit-exclude.conf filter-pre-force-utf8.conf filter-retag-journal.conf filter-syslog-record-transform.conf filter-viaq-data-model.conf input-pre-audit-log.conf input-pre-systemd.conf output-applications.conf output-es-config.conf output-es-ops-config.conf output-operations.conf system.conf fluentd configmap: apiVersion: v1 data: fluent.conf: | # This file is the fluentd configuration entrypoint. Edit with care. @include configs.d/openshift/system.conf # In each section below, pre- and post- includes don't include anything initially; # they exist to enable future additions to openshift conf as needed. ## sources ## ordered so that syslog always runs last... @include configs.d/openshift/input-pre-*.conf @include configs.d/dynamic/input-docker-*.conf @include configs.d/dynamic/input-syslog-*.conf @include configs.d/openshift/input-post-*.conf ## <label @INGRESS> ## filters @include configs.d/openshift/filter-pre-*.conf @include configs.d/openshift/filter-retag-journal.conf @include configs.d/openshift/filter-k8s-meta.conf @include configs.d/openshift/filter-kibana-transform.conf @include configs.d/openshift/filter-k8s-flatten-hash.conf @include configs.d/openshift/filter-k8s-record-transform.conf @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-viaq-data-model.conf @include configs.d/openshift/filter-post-*.conf ## </label> <label @OUTPUT> ## matches @include configs.d/openshift/output-pre-*.conf @include configs.d/openshift/output-operations.conf @include configs.d/openshift/output-applications.conf # no post - applications.conf matches everything left ## </label> secure-forward.conf: | # <store> # @type secure_forward # self_hostname ${hostname} # shared_key <SECRET_STRING> # secure yes # enable_strict_verification yes # ca_cert_path /etc/fluent/keys/your_ca_cert # ca_private_key_path /etc/fluent/keys/your_private_key # for private CA secret key # ca_private_key_passphrase passphrase # <server> # or IP # host server.fqdn.example.com # port 24284 # </server> # <server> # ip address to connect # host 203.0.113.8 # specify hostlabel for FQDN verification if ipaddress is used for host # hostlabel server.fqdn.example.com # </server> # </store> throttle-config.yaml: | # Logging example fluentd throttling config file #example-project: # read_lines_limit: 10 # #.operations: # read_lines_limit: 100 kind: ConfigMap
Verified on 3.10.0-0.41.0 - filter-post-genid.conf is present. No issue with duplicate message in the scale lab cluster.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816