Encountered another case of "invalid byte sequence in UTF-8" in fluentd causing the queue to block entirely. This was 3.9 deployed in a starter cluster. 2018-03-29 03:56:58 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-03-29 03:56:58 +0000 error_class="L" error="invalid byte sequence in UTF-8" plugin_id="object:3fec5959ab5c" 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `downcase' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `block in write_objects' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `each' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `block in msgpack_each' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/plugin/buf_file.rb:71:in `open' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:120:in `msgpack_each' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:319:in `write_objects' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:490:in `write' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:354:in `write_chunk' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:333:in `pop' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:342:in `try_flush' 2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:149:in `run'
We have another manifestation of this error: 2018-03-29 21:55:15 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-03-29 22:00:13 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"error\":{\"root_cause\":[{\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: [B@5d150508; line: 1, column: 89]\"}],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: [B@5d150508; line: 1, column: 89]\"},\"status\":500}" plugin_id="object:3fed0f088aa4" 2018-03-29 21:55:15 +0000 [warn]: suppressed same stacktrace This also effectively stops the queue entirely.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1511116
@pportant, could you please share these outputs? Thanks. oc rsh $FLUENTD_POD -- env | egrep ENABLE_UTF8_FILTER oc rsh $FLUENTD_POD -- cat /etc/fluent/configs.d/openshift/filter-pre-force-utf8.conf
ENABLE_UTF8_FILTER environment variable does not exist, and /etc/fluent/configs.d/openshift/filter-pre-force-utf8.conf is an empty file.
(In reply to Peter Portante from comment #6) > ENABLE_UTF8_FILTER environment variable does not exist, and > /etc/fluent/configs.d/openshift/filter-pre-force-utf8.conf is an empty file. Thanks, @Peter. That explains the "invalid byte sequence in UTF-8" error. The utf8 conversion filter is disabled, by default, due to the performance regression caused by the filter. https://bugzilla.redhat.com/show_bug.cgi?id=1513254 The utf8 filter causes the peformance to be cut in half. Disable it by default. If you need to use it, and can pay the performance penalty, oc set env ds/logging-fluentd ENABLE_UTF8_FILTER=true Unfortunately, we have to choose either performance degradation or no utf8 error... :( I think @Jeff is working on a tool/plugin to work around the queue blocking problem. So, once it's implemented, although we have to live with the utf8 error, it won't freeze the queue.
I don't believe this is related to encoding specifically but some sort of corrupt buffer. I am unable to determine the source of the corruption. Unpacking the buffers and applying similar logic used by the elasticsearch plugin shows some of the unpacked messages either do not include valid json or are badly encoded. The attached PR provides a utility by which the buffers can be manually sanitized. We need to additionally, provide a procedure for when and how to use, and also modify the elasticsearch plugin to skip bad records
Fixed with fluent-plugin-elasticsearch v0.14.0
The issue still can be reproduced in openshift3/logging-fluentd/images/v3.9.25-1 fluent-plugin-elasticsearch (1.13.2) FLUENTD_VERSION=0.12.42 DATA_VERSION=1.6.0 VERBOSE=true RUBY_VERSION=2.0 2018-04-23 05:02:10 -0400 [info]: fluent/supervisor.rb:471:read_config: reading config file path="/etc/fluent/fluent.conf" 2018-04-23 05:02:11 -0400 [warn]: fluent/buffer.rb:164:configure: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not 2018-04-23 05:02:11 -0400 [warn]: plugin/in_tail.rb:185:block (2 levels) in expand_paths: /var/log/containers/logging-fluentd-m9fjn_logging_fluentd-elasticsearch-c50f3ac46c9de16f083fa0a05f9eff3983ee6f1f905a8540f3e1fceff7d0861f.log unreadable. It is excluded and would be examined next time. 2018-04-23 05:02:12 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:12 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734" 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `downcase' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `block in write_objects' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `each' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `block in msgpack_each' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/plugin/buf_file.rb:71:in `open' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:120:in `msgpack_each' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:319:in `write_objects' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:490:in `write' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:354:in `write_chunk' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:333:in `pop' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:342:in `try_flush' 2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:149:in `run' 2018-04-23 05:02:13 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:15 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734" 2018-04-23 05:02:13 -0400 [warn]: suppressed same stacktrace 2018-04-23 05:02:15 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:18 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734" 2018-04-23 05:02:15 -0400 [warn]: suppressed same stacktrace 2018-04-23 05:02:18 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:25 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734" 2018-04-23 05:02:18 -0400 [warn]: suppressed same stacktrace 2018-04-23 05:02:25 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:39 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734" 2018-04-23 05:02:25 -0400 [warn]: suppressed same stacktrace
You may need to wait for the nextbuild. The required plugin exists in the puddle: [1]http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.9/latest/x86_64/os/Packages/rubygem-fluent-plugin-elasticsearch-1.15.0-1.el7.noarch.rpm
This is not the latest code which provides additional improvements, but it does include the changes to drop corrupt messages. Are you seeing a 'stuck' fluentd cycling over the same buffer or simply an error message? This looks to be an expected error from a corrupt message that is sent through the error pipeline instead of trying to push it into elastic.
@jeff, Extracted the attached buffer in /var/lib/fluentd, all buffer files are stuck in /var/lib/fluentd.
Created attachment 1430467 [details] stuck fluentd logs and configuration
Actually, This is still functioning like previously because we need to release v1.16.0. The issue is: * We dlq the bad records in the chunk * We submit records to ES * ES returns a bulk failure * We recoginize the failure and throw * The chunk is still the same with the bad records * Repeat v1.16.0 will only repeat the records that were unsuccessful if properly configured. We need to update both the plugin and configuration.
Additional PRs to handle retry logic: * https://github.com/uken/fluent-plugin-elasticsearch/pull/419 * https://github.com/openshift/origin-aggregated-logging/pull/1139
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/a793b5fdf1f56ba2f4f9df33aa2d26256e7f7c14 bug 1562004. Add retry logic https://github.com/openshift/origin-aggregated-logging/commit/8524326775b4c7c250072e0f5eb95aea8f7753f6 Merge pull request #1139 from jcantrill/retry_logs bug 1562004. Add retry logic
additional 3.9 cherrypick https://github.com/openshift/origin-aggregated-logging/pull/1151
It seems the stuck in comment 17 was caused by the buffer full. In this testing once I enlarge the BUFFER_QUEUE_LIMIT=35,All buffer file are swallowed, no stuck. For the PR pull/1151 wasn't merged to latest image, so moved to modified status to waiting new images. Current images: openshift3/logging-fluentd/images/v3.9.27-1 "FLUENTD_VERSION=0.12.42"
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/12cb0cac6ad6680b0b73190c53157ffe326ccb1e bug 1562004. Provide utility to sanitize chunks https://github.com/openshift/origin-aggregated-logging/commit/183d0e8c8e15709904640c6802cfcf05c2a43472 Merge pull request #1070 from jcantrill/1562004_sanitize_buffers bug 1562004. Provide utility to sanitize chunks
Created attachment 1452831 [details] log dump and fluentd log The query are still blocked in openshift3/logging-fluentd/images/v3.9.31-2. Steps: 1. Put the attached buffer file under /var/lib/fluentd 2. oc set env ds/logging-fluentd ENABLE_UTF8_FILTER=true oc set env ds/logging-fluentd DEBUG=true VERBOSE=true DEBUG_TRANSPORTER=true TRANSPORTER_LOG_LEVEL=debug oc set env ds/logging-fluentd BUFFER_QUEUE_LIMIT=40 FILE_BUFFER_LIMIT=320Mi 3. watch the buffer query size # ls /var/lib/fluentd/buffer-output-es-config.output_tag.* |wc -l 36 4. watch the log 2018-06-19 15:30:32 +0800 [warn]: fluent/root_agent.rb:187:emit_error_event: dump an error event: error_class=ArgumentError error="invalid byte sequence in UTF-8" location="/usr/share/gems/gems/fluent-plugin-elasticsearch-1.16.1/lib/fluent/plugin/out_elasticsearch.rb:383:in `downcase'" tag="output_tag" time=1522079957 record={"systemd"=>{"t"=>{"BOOT_ID"=>"f4becb7fb2ae42b8abb3076654c926cc", "CAP_EFFECTIVE"=>"1fffffffff", "CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver json-file --log-opt max-size=50m --signature-verification=False --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker_vg-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.basesize=3G --mtu=8951 --add-registry registry.access.redhat.com --add-registry registry.access.redhat.com", "COMM"=>"dockerd-current", "EXE"=>"/usr/bin/dockerd-current", "GID"=>"0", "MACHINE_ID"=>"d52c597d0f1a42aeb01b5a7d71e63f24", "PID"=>"2961", "SELINUX_CONTEXT"=>"system_u:system_r:init_t:s0", "SYSTEMD_CGROUP"=>"/system.slice/docker.service", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"docker.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"dockerd-current"}}, "level"=>"info", "message"=>"time=\"2018-03-26T15:59:17.716724325Z\" level=error msg=\"Failed to log msg \\\" 2018-03-26 15:58:49 +0000 [warn]: suppressed same stacktrace\\\" for logger json-file: write /var/lib/docker/containers/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46-json.log: no space left on device\"", "hostname"=>"ip-172-31-31-141.us-west-1.compute.internal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.131.60.182", "ipaddr6"=>"fe80::c802:5fff:fe7a:629a", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2018-03-26T16:00:23.009320+00:00", "version"=>"0.12.42 1.6.0"}}, "@timestamp"=>"2018-03-26T15:59:17.000000+00:00", 90=>"\u0018\u05C8\xA7systemd\x82\xA1t\x8E\xA7BOOT_ID\xC4 "} 2018-06-19 15:30:34 +0800 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-06-19 15:35:31 +0800 error_class="Fluent::ElasticsearchErrorHandler::ElasticsearchError" error="The number of records submitted 6837 do not match the number returned 6835. Unable to process bulk response." plugin_id="object:3fdebe407fa4" 2018-06-19 15:30:34 +0800 [warn]: suppressed same stacktrace 2018-06-19 15:30:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:31:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:31:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:32:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:32:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:33:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:33:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:34:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:34:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:35:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:35:32 +0800 [warn]: fluent/root_agent.rb:187:emit_error_event: dump an error event: error_class=ArgumentError error="invalid byte sequence in UTF-8" location="/usr/share/gems/gems/fluent-plugin-elasticsearch-1.16.1/lib/fluent/plugin/out_elasticsearch.rb:383:in `downcase'" tag="output_tag" time=1522079957 record={"systemd"=>{"t"=>{"BOOT_ID"=>"f4becb7fb2ae42b8abb3076654c926cc", "CAP_EFFECTIVE"=>"1fffffffff", "CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver json-file --log-opt max-size=50m --signature-verification=False --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker_vg-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.basesize=3G --mtu=8951 --add-registry registry.access.redhat.com --add-registry registry.access.redhat.com", "COMM"=>"dockerd-current", "EXE"=>"/usr/bin/dockerd-current", "GID"=>"0", "MACHINE_ID"=>"d52c597d0f1a42aeb01b5a7d71e63f24", "PID"=>"2961", "SELINUX_CONTEXT"=>"system_u:system_r:init_t:s0", "SYSTEMD_CGROUP"=>"/system.slice/docker.service", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"docker.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"dockerd-current"}}, "level"=>"info", "message"=>"time=\"2018-03-26T15:59:17.716724325Z\" level=error msg=\"Failed to log msg \\\" 2018-03-26 15:58:49 +0000 [warn]: suppressed same stacktrace\\\" for logger json-file: write /var/lib/docker/containers/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46-json.log: no space left on device\"", "hostname"=>"ip-172-31-31-141.us-west-1.compute.internal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.131.60.182", "ipaddr6"=>"fe80::c802:5fff:fe7a:629a", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2018-03-26T16:00:23.009320+00:00", "version"=>"0.12.42 1.6.0"}}, "@timestamp"=>"2018-03-26T15:59:17.000000+00:00", 90=>"\u0018\u05C8\xA7systemd\x82\xA1t\x8E\xA7BOOT_ID\xC4 "} 2018-06-19 15:35:34 +0800 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-06-19 15:40:31 +0800 error_class="Fluent::ElasticsearchErrorHandler::ElasticsearchError" error="The number of records submitted 6837 do not match the number returned 6835. Unable to process bulk response." plugin_id="object:3fdebe407fa4" 2018-06-19 15:35:34 +0800 [warn]: suppressed same stacktrace 2018-06-19 15:35:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:36:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:36:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:37:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:37:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3 2018-06-19 15:38:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
Created attachment 1453065 [details] Fluend trace logs
Updated dependent plugin to properly handle this error scenario: https://github.com/uken/fluent-plugin-elasticsearch/pull/437 Our theory is the record is missing the required field because of deserialization issue and the plugin does not properly handle these silent drops.
Spinning out #c30 into https://bugzilla.redhat.com/show_bug.cgi?id=1593297 since this is a different issue then originally described in this issue.. Fix for #c30 will be part of 1.17.0 of the fluent-plugin-elasticsearch
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/3421d0250c8aa28652dac773abfd15c2710b50ef bug 1562004. Add retry logic https://github.com/openshift/origin-aggregated-logging/commit/0d669546f76d3182838455d49cc72365ef78dfd9 Merge pull request #1152 from openshift-cherrypick-robot/cherry-pick-1139-to-es5.x [es5.x] bug 1562004. Add retry logic
Created attachment 1470765 [details] Fluentd errors logs No queue was blocked. But there are some error message in fluentd.log. and found the index project.opentrip.f7a4543c-9e1c-11e7-b86c-06e7d92b1aa4.2018.03.26 couldn't be found in ES. Can you have a check?
(In reply to Anping Li from comment #41) > Created attachment 1470765 [details] > Fluentd errors logs > > No queue was blocked. But there are some error message in fluentd.log. and > found the index > project.opentrip.f7a4543c-9e1c-11e7-b86c-06e7d92b1aa4.2018.03.26 couldn't be > found in ES. Can you have a check? What does this mean? You couldn't find it using Kibana? Via Kibana, if you are an admin, all index-patterns have been reduced to using 'project.*'. We need to clean up the old index patterns. If you can 'oc exec -c elasticsearch $POD -- indices' and find the index desired then logs are being collected for that namespace.
Move this bug to verified, as no "invalid byte sequence in UTF-8" in fluentd causing the queue to block entirely.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2335
@Anping are you able to provide information regarding what version you verified in #c44
@jeff Sorry for the later. From the comment date, it should be 3.9.40.
Closing again based on #c52. Please re-open if there is additional information
Need to reopen (let me know if you prefer a new-cloned ticket) Jef, Peter. Using fluentd v3.9.41-2 and hitting the same issue. Mysteriously, the fluent-plugin-elasticsearch version loaded is 1.17.2 instead of 1.17.2-1. [nnosenzo@nnm fluentd]$ grep -i docker logging-fluentd-* logging-fluentd-25dph:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-2ps6b:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-2t7cm:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-4hcqn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-4rwrk:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-56xcv:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-652hn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-6q4fm:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-6s7t8:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-95m7p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-bzx5v:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-cwghb:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-d556x:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-dfmgd:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-dhk4v:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-dhmq9:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-dj2kl:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-djd89:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-fcc2p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-fgbwg:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-fjwkk:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-flxjv:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-gm285:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-gqklv:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-gr6bq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-h576v:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-kztrt:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-l77j2:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-lc6zs:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-lchx9:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-ldscd:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-m7mdd:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-nfgnx:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-nlx7z:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-nwsxq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-nxk5p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.43-2 logging-fluentd-nzwcq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-pnd57:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-q4qqn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-qjp67:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-qlvd7:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-rkf9p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-sknpt:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-sv6hn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-t4tcw:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.43-2 logging-fluentd-t8t5n:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-tg6ts:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-vcfmq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-vnq59:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-wjzsm:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-wrqtj:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-x444c:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-xjqd2:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2 logging-fluentd-z2vd9:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.43-2 [nnosenzo@nnm fluentd]$ grep -i fluent-plugin-elasticsearch logs/*.log | grep version logs/logging-fluentd-25dph-fluentd-elasticsearch.log:2018-11-05 14:06:17 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-2ps6b-fluentd-elasticsearch.log:2018-11-05 14:05:07 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-2t7cm-fluentd-elasticsearch.log:2018-11-05 14:06:08 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-4hcqn-fluentd-elasticsearch.log:2018-11-05 14:01:51 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-4rwrk-fluentd-elasticsearch.log:2018-11-05 14:03:03 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-56xcv-fluentd-elasticsearch.log:2018-11-05 14:01:41 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-652hn-fluentd-elasticsearch.log:2018-11-05 14:06:37 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-6q4fm-fluentd-elasticsearch.log:2018-11-05 14:03:50 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-6s7t8-fluentd-elasticsearch.log:2018-11-05 14:05:57 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-95m7p-fluentd-elasticsearch.log:2018-11-05 14:02:52 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-bzx5v-fluentd-elasticsearch.log:2018-11-05 14:05:55 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-cwghb-fluentd-elasticsearch.log:2018-11-05 14:01:48 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-d556x-fluentd-elasticsearch.log:2018-11-05 14:03:18 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-dfmgd-fluentd-elasticsearch.log:2018-11-05 14:03:22 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-dhk4v-fluentd-elasticsearch.log:2018-11-05 14:02:54 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-dhmq9-fluentd-elasticsearch.log:2018-11-05 14:04:16 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-dj2kl-fluentd-elasticsearch.log:2018-11-05 14:06:32 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-djd89-fluentd-elasticsearch.log:2018-11-05 14:06:32 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-fcc2p-fluentd-elasticsearch.log:2018-11-05 14:02:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-fgbwg-fluentd-elasticsearch.log:2018-11-05 14:06:39 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-fjwkk-fluentd-elasticsearch.log:2018-11-05 14:04:17 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-flxjv-fluentd-elasticsearch.log:2018-11-05 14:02:31 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-gm285-fluentd-elasticsearch.log:2018-11-05 14:03:25 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-gqklv-fluentd-elasticsearch.log:2018-11-05 14:02:13 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-gr6bq-fluentd-elasticsearch.log:2018-11-05 14:02:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-h576v-fluentd-elasticsearch.log:2018-11-05 14:01:44 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-kztrt-fluentd-elasticsearch.log:2018-11-05 14:04:03 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-l77j2-fluentd-elasticsearch.log:2018-11-05 14:04:45 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-lc6zs-fluentd-elasticsearch.log:2018-11-05 14:03:21 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-lchx9-fluentd-elasticsearch.log:2018-11-05 14:02:49 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-ldscd-fluentd-elasticsearch.log:2018-11-05 14:06:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-m7mdd-fluentd-elasticsearch.log:2018-11-05 14:01:57 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-nfgnx-fluentd-elasticsearch.log:2018-11-05 14:05:02 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-nlx7z-fluentd-elasticsearch.log:2018-11-05 14:04:14 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-nwsxq-fluentd-elasticsearch.log:2018-11-05 14:01:59 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-nxk5p-fluentd-elasticsearch.log:2018-11-05 14:02:36 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-nzwcq-fluentd-elasticsearch.log:2018-11-05 14:01:57 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-pnd57-fluentd-elasticsearch.log:2018-11-05 14:02:42 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-q4qqn-fluentd-elasticsearch.log:2018-11-05 14:01:11 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-qjp67-fluentd-elasticsearch.log:2018-11-05 14:02:08 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-qlvd7-fluentd-elasticsearch.log:2018-11-05 14:04:45 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-rkf9p-fluentd-elasticsearch.log:2018-11-05 14:01:30 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-sknpt-fluentd-elasticsearch.log:2018-11-05 14:05:35 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-sv6hn-fluentd-elasticsearch.log:2018-11-05 14:05:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-t4tcw-fluentd-elasticsearch.log:2018-11-05 14:06:26 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-t8t5n-fluentd-elasticsearch.log:2018-11-05 14:04:13 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-tg6ts-fluentd-elasticsearch.log:2018-11-05 14:04:02 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-vcfmq-fluentd-elasticsearch.log:2018-11-05 14:03:05 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-vnq59-fluentd-elasticsearch.log:2018-11-05 14:01:42 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-wjzsm-fluentd-elasticsearch.log:2018-11-05 14:04:49 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-wrqtj-fluentd-elasticsearch.log:2018-11-05 14:01:48 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-x444c-fluentd-elasticsearch.log:2018-11-05 14:01:54 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-xjqd2-fluentd-elasticsearch.log:2018-11-05 14:04:51 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' logs/logging-fluentd-z2vd9-fluentd-elasticsearch.log:2018-11-05 14:01:42 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2' [nnosenzo@nnm fluentd]$ grep -i "Invalid UTF-8 start byte 0x92" logs/*.log -c logs/logging-fluentd-25dph-fluentd-elasticsearch.log:12 logs/logging-fluentd-2ps6b-fluentd-elasticsearch.log:0 logs/logging-fluentd-2t7cm-fluentd-elasticsearch.log:12 logs/logging-fluentd-4hcqn-fluentd-elasticsearch.log:0 logs/logging-fluentd-4rwrk-fluentd-elasticsearch.log:12 logs/logging-fluentd-56xcv-fluentd-elasticsearch.log:12 logs/logging-fluentd-652hn-fluentd-elasticsearch.log:0 logs/logging-fluentd-6q4fm-fluentd-elasticsearch.log:0 logs/logging-fluentd-6s7t8-fluentd-elasticsearch.log:0 logs/logging-fluentd-95m7p-fluentd-elasticsearch.log:0 logs/logging-fluentd-bzx5v-fluentd-elasticsearch.log:12 logs/logging-fluentd-cwghb-fluentd-elasticsearch.log:0 logs/logging-fluentd-d556x-fluentd-elasticsearch.log:13 logs/logging-fluentd-dfmgd-fluentd-elasticsearch.log:0 logs/logging-fluentd-dhk4v-fluentd-elasticsearch.log:13 logs/logging-fluentd-dhmq9-fluentd-elasticsearch.log:0 logs/logging-fluentd-dj2kl-fluentd-elasticsearch.log:0 logs/logging-fluentd-djd89-fluentd-elasticsearch.log:0 logs/logging-fluentd-fcc2p-fluentd-elasticsearch.log:0 logs/logging-fluentd-fgbwg-fluentd-elasticsearch.log:0 logs/logging-fluentd-fjwkk-fluentd-elasticsearch.log:0 logs/logging-fluentd-flxjv-fluentd-elasticsearch.log:0 logs/logging-fluentd-gm285-fluentd-elasticsearch.log:0 logs/logging-fluentd-gqklv-fluentd-elasticsearch.log:0 logs/logging-fluentd-gr6bq-fluentd-elasticsearch.log:14 logs/logging-fluentd-h576v-fluentd-elasticsearch.log:0 logs/logging-fluentd-kztrt-fluentd-elasticsearch.log:0 logs/logging-fluentd-l77j2-fluentd-elasticsearch.log:0 logs/logging-fluentd-lc6zs-fluentd-elasticsearch.log:14 logs/logging-fluentd-lchx9-fluentd-elasticsearch.log:14 logs/logging-fluentd-ldscd-fluentd-elasticsearch.log:0 logs/logging-fluentd-m7mdd-fluentd-elasticsearch.log:0 logs/logging-fluentd-nfgnx-fluentd-elasticsearch.log:0 logs/logging-fluentd-nlx7z-fluentd-elasticsearch.log:0 logs/logging-fluentd-nwsxq-fluentd-elasticsearch.log:0 logs/logging-fluentd-nxk5p-fluentd-elasticsearch.log:0 logs/logging-fluentd-nzwcq-fluentd-elasticsearch.log:15 logs/logging-fluentd-pnd57-fluentd-elasticsearch.log:14 logs/logging-fluentd-q4qqn-fluentd-elasticsearch.log:0 logs/logging-fluentd-qjp67-fluentd-elasticsearch.log:0 logs/logging-fluentd-qlvd7-fluentd-elasticsearch.log:0 logs/logging-fluentd-rkf9p-fluentd-elasticsearch.log:0 logs/logging-fluentd-sknpt-fluentd-elasticsearch.log:0 logs/logging-fluentd-sv6hn-fluentd-elasticsearch.log:0 logs/logging-fluentd-t4tcw-fluentd-elasticsearch.log:0 logs/logging-fluentd-t8t5n-fluentd-elasticsearch.log:15 logs/logging-fluentd-tg6ts-fluentd-elasticsearch.log:0 logs/logging-fluentd-vcfmq-fluentd-elasticsearch.log:0 logs/logging-fluentd-vnq59-fluentd-elasticsearch.log:0 logs/logging-fluentd-wjzsm-fluentd-elasticsearch.log:0 logs/logging-fluentd-wrqtj-fluentd-elasticsearch.log:0 logs/logging-fluentd-x444c-fluentd-elasticsearch.log:15 logs/logging-fluentd-xjqd2-fluentd-elasticsearch.log:0 logs/logging-fluentd-z2vd9-fluentd-elasticsearch.log:0
Please open a new issue with this information and additionally attach the fluent config files, which should be in the logging-fluent configmap. The dash in the version is representative of it being built by the Redhat build process. The versions are likely the same.
Ack, opened BZ 1654704 for this.