Bug 1562004 - Encountered another case of "invalid byte sequence in UTF-8" in fluentd causing the queue to block entirely
Summary: Encountered another case of "invalid byte sequence in UTF-8" in fluentd causi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: ---
: 3.9.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1593297 1593310 1593312
Blocks: 1567265
TreeView+ depends on / blocked
 
Reported: 2018-03-29 11:01 UTC by Peter Portante
Modified: 2018-11-29 16:03 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Corrupt File Chunk Buffers Consequence: Fluentd is blocked processing message until the buffer is removed Fix: Introduce handler to remove corrupt messages from the processing work flow Result: Corrupt messages are sent to a 'dead letter queue' while continuing to process other messages and the pipleline is no longer blocked
Clone Of:
: 1567265 1714716 (view as bug list)
Environment:
Last Closed: 2018-11-29 13:06:10 UTC
Target Upstream Version:


Attachments (Terms of Use)
stuck fluentd logs and configuration (7.30 KB, application/x-gzip)
2018-05-03 06:27 UTC, Anping Li
no flags Details
log dump and fluentd log (64.86 KB, application/x-gzip)
2018-06-19 07:52 UTC, Anping Li
no flags Details
Fluend trace logs (7.14 MB, application/x-gzip)
2018-06-20 02:03 UTC, Anping Li
no flags Details
Fluentd errors logs (2.18 KB, application/x-gzip)
2018-07-26 12:50 UTC, Anping Li
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1070 'None' closed bug 1562004. Provide utility to sanitize chunks 2021-01-09 21:21:45 UTC
Github uken fluent-plugin-elasticsearch pull 398 'None' closed introduce dead letter queue to handle issues unpacking file buffer chunks 2021-01-09 21:21:43 UTC
Red Hat Product Errata RHBA-2018:2335 None None None 2018-08-09 22:14:25 UTC

Internal Links: 1625254

Description Peter Portante 2018-03-29 11:01:12 UTC
Encountered another case of "invalid byte sequence in UTF-8" in fluentd causing the queue to block entirely.  This was 3.9 deployed in a starter cluster.

2018-03-29 03:56:58 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-03-29 03:56:58 +0000 error_class="L" error="invalid byte sequence in UTF-8" plugin_id="object:3fec5959ab5c"
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `downcase'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `block in write_objects'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `each'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `block in msgpack_each'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/plugin/buf_file.rb:71:in `open'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:120:in `msgpack_each'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:319:in `write_objects'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:490:in `write'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:354:in `write_chunk'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:333:in `pop'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:342:in `try_flush'
  2018-03-29 03:56:58 +0000 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:149:in `run'

Comment 3 Peter Portante 2018-03-29 22:01:52 UTC
We have another manifestation of this error:

2018-03-29 21:55:15 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-03-29 22:00:13 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"error\":{\"root_cause\":[{\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: [B@5d150508; line: 1, column: 89]\"}],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: [B@5d150508; line: 1, column: 89]\"},\"status\":500}" plugin_id="object:3fed0f088aa4"
  2018-03-29 21:55:15 +0000 [warn]: suppressed same stacktrace

This also effectively stops the queue entirely.

Comment 4 Jeff Cantrill 2018-04-02 20:54:52 UTC
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1511116

Comment 5 Noriko Hosoi 2018-04-02 22:08:08 UTC
@pportant, could you please share these outputs?  Thanks.

oc rsh $FLUENTD_POD -- env | egrep ENABLE_UTF8_FILTER

oc rsh $FLUENTD_POD -- cat /etc/fluent/configs.d/openshift/filter-pre-force-utf8.conf

Comment 6 Peter Portante 2018-04-04 02:16:59 UTC
ENABLE_UTF8_FILTER environment variable does not exist, and /etc/fluent/configs.d/openshift/filter-pre-force-utf8.conf is an empty file.

Comment 7 Noriko Hosoi 2018-04-04 04:14:47 UTC
(In reply to Peter Portante from comment #6)
> ENABLE_UTF8_FILTER environment variable does not exist, and
> /etc/fluent/configs.d/openshift/filter-pre-force-utf8.conf is an empty file.

Thanks, @Peter.  That explains the "invalid byte sequence in UTF-8" error.

The utf8 conversion filter is disabled, by default, due to the performance regression caused by the filter.
    
  https://bugzilla.redhat.com/show_bug.cgi?id=1513254
  The utf8 filter causes the peformance to be cut in half.  Disable
  it by default.  If you need to use it, and can pay the performance
  penalty,
    
    oc set env ds/logging-fluentd ENABLE_UTF8_FILTER=true

Unfortunately, we have to choose either performance degradation or no utf8 error... :(

I think @Jeff is working on a tool/plugin to work around the queue blocking problem.  So, once it's implemented, although we have to live with the utf8 error, it won't freeze the queue.

Comment 9 Jeff Cantrill 2018-04-09 14:08:48 UTC
I don't believe this is related to encoding specifically but some sort of corrupt buffer.  I am unable to determine the source of the corruption.  Unpacking the buffers and applying similar logic used by the elasticsearch plugin shows some of the unpacked messages either do not include valid json or are badly encoded.  The attached PR provides a utility by which the buffers can be manually sanitized.  We need to additionally, provide a procedure for when and how to use, and also modify the elasticsearch plugin to skip bad records

Comment 10 Jeff Cantrill 2018-04-13 17:12:15 UTC
Fixed with fluent-plugin-elasticsearch v0.14.0

Comment 13 Anping Li 2018-04-23 09:13:14 UTC
The issue still can be reproduced in  openshift3/logging-fluentd/images/v3.9.25-1
fluent-plugin-elasticsearch (1.13.2)
FLUENTD_VERSION=0.12.42
DATA_VERSION=1.6.0
VERBOSE=true
RUBY_VERSION=2.0


2018-04-23 05:02:10 -0400 [info]: fluent/supervisor.rb:471:read_config: reading config file path="/etc/fluent/fluent.conf"
2018-04-23 05:02:11 -0400 [warn]: fluent/buffer.rb:164:configure: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2018-04-23 05:02:11 -0400 [warn]: plugin/in_tail.rb:185:block (2 levels) in expand_paths: /var/log/containers/logging-fluentd-m9fjn_logging_fluentd-elasticsearch-c50f3ac46c9de16f083fa0a05f9eff3983ee6f1f905a8540f3e1fceff7d0861f.log unreadable. It is excluded and would be examined next time.
2018-04-23 05:02:12 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:12 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734"
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `downcase'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:358:in `block in write_objects'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `each'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:123:in `block in msgpack_each'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/plugin/buf_file.rb:71:in `open'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:120:in `msgpack_each'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.13.2/lib/fluent/plugin/out_elasticsearch.rb:319:in `write_objects'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:490:in `write'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:354:in `write_chunk'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/buffer.rb:333:in `pop'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:342:in `try_flush'
  2018-04-23 05:02:12 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.42/lib/fluent/output.rb:149:in `run'
2018-04-23 05:02:13 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:15 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734"
  2018-04-23 05:02:13 -0400 [warn]: suppressed same stacktrace
2018-04-23 05:02:15 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:18 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734"
  2018-04-23 05:02:15 -0400 [warn]: suppressed same stacktrace
2018-04-23 05:02:18 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:25 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734"
  2018-04-23 05:02:18 -0400 [warn]: suppressed same stacktrace
2018-04-23 05:02:25 -0400 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-04-23 05:02:39 -0400 error_class="ArgumentError" error="invalid byte sequence in UTF-8" plugin_id="object:3fbc0bd6f734"
  2018-04-23 05:02:25 -0400 [warn]: suppressed same stacktrace

Comment 14 Jeff Cantrill 2018-04-23 13:27:49 UTC
You may need to wait for the nextbuild.  The required plugin exists in the puddle:

[1]http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.9/latest/x86_64/os/Packages/rubygem-fluent-plugin-elasticsearch-1.15.0-1.el7.noarch.rpm

Comment 16 Jeff Cantrill 2018-05-02 13:04:03 UTC
This is not the latest code which provides additional improvements, but it does include the changes to drop corrupt messages.  Are you seeing a 'stuck' fluentd cycling over the same buffer or simply an error message? This looks to be an expected error from a corrupt message that is sent through the error pipeline instead of trying to push it into elastic.

Comment 17 Anping Li 2018-05-03 06:23:21 UTC
@jeff,  Extracted the attached buffer in /var/lib/fluentd, all buffer files are stuck in /var/lib/fluentd.

Comment 18 Anping Li 2018-05-03 06:27:14 UTC
Created attachment 1430467 [details]
stuck fluentd logs and configuration

Comment 19 Jeff Cantrill 2018-05-03 18:59:55 UTC
Actually,

This is still functioning like previously because we need to release v1.16.0.  The issue is:

* We dlq the bad records in the chunk
* We submit records to ES
* ES returns a bulk failure
* We recoginize the failure and throw
* The chunk is still the same with the bad records
* Repeat

v1.16.0 will only repeat the records that were unsuccessful if properly configured.  We need to update both the plugin and configuration.

Comment 22 Jeff Cantrill 2018-05-09 17:40:03 UTC
additional 3.9 cherrypick https://github.com/openshift/origin-aggregated-logging/pull/1151

Comment 24 Anping Li 2018-05-11 08:34:18 UTC
It seems the stuck in comment 17 was caused by the buffer full. In this testing once I enlarge the BUFFER_QUEUE_LIMIT=35,All buffer file are swallowed, no stuck.

For the PR pull/1151 wasn't merged to latest image, so moved to modified status to waiting new images.

Current images:
openshift3/logging-fluentd/images/v3.9.27-1
"FLUENTD_VERSION=0.12.42"

Comment 27 openshift-github-bot 2018-06-12 15:46:05 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/12cb0cac6ad6680b0b73190c53157ffe326ccb1e
bug 1562004. Provide utility to sanitize chunks

https://github.com/openshift/origin-aggregated-logging/commit/183d0e8c8e15709904640c6802cfcf05c2a43472
Merge pull request #1070 from jcantrill/1562004_sanitize_buffers

bug 1562004. Provide utility to sanitize chunks

Comment 30 Anping Li 2018-06-19 07:52:40 UTC
Created attachment 1452831 [details]
log dump and fluentd log

The query are still blocked in openshift3/logging-fluentd/images/v3.9.31-2.

Steps:
1. Put the attached buffer file under /var/lib/fluentd
2. oc set env ds/logging-fluentd ENABLE_UTF8_FILTER=true
   oc set env ds/logging-fluentd DEBUG=true VERBOSE=true DEBUG_TRANSPORTER=true TRANSPORTER_LOG_LEVEL=debug
   oc set env ds/logging-fluentd BUFFER_QUEUE_LIMIT=40  FILE_BUFFER_LIMIT=320Mi
3. watch the buffer query size
# ls /var/lib/fluentd/buffer-output-es-config.output_tag.* |wc -l
36

4. watch the log
    
2018-06-19 15:30:32 +0800 [warn]: fluent/root_agent.rb:187:emit_error_event: dump an error event: error_class=ArgumentError error="invalid byte sequence in UTF-8" location="/usr/share/gems/gems/fluent-plugin-elasticsearch-1.16.1/lib/fluent/plugin/out_elasticsearch.rb:383:in `downcase'" tag="output_tag" time=1522079957 record={"systemd"=>{"t"=>{"BOOT_ID"=>"f4becb7fb2ae42b8abb3076654c926cc", "CAP_EFFECTIVE"=>"1fffffffff", "CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver json-file --log-opt max-size=50m --signature-verification=False --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker_vg-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.basesize=3G --mtu=8951 --add-registry registry.access.redhat.com --add-registry registry.access.redhat.com", "COMM"=>"dockerd-current", "EXE"=>"/usr/bin/dockerd-current", "GID"=>"0", "MACHINE_ID"=>"d52c597d0f1a42aeb01b5a7d71e63f24", "PID"=>"2961", "SELINUX_CONTEXT"=>"system_u:system_r:init_t:s0", "SYSTEMD_CGROUP"=>"/system.slice/docker.service", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"docker.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"dockerd-current"}}, "level"=>"info", "message"=>"time=\"2018-03-26T15:59:17.716724325Z\" level=error msg=\"Failed to log msg \\\"  2018-03-26 15:58:49 +0000 [warn]: suppressed same stacktrace\\\" for logger json-file: write /var/lib/docker/containers/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46-json.log: no space left on device\"", "hostname"=>"ip-172-31-31-141.us-west-1.compute.internal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.131.60.182", "ipaddr6"=>"fe80::c802:5fff:fe7a:629a", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2018-03-26T16:00:23.009320+00:00", "version"=>"0.12.42 1.6.0"}}, "@timestamp"=>"2018-03-26T15:59:17.000000+00:00", 90=>"\u0018\u05C8\xA7systemd\x82\xA1t\x8E\xA7BOOT_ID\xC4 "}
2018-06-19 15:30:34 +0800 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-06-19 15:35:31 +0800 error_class="Fluent::ElasticsearchErrorHandler::ElasticsearchError" error="The number of records submitted 6837 do not match the number returned 6835. Unable to process bulk response." plugin_id="object:3fdebe407fa4"
  2018-06-19 15:30:34 +0800 [warn]: suppressed same stacktrace
2018-06-19 15:30:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:31:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:31:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:32:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:32:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:33:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:33:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:34:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:34:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:35:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:35:32 +0800 [warn]: fluent/root_agent.rb:187:emit_error_event: dump an error event: error_class=ArgumentError error="invalid byte sequence in UTF-8" location="/usr/share/gems/gems/fluent-plugin-elasticsearch-1.16.1/lib/fluent/plugin/out_elasticsearch.rb:383:in `downcase'" tag="output_tag" time=1522079957 record={"systemd"=>{"t"=>{"BOOT_ID"=>"f4becb7fb2ae42b8abb3076654c926cc", "CAP_EFFECTIVE"=>"1fffffffff", "CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver json-file --log-opt max-size=50m --signature-verification=False --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker_vg-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.basesize=3G --mtu=8951 --add-registry registry.access.redhat.com --add-registry registry.access.redhat.com", "COMM"=>"dockerd-current", "EXE"=>"/usr/bin/dockerd-current", "GID"=>"0", "MACHINE_ID"=>"d52c597d0f1a42aeb01b5a7d71e63f24", "PID"=>"2961", "SELINUX_CONTEXT"=>"system_u:system_r:init_t:s0", "SYSTEMD_CGROUP"=>"/system.slice/docker.service", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"docker.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"dockerd-current"}}, "level"=>"info", "message"=>"time=\"2018-03-26T15:59:17.716724325Z\" level=error msg=\"Failed to log msg \\\"  2018-03-26 15:58:49 +0000 [warn]: suppressed same stacktrace\\\" for logger json-file: write /var/lib/docker/containers/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46/a3fca2c0a2f4f44321a7d8165b0cd70c041360ae31960e0959ac8cc7f25dae46-json.log: no space left on device\"", "hostname"=>"ip-172-31-31-141.us-west-1.compute.internal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.131.60.182", "ipaddr6"=>"fe80::c802:5fff:fe7a:629a", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2018-03-26T16:00:23.009320+00:00", "version"=>"0.12.42 1.6.0"}}, "@timestamp"=>"2018-03-26T15:59:17.000000+00:00", 90=>"\u0018\u05C8\xA7systemd\x82\xA1t\x8E\xA7BOOT_ID\xC4 "}
2018-06-19 15:35:34 +0800 [warn]: fluent/output.rb:381:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2018-06-19 15:40:31 +0800 error_class="Fluent::ElasticsearchErrorHandler::ElasticsearchError" error="The number of records submitted 6837 do not match the number returned 6835. Unable to process bulk response." plugin_id="object:3fdebe407fa4"
  2018-06-19 15:35:34 +0800 [warn]: suppressed same stacktrace
2018-06-19 15:35:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:36:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:36:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:37:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:37:43 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3
2018-06-19 15:38:13 +0800 [info]: plugin/filter_kubernetes_metadata.rb:109:dump_stats: stats - namespace_cache_size: 2, pod_cache_size: 3, namespace_cache_api_updates: 3, pod_cache_api_updates: 3, id_cache_miss: 3

Comment 33 Anping Li 2018-06-20 02:03:00 UTC
Created attachment 1453065 [details]
Fluend trace logs

Comment 35 Jeff Cantrill 2018-06-20 02:45:37 UTC
Updated dependent plugin to properly handle this error scenario: https://github.com/uken/fluent-plugin-elasticsearch/pull/437  

Our theory is the record is missing the required field because of deserialization issue and the plugin does not properly handle these silent drops.

Comment 36 Jeff Cantrill 2018-06-20 13:41:35 UTC
Spinning out #c30 into https://bugzilla.redhat.com/show_bug.cgi?id=1593297 since this is a different issue then originally described in this issue..   Fix for #c30 will be part of 1.17.0 of the fluent-plugin-elasticsearch

Comment 37 openshift-github-bot 2018-06-20 15:06:30 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/3421d0250c8aa28652dac773abfd15c2710b50ef
bug 1562004. Add retry logic

https://github.com/openshift/origin-aggregated-logging/commit/0d669546f76d3182838455d49cc72365ef78dfd9
Merge pull request #1152 from openshift-cherrypick-robot/cherry-pick-1139-to-es5.x

[es5.x] bug 1562004. Add retry logic

Comment 41 Anping Li 2018-07-26 12:50:41 UTC
Created attachment 1470765 [details]
Fluentd errors logs

No queue was blocked. But there are some error message in fluentd.log. and found the index project.opentrip.f7a4543c-9e1c-11e7-b86c-06e7d92b1aa4.2018.03.26 couldn't be found in ES.  Can you have a check?

Comment 42 Jeff Cantrill 2018-07-26 14:43:19 UTC
(In reply to Anping Li from comment #41)
> Created attachment 1470765 [details]
> Fluentd errors logs
> 
> No queue was blocked. But there are some error message in fluentd.log. and
> found the index
> project.opentrip.f7a4543c-9e1c-11e7-b86c-06e7d92b1aa4.2018.03.26 couldn't be
> found in ES.  Can you have a check?

What does this mean?  You couldn't find it using Kibana? Via Kibana, if you are an admin, all index-patterns have been reduced to using 'project.*'.  We need to clean up the old index patterns.

If you can 'oc exec -c elasticsearch $POD -- indices' and find the index desired then logs are being collected for that namespace.

Comment 44 Anping Li 2018-07-31 06:07:07 UTC
Move this bug to verified, as no "invalid byte sequence in UTF-8" in fluentd causing the queue to block entirely.

Comment 46 errata-xmlrpc 2018-08-09 22:13:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2335

Comment 47 Jeff Cantrill 2018-10-11 17:02:19 UTC
@Anping are you able to provide information regarding what version you verified in #c44

Comment 51 Anping Li 2018-10-23 03:06:49 UTC
@jeff Sorry for the later. From the  comment date, it should be 3.9.40.

Comment 53 Jeff Cantrill 2018-10-23 17:36:32 UTC
Closing again based on #c52.  Please re-open if there is additional information

Comment 54 Nicolas Nosenzo 2018-11-29 12:55:06 UTC
Need to reopen (let me know if you prefer a new-cloned ticket)

Jef, Peter. Using fluentd v3.9.41-2 and hitting the same issue. Mysteriously, the fluent-plugin-elasticsearch version loaded is 1.17.2 instead of 1.17.2-1.

[nnosenzo@nnm fluentd]$ grep -i docker logging-fluentd-*
logging-fluentd-25dph:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-2ps6b:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-2t7cm:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-4hcqn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-4rwrk:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-56xcv:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-652hn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-6q4fm:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-6s7t8:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-95m7p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-bzx5v:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-cwghb:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-d556x:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-dfmgd:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-dhk4v:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-dhmq9:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-dj2kl:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-djd89:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-fcc2p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-fgbwg:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-fjwkk:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-flxjv:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-gm285:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-gqklv:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-gr6bq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-h576v:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-kztrt:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-l77j2:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-lc6zs:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-lchx9:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-ldscd:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-m7mdd:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-nfgnx:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-nlx7z:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-nwsxq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-nxk5p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.43-2
logging-fluentd-nzwcq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-pnd57:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-q4qqn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-qjp67:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-qlvd7:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-rkf9p:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-sknpt:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-sv6hn:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-t4tcw:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.43-2
logging-fluentd-t8t5n:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-tg6ts:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-vcfmq:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-vnq59:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-wjzsm:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-wrqtj:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-x444c:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-xjqd2:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.41-2
logging-fluentd-z2vd9:Dockerfile info: /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.43-2


[nnosenzo@nnm fluentd]$ grep -i fluent-plugin-elasticsearch logs/*.log | grep version
logs/logging-fluentd-25dph-fluentd-elasticsearch.log:2018-11-05 14:06:17 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-2ps6b-fluentd-elasticsearch.log:2018-11-05 14:05:07 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-2t7cm-fluentd-elasticsearch.log:2018-11-05 14:06:08 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-4hcqn-fluentd-elasticsearch.log:2018-11-05 14:01:51 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-4rwrk-fluentd-elasticsearch.log:2018-11-05 14:03:03 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-56xcv-fluentd-elasticsearch.log:2018-11-05 14:01:41 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-652hn-fluentd-elasticsearch.log:2018-11-05 14:06:37 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-6q4fm-fluentd-elasticsearch.log:2018-11-05 14:03:50 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-6s7t8-fluentd-elasticsearch.log:2018-11-05 14:05:57 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-95m7p-fluentd-elasticsearch.log:2018-11-05 14:02:52 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-bzx5v-fluentd-elasticsearch.log:2018-11-05 14:05:55 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-cwghb-fluentd-elasticsearch.log:2018-11-05 14:01:48 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-d556x-fluentd-elasticsearch.log:2018-11-05 14:03:18 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-dfmgd-fluentd-elasticsearch.log:2018-11-05 14:03:22 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-dhk4v-fluentd-elasticsearch.log:2018-11-05 14:02:54 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-dhmq9-fluentd-elasticsearch.log:2018-11-05 14:04:16 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-dj2kl-fluentd-elasticsearch.log:2018-11-05 14:06:32 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-djd89-fluentd-elasticsearch.log:2018-11-05 14:06:32 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-fcc2p-fluentd-elasticsearch.log:2018-11-05 14:02:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-fgbwg-fluentd-elasticsearch.log:2018-11-05 14:06:39 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-fjwkk-fluentd-elasticsearch.log:2018-11-05 14:04:17 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-flxjv-fluentd-elasticsearch.log:2018-11-05 14:02:31 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-gm285-fluentd-elasticsearch.log:2018-11-05 14:03:25 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-gqklv-fluentd-elasticsearch.log:2018-11-05 14:02:13 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-gr6bq-fluentd-elasticsearch.log:2018-11-05 14:02:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-h576v-fluentd-elasticsearch.log:2018-11-05 14:01:44 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-kztrt-fluentd-elasticsearch.log:2018-11-05 14:04:03 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-l77j2-fluentd-elasticsearch.log:2018-11-05 14:04:45 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-lc6zs-fluentd-elasticsearch.log:2018-11-05 14:03:21 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-lchx9-fluentd-elasticsearch.log:2018-11-05 14:02:49 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-ldscd-fluentd-elasticsearch.log:2018-11-05 14:06:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-m7mdd-fluentd-elasticsearch.log:2018-11-05 14:01:57 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-nfgnx-fluentd-elasticsearch.log:2018-11-05 14:05:02 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-nlx7z-fluentd-elasticsearch.log:2018-11-05 14:04:14 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-nwsxq-fluentd-elasticsearch.log:2018-11-05 14:01:59 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-nxk5p-fluentd-elasticsearch.log:2018-11-05 14:02:36 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-nzwcq-fluentd-elasticsearch.log:2018-11-05 14:01:57 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-pnd57-fluentd-elasticsearch.log:2018-11-05 14:02:42 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-q4qqn-fluentd-elasticsearch.log:2018-11-05 14:01:11 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-qjp67-fluentd-elasticsearch.log:2018-11-05 14:02:08 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-qlvd7-fluentd-elasticsearch.log:2018-11-05 14:04:45 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-rkf9p-fluentd-elasticsearch.log:2018-11-05 14:01:30 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-sknpt-fluentd-elasticsearch.log:2018-11-05 14:05:35 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-sv6hn-fluentd-elasticsearch.log:2018-11-05 14:05:20 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-t4tcw-fluentd-elasticsearch.log:2018-11-05 14:06:26 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-t8t5n-fluentd-elasticsearch.log:2018-11-05 14:04:13 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-tg6ts-fluentd-elasticsearch.log:2018-11-05 14:04:02 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-vcfmq-fluentd-elasticsearch.log:2018-11-05 14:03:05 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-vnq59-fluentd-elasticsearch.log:2018-11-05 14:01:42 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-wjzsm-fluentd-elasticsearch.log:2018-11-05 14:04:49 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-wrqtj-fluentd-elasticsearch.log:2018-11-05 14:01:48 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-x444c-fluentd-elasticsearch.log:2018-11-05 14:01:54 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-xjqd2-fluentd-elasticsearch.log:2018-11-05 14:04:51 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'
logs/logging-fluentd-z2vd9-fluentd-elasticsearch.log:2018-11-05 14:01:42 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.17.2'


[nnosenzo@nnm fluentd]$ grep -i "Invalid UTF-8 start byte 0x92" logs/*.log -c
logs/logging-fluentd-25dph-fluentd-elasticsearch.log:12
logs/logging-fluentd-2ps6b-fluentd-elasticsearch.log:0
logs/logging-fluentd-2t7cm-fluentd-elasticsearch.log:12
logs/logging-fluentd-4hcqn-fluentd-elasticsearch.log:0
logs/logging-fluentd-4rwrk-fluentd-elasticsearch.log:12
logs/logging-fluentd-56xcv-fluentd-elasticsearch.log:12
logs/logging-fluentd-652hn-fluentd-elasticsearch.log:0
logs/logging-fluentd-6q4fm-fluentd-elasticsearch.log:0
logs/logging-fluentd-6s7t8-fluentd-elasticsearch.log:0
logs/logging-fluentd-95m7p-fluentd-elasticsearch.log:0
logs/logging-fluentd-bzx5v-fluentd-elasticsearch.log:12
logs/logging-fluentd-cwghb-fluentd-elasticsearch.log:0
logs/logging-fluentd-d556x-fluentd-elasticsearch.log:13
logs/logging-fluentd-dfmgd-fluentd-elasticsearch.log:0
logs/logging-fluentd-dhk4v-fluentd-elasticsearch.log:13
logs/logging-fluentd-dhmq9-fluentd-elasticsearch.log:0
logs/logging-fluentd-dj2kl-fluentd-elasticsearch.log:0
logs/logging-fluentd-djd89-fluentd-elasticsearch.log:0
logs/logging-fluentd-fcc2p-fluentd-elasticsearch.log:0
logs/logging-fluentd-fgbwg-fluentd-elasticsearch.log:0
logs/logging-fluentd-fjwkk-fluentd-elasticsearch.log:0
logs/logging-fluentd-flxjv-fluentd-elasticsearch.log:0
logs/logging-fluentd-gm285-fluentd-elasticsearch.log:0
logs/logging-fluentd-gqklv-fluentd-elasticsearch.log:0
logs/logging-fluentd-gr6bq-fluentd-elasticsearch.log:14
logs/logging-fluentd-h576v-fluentd-elasticsearch.log:0
logs/logging-fluentd-kztrt-fluentd-elasticsearch.log:0
logs/logging-fluentd-l77j2-fluentd-elasticsearch.log:0
logs/logging-fluentd-lc6zs-fluentd-elasticsearch.log:14
logs/logging-fluentd-lchx9-fluentd-elasticsearch.log:14
logs/logging-fluentd-ldscd-fluentd-elasticsearch.log:0
logs/logging-fluentd-m7mdd-fluentd-elasticsearch.log:0
logs/logging-fluentd-nfgnx-fluentd-elasticsearch.log:0
logs/logging-fluentd-nlx7z-fluentd-elasticsearch.log:0
logs/logging-fluentd-nwsxq-fluentd-elasticsearch.log:0
logs/logging-fluentd-nxk5p-fluentd-elasticsearch.log:0
logs/logging-fluentd-nzwcq-fluentd-elasticsearch.log:15
logs/logging-fluentd-pnd57-fluentd-elasticsearch.log:14
logs/logging-fluentd-q4qqn-fluentd-elasticsearch.log:0
logs/logging-fluentd-qjp67-fluentd-elasticsearch.log:0
logs/logging-fluentd-qlvd7-fluentd-elasticsearch.log:0
logs/logging-fluentd-rkf9p-fluentd-elasticsearch.log:0
logs/logging-fluentd-sknpt-fluentd-elasticsearch.log:0
logs/logging-fluentd-sv6hn-fluentd-elasticsearch.log:0
logs/logging-fluentd-t4tcw-fluentd-elasticsearch.log:0
logs/logging-fluentd-t8t5n-fluentd-elasticsearch.log:15
logs/logging-fluentd-tg6ts-fluentd-elasticsearch.log:0
logs/logging-fluentd-vcfmq-fluentd-elasticsearch.log:0
logs/logging-fluentd-vnq59-fluentd-elasticsearch.log:0
logs/logging-fluentd-wjzsm-fluentd-elasticsearch.log:0
logs/logging-fluentd-wrqtj-fluentd-elasticsearch.log:0
logs/logging-fluentd-x444c-fluentd-elasticsearch.log:15
logs/logging-fluentd-xjqd2-fluentd-elasticsearch.log:0
logs/logging-fluentd-z2vd9-fluentd-elasticsearch.log:0

Comment 55 Jeff Cantrill 2018-11-29 13:06:10 UTC
Please open a new issue with this information and additionally attach the fluent config files, which should be in the logging-fluent configmap.  The dash in the version is representative of it being built by the Redhat build process.  The versions are likely the same.

Comment 56 Nicolas Nosenzo 2018-11-29 16:03:50 UTC
Ack, opened BZ 1654704 for this.


Note You need to log in before you can comment on or make changes to this bug.