Bug 1501993
| Summary: | Encoding UndefinedConversionError from ASCII-8BIT to UTF-8 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ruben Romero Montes <rromerom> |
| Component: | Logging | Assignee: | Noriko Hosoi <nhosoi> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.6.0 | CC: | aos-bugs, doperry, jmalde, nhosoi, rmeggins, stwalter |
| Target Milestone: | --- | ||
| Target Release: | 3.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Fluentd input plugins treat messages as ASCII-8BIT by default.
Consequence: As a result, if one of these messages or any other message with UTF-8 characters is read in and later needs to be converted to UTF-8, the conversion will fail with an UndefinedConversionError.
Fix: The problem is fixed by adding new record_modifier filter which treats the messages as UTF-8.
Result: No more UndefinedConversionError.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-14 21:01:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
https://github.com/openshift/origin-aggregated-logging/pull/722 Force log messages to be treated as UTF-8 (3.6 backport of #622) koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=615164 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-57336-20171027134826 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61-2 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6 Test with logging-fluentd: 3.6.173.0.72, the logging works well with the non ascii characters. The non ascii characters can be display in kibana, and the external fluentd. So move bug to verified. For example: logger test ㄅㄉˇˋㄓˊ˙ㄚㄞㄢㄦㄆ 中国 883.317µs ā á ǎ à ō ó ▅ ▆ ▇ █ 々 I had a customer hit this with 3.5.0-26. Will the fix be implemented in 3.5 as well? If not are there any workarounds? Updating severity to match customer case; this is preventing customer from forwarding logs to their external logging stack It is preventing customer from forwarding their logs to external logging aggregator Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3438 |
Description of problem: After upgrading EFK stack to 3.6, forwarding logs to an external fluentd is causing the following error: 13 Oct 2017 09:44:29.944{ "error_class": "Encoding::UndefinedConversionError", "error": "\"\\xC2\" from ASCII-8BIT to UTF-8", "tag": "kubernetes.journal.container", "message": "emit transaction failed: error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xC2\\\" from ASCII-8BIT to UTF-8\" tag=\"kubernetes.journal.container\"" } 13 Oct 2017 09:44:29.944{ "error_class": "Encoding::UndefinedConversionError", "error": "#<Encoding::UndefinedConversionError: \"\\xC2\" from ASCII-8BIT to UTF-8>", "message": "unexpected error in in_secure_forward from 192.168.100.100:45400 error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: \"\\xC2\" from ASCII-8BIT to UTF-8>" } Version-Release number of selected component (if applicable): FLUENTD_VERSION=0.12.39 Dockerfile-openshift3-logging-fluentd-v3.6.173.0.21-17 How reproducible: Always Steps to Reproduce: 1. Configure secure-forward-plugin in fluentd @type secure_forward self_hostname "#{ENV['HOSTNAME']}" shared_key xxxxxxxxx secure yes # enable_strict_verification yes ca_cert_path /etc/fluent/keys/external_ca_cert.pem ca_private_key_path /etc/fluent/keys/external_ca_key.pem ca_private_key_passphrase xxxxxxxxx <server> host 192.168.100.100 port 24284 </server> Additional info: Seems to be already identified in https://github.com/openshift/origin-aggregated-logging/issues/621 External fluentd instance initialization logs: 2017-10-13 11:59:20 +0100 [info]: reading config file path="/etc/td-agent/td-agent.conf" 2017-10-13 11:59:20 +0100 [info]: starting fluentd-0.12.40 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-kafka' version '0.6.1' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-mongo' version '0.8.1' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-s3' version '0.8.5' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-scribe' version '0.10.14' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-splunk-ex' version '1.0.2' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-td' version '0.10.29' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-webhdfs' version '0.7.1' 2017-10-13 11:59:20 +0100 [info]: gem 'fluentd' version '0.12.40' 2017-10-13 11:59:20 +0100 [info]: gem 'fluentd' version '0.10.62' 2017-10-13 11:59:20 +0100 [info]: adding match pattern="td.*.*" type="tdlog" 2017-10-13 11:59:20 +0100 [info]: adding match pattern="debug.**" type="stdout" 2017-10-13 11:59:20 +0100 [info]: adding match pattern="**" type="splunk_ex" 2017-10-13 11:59:20 +0100 [info]: adding source type="secure_forward" 2017-10-13 11:59:20 +0100 [info]: adding source type="forward" 2017-10-13 11:59:20 +0100 [info]: adding source type="http" 2017-10-13 11:59:20 +0100 [info]: adding source type="debug_agent" 2017-10-13 11:59:20 +0100 [info]: using configuration file: <ROOT> <match td.*.*> @type tdlog apikey xxxxxx auto_create_table buffer_type file buffer_path /var/log/td-agent/buffer/td buffer_chunk_limit 33554432 <secondary> @type file path /var/log/td-agent/failed_records buffer_path /var/log/td-agent/failed_records.* </secondary> </match> <match debug.**> @type stdout </match> <match **> type splunk_ex host 192.168.100.101 port 10000 output_format json </match> <source> @type secure_forward self_hostname bind 0.0.0.0 port 24284 shared_key xxxxxx secure yes cert_path /etc/td-agent/certs/ca_cert.pem private_key_path /etc/td-agent/certs/ca_key.pem private_key_passphrase xxxxxx </source> <source> @type forward </source> <source> @type http port 8888 </source> <source> @type debug_agent bind 127.0.0.1 port 24230 </source> </ROOT> 2017-10-13 11:59:20 +0100 [info]: listening fluent socket on 0.0.0.0:24224 2017-10-13 11:59:20 +0100 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine" 2017-10-13 11:59:35 +0100 [warn]: emit transaction failed: error_class=Encoding::UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8" tag="kubernetes.journal.container" 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `encode' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `to_json' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `format_json' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:66:in `call' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:66:in `block in emit' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:149:in `feed_each' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:149:in `each' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:65:in `emit' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_router.rb:90:in `emit_stream' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/in_secure_forward.rb:293:in `on_message' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:124:in `on_read' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:185:in `feed_each' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:185:in `block in start' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:178:in `loop' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:178:in `start' 2017-10-13 11:59:35 +0100 [warn]: unexpected error in in_secure_forward from 192.168.100.100:54900 error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: "\xC2" from ASCII-8BIT to UTF-8>