Description of problem: After upgrading EFK stack to 3.6, forwarding logs to an external fluentd is causing the following error: 13 Oct 2017 09:44:29.944{ "error_class": "Encoding::UndefinedConversionError", "error": "\"\\xC2\" from ASCII-8BIT to UTF-8", "tag": "kubernetes.journal.container", "message": "emit transaction failed: error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xC2\\\" from ASCII-8BIT to UTF-8\" tag=\"kubernetes.journal.container\"" } 13 Oct 2017 09:44:29.944{ "error_class": "Encoding::UndefinedConversionError", "error": "#<Encoding::UndefinedConversionError: \"\\xC2\" from ASCII-8BIT to UTF-8>", "message": "unexpected error in in_secure_forward from 192.168.100.100:45400 error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: \"\\xC2\" from ASCII-8BIT to UTF-8>" } Version-Release number of selected component (if applicable): FLUENTD_VERSION=0.12.39 Dockerfile-openshift3-logging-fluentd-v3.6.173.0.21-17 How reproducible: Always Steps to Reproduce: 1. Configure secure-forward-plugin in fluentd @type secure_forward self_hostname "#{ENV['HOSTNAME']}" shared_key xxxxxxxxx secure yes # enable_strict_verification yes ca_cert_path /etc/fluent/keys/external_ca_cert.pem ca_private_key_path /etc/fluent/keys/external_ca_key.pem ca_private_key_passphrase xxxxxxxxx <server> host 192.168.100.100 port 24284 </server> Additional info: Seems to be already identified in https://github.com/openshift/origin-aggregated-logging/issues/621 External fluentd instance initialization logs: 2017-10-13 11:59:20 +0100 [info]: reading config file path="/etc/td-agent/td-agent.conf" 2017-10-13 11:59:20 +0100 [info]: starting fluentd-0.12.40 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-kafka' version '0.6.1' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-mongo' version '0.8.1' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-s3' version '0.8.5' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-scribe' version '0.10.14' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-splunk-ex' version '1.0.2' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-td' version '0.10.29' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3' 2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-webhdfs' version '0.7.1' 2017-10-13 11:59:20 +0100 [info]: gem 'fluentd' version '0.12.40' 2017-10-13 11:59:20 +0100 [info]: gem 'fluentd' version '0.10.62' 2017-10-13 11:59:20 +0100 [info]: adding match pattern="td.*.*" type="tdlog" 2017-10-13 11:59:20 +0100 [info]: adding match pattern="debug.**" type="stdout" 2017-10-13 11:59:20 +0100 [info]: adding match pattern="**" type="splunk_ex" 2017-10-13 11:59:20 +0100 [info]: adding source type="secure_forward" 2017-10-13 11:59:20 +0100 [info]: adding source type="forward" 2017-10-13 11:59:20 +0100 [info]: adding source type="http" 2017-10-13 11:59:20 +0100 [info]: adding source type="debug_agent" 2017-10-13 11:59:20 +0100 [info]: using configuration file: <ROOT> <match td.*.*> @type tdlog apikey xxxxxx auto_create_table buffer_type file buffer_path /var/log/td-agent/buffer/td buffer_chunk_limit 33554432 <secondary> @type file path /var/log/td-agent/failed_records buffer_path /var/log/td-agent/failed_records.* </secondary> </match> <match debug.**> @type stdout </match> <match **> type splunk_ex host 192.168.100.101 port 10000 output_format json </match> <source> @type secure_forward self_hostname bind 0.0.0.0 port 24284 shared_key xxxxxx secure yes cert_path /etc/td-agent/certs/ca_cert.pem private_key_path /etc/td-agent/certs/ca_key.pem private_key_passphrase xxxxxx </source> <source> @type forward </source> <source> @type http port 8888 </source> <source> @type debug_agent bind 127.0.0.1 port 24230 </source> </ROOT> 2017-10-13 11:59:20 +0100 [info]: listening fluent socket on 0.0.0.0:24224 2017-10-13 11:59:20 +0100 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine" 2017-10-13 11:59:35 +0100 [warn]: emit transaction failed: error_class=Encoding::UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8" tag="kubernetes.journal.container" 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `encode' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `to_json' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `format_json' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:66:in `call' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:66:in `block in emit' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:149:in `feed_each' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:149:in `each' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:65:in `emit' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_router.rb:90:in `emit_stream' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/in_secure_forward.rb:293:in `on_message' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:124:in `on_read' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:185:in `feed_each' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:185:in `block in start' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:178:in `loop' 2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:178:in `start' 2017-10-13 11:59:35 +0100 [warn]: unexpected error in in_secure_forward from 192.168.100.100:54900 error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: "\xC2" from ASCII-8BIT to UTF-8>
https://github.com/openshift/origin-aggregated-logging/pull/622
https://github.com/openshift/origin-aggregated-logging/pull/722 Force log messages to be treated as UTF-8 (3.6 backport of #622)
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=615164 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-57336-20171027134826 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61-2 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6
Test with logging-fluentd: 3.6.173.0.72, the logging works well with the non ascii characters. The non ascii characters can be display in kibana, and the external fluentd. So move bug to verified. For example: logger test ㄅㄉˇˋㄓˊ˙ㄚㄞㄢㄦㄆ 中国 883.317µs ā á ǎ à ō ó ▅ ▆ ▇ █ 々
I had a customer hit this with 3.5.0-26. Will the fix be implemented in 3.5 as well? If not are there any workarounds?
Updating severity to match customer case; this is preventing customer from forwarding logs to their external logging stack
It is preventing customer from forwarding their logs to external logging aggregator
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3438