Bug 1501993 - Encoding UndefinedConversionError from ASCII-8BIT to UTF-8
Summary: Encoding UndefinedConversionError from ASCII-8BIT to UTF-8
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 3.6.z
Assignee: Noriko Hosoi
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-13 16:28 UTC by Ruben Romero Montes
Modified: 2018-04-17 09:10 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd input plugins treat messages as ASCII-8BIT by default. Consequence: As a result, if one of these messages or any other message with UTF-8 characters is read in and later needs to be converted to UTF-8, the conversion will fail with an UndefinedConversionError. Fix: The problem is fixed by adding new record_modifier filter which treats the messages as UTF-8. Result: No more UndefinedConversionError.
Clone Of:
Environment:
Last Closed: 2017-12-14 21:01:55 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 722 None closed Force log messages to be treated as UTF-8 (3.6 backport of #622) 2020-08-11 17:21:58 UTC
Red Hat Knowledge Base (Solution) 3264551 None None None 2017-12-06 22:28:56 UTC
Red Hat Product Errata RHBA-2017:3438 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Ruben Romero Montes 2017-10-13 16:28:08 UTC
Description of problem:
After upgrading EFK stack to 3.6, forwarding logs to an external fluentd is causing the following error:


13 Oct 2017 09:44:29.944{ "error_class": "Encoding::UndefinedConversionError", "error": "\"\\xC2\" from ASCII-8BIT to UTF-8", "tag": "kubernetes.journal.container", "message": "emit transaction failed: error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xC2\\\" from ASCII-8BIT to UTF-8\" tag=\"kubernetes.journal.container\"" }
13 Oct 2017 09:44:29.944{ "error_class": "Encoding::UndefinedConversionError", "error": "#<Encoding::UndefinedConversionError: \"\\xC2\" from ASCII-8BIT to UTF-8>", "message": "unexpected error in in_secure_forward from 192.168.100.100:45400 error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: \"\\xC2\" from ASCII-8BIT to UTF-8>" }


Version-Release number of selected component (if applicable):
FLUENTD_VERSION=0.12.39
Dockerfile-openshift3-logging-fluentd-v3.6.173.0.21-17

How reproducible:
Always

Steps to Reproduce:
1. Configure secure-forward-plugin in fluentd
@type secure_forward

self_hostname "#{ENV['HOSTNAME']}"
shared_key xxxxxxxxx

secure yes
# enable_strict_verification yes

ca_cert_path /etc/fluent/keys/external_ca_cert.pem
ca_private_key_path /etc/fluent/keys/external_ca_key.pem
ca_private_key_passphrase xxxxxxxxx

<server>
   host 192.168.100.100
   port 24284
</server>


Additional info:
Seems to be already identified in 
 https://github.com/openshift/origin-aggregated-logging/issues/621


External fluentd instance initialization logs:

2017-10-13 11:59:20 +0100 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2017-10-13 11:59:20 +0100 [info]: starting fluentd-0.12.40
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-kafka' version '0.6.1'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-mongo' version '0.8.1'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-s3' version '0.8.5'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-splunk-ex' version '1.0.2'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2017-10-13 11:59:20 +0100 [info]: gem 'fluent-plugin-webhdfs' version '0.7.1'
2017-10-13 11:59:20 +0100 [info]: gem 'fluentd' version '0.12.40'
2017-10-13 11:59:20 +0100 [info]: gem 'fluentd' version '0.10.62'
2017-10-13 11:59:20 +0100 [info]: adding match pattern="td.*.*" type="tdlog"
2017-10-13 11:59:20 +0100 [info]: adding match pattern="debug.**" type="stdout"
2017-10-13 11:59:20 +0100 [info]: adding match pattern="**" type="splunk_ex"
2017-10-13 11:59:20 +0100 [info]: adding source type="secure_forward"
2017-10-13 11:59:20 +0100 [info]: adding source type="forward"
2017-10-13 11:59:20 +0100 [info]: adding source type="http"
2017-10-13 11:59:20 +0100 [info]: adding source type="debug_agent"
2017-10-13 11:59:20 +0100 [info]: using configuration file: <ROOT>
  <match td.*.*>
    @type tdlog
    apikey xxxxxx
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    buffer_chunk_limit 33554432
    <secondary>
      @type file
      path /var/log/td-agent/failed_records
      buffer_path /var/log/td-agent/failed_records.*
    </secondary>
  </match>
  <match debug.**>
    @type stdout
  </match>
  <match **>
    type splunk_ex
    host 192.168.100.101
    port 10000
    output_format json
  </match>
  <source>
    @type secure_forward
    self_hostname
    bind 0.0.0.0
    port 24284
    shared_key xxxxxx
    secure yes
    cert_path /etc/td-agent/certs/ca_cert.pem
    private_key_path /etc/td-agent/certs/ca_key.pem
    private_key_passphrase xxxxxx
  </source>
  <source>
    @type forward
  </source>
  <source>
    @type http
    port 8888
  </source>
  <source>
    @type debug_agent
    bind 127.0.0.1
    port 24230
  </source>
</ROOT>
2017-10-13 11:59:20 +0100 [info]: listening fluent socket on 0.0.0.0:24224
2017-10-13 11:59:20 +0100 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2017-10-13 11:59:35 +0100 [warn]: emit transaction failed: error_class=Encoding::UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8" tag="kubernetes.journal.container"
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `encode'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `to_json'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:83:in `format_json'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:66:in `call'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:66:in `block in emit'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:149:in `feed_each'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:149:in `each'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-splunk-ex-1.0.2/lib/fluent/plugin/out_splunk_ex.rb:65:in `emit'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_router.rb:90:in `emit_stream'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/in_secure_forward.rb:293:in `on_message'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:124:in `on_read'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:185:in `feed_each'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:185:in `block in start'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:178:in `loop'
  2017-10-13 11:59:35 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-secure-forward-0.4.5/lib/fluent/plugin/input_session.rb:178:in `start'
2017-10-13 11:59:35 +0100 [warn]: unexpected error in in_secure_forward from 192.168.100.100:54900 error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: "\xC2" from ASCII-8BIT to UTF-8>

Comment 4 Noriko Hosoi 2017-10-13 23:06:41 UTC
https://github.com/openshift/origin-aggregated-logging/pull/722
Force log messages to be treated as UTF-8 (3.6 backport of #622)

Comment 5 Rich Megginson 2017-10-27 14:19:02 UTC
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=615164
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-57336-20171027134826
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61-2
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6

Comment 6 Anping Li 2017-11-12 14:13:33 UTC
Test with logging-fluentd: 3.6.173.0.72, the logging works well with the non ascii characters. The non ascii characters can be display in kibana, and the external fluentd. So move bug to verified.

For example: logger test  ㄅㄉˇˋㄓˊ˙ㄚㄞㄢㄦㄆ 中国 883.317µs ā á ǎ à ō ó   ▅ ▆ ▇ █ 々

Comment 7 Steven Walter 2017-12-05 00:24:54 UTC
I had a customer hit this with 3.5.0-26. Will the fix be implemented in 3.5 as well? If not are there any workarounds?

Comment 8 Steven Walter 2017-12-05 01:00:26 UTC
Updating severity to match customer case; this is preventing customer from forwarding logs to their external logging stack

Comment 9 Jatan Malde 2017-12-05 11:18:48 UTC
It is preventing customer from forwarding their logs to external logging aggregator

Comment 15 errata-xmlrpc 2017-12-14 21:01:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438


Note You need to log in before you can comment on or make changes to this bug.