Bug 1716553
| Summary: | Invalid UTF-8 start byte 0x92\\n (logs are not sent to Elasticsearch cluster) | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Radomir Ludva <rludva> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED DUPLICATE | QA Contact: | Anping Li <anli> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.11.0 | CC: | aos-bugs, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-06-04 15:55:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to Radomir Ludva from comment #0) > Description of problem: > Logs are not sent to Elasticsearch cluster and the buffer is not flushed in > the fluentd pod. > > Version-Release number of selected component (if applicable): > openshift3/ose-logging-fluentd/images/v3.11.98-6 > > sh-4.2# echo $FLUENTD_VERSION > 0.12.43 > > sh-4.2# fluent-gem list | grep fluent-plugin-elasticsearch > > fluent-plugin-elasticsearch (1.17.2) > > > How to reproduce: > I do not know how to reproduce this issue. > > > Actual results: > 2019-06-03 14:10:23 +0200 [warn]: temporarily failed to flush the buffer. > next_retry=2019-06-03 14:15:23 +0200 > error_class="Elasticsearch::Transport::Transport::Errors:: > InternalServerError" error="[500] > {\"error\":{\"root_cause\":[{\"type\":\"json_parse_exception\",\"reason\": > \"Invalid UTF-8 start byte 0x92\\n at [Source: > org.elasticsearch.transport.netty4.ByteBufStreamInput@524e9b40; line: 1, > column: 135]\"}],\"type\":\"json_parse_exception\",\"reason\":\"Invalid > UTF-8 start byte 0x92\\n at [Source: > org.elasticsearch.transport.netty4.ByteBufStreamInput@524e9b40; line: 1, > column: 135]\"},\"status\":500}" plugin_id="elasticsearch-apps" > > Expected results: > Logs are forwarded to Elasticsearch successfully. > > Additional info: > As a workaround enabled the UTF8 Filter > $ oc set env ds/logging-fluentd ENABLE_UTF8_FILTER=true How does this work? Are you sure you aren't missing any records, or any data within those records? I mean, using ENABLE_UTF8_FILTER=true might make the error 500 go away, but is it losing data? If there is already non-UTF8 characters in the fluentd buffer, the issue is the same and it is important to delete logs. We do not know if this change will work for future logs only. Please how to check or confirm that it is losing data? ENABLE_UTF8_FILTER did not help to manage this issue and is right now disabled. We realize that same records with this UTF-8 character were processed before enabling it. *** This bug has been marked as a duplicate of bug 1654704 *** |
Description of problem: Logs are not sent to Elasticsearch cluster and the buffer is not flushed in the fluentd pod. Version-Release number of selected component (if applicable): openshift3/ose-logging-fluentd/images/v3.11.98-6 sh-4.2# echo $FLUENTD_VERSION 0.12.43 sh-4.2# fluent-gem list | grep fluent-plugin-elasticsearch fluent-plugin-elasticsearch (1.17.2) How to reproduce: I do not know how to reproduce this issue. Actual results: 2019-06-03 14:10:23 +0200 [warn]: temporarily failed to flush the buffer. next_retry=2019-06-03 14:15:23 +0200 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"error\":{\"root_cause\":[{\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@524e9b40; line: 1, column: 135]\"}],\"type\":\"json_parse_exception\",\"reason\":\"Invalid UTF-8 start byte 0x92\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@524e9b40; line: 1, column: 135]\"},\"status\":500}" plugin_id="elasticsearch-apps" Expected results: Logs are forwarded to Elasticsearch successfully. Additional info: As a workaround enabled the UTF8 Filter $ oc set env ds/logging-fluentd ENABLE_UTF8_FILTER=true