Bug 1654704
| Summary: | Invalid UTF-8 start byte 0x92 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Nicolas Nosenzo <nnosenzo> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Anping Li <anli> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 3.9.0 | CC: | aos-bugs, dyocum, gferrazs, jcantril, nnosenzo, rludva, rmeggins, sgarciam, travi |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-26 14:57:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Nicolas Nosenzo
2018-11-29 13:14:48 UTC
Carrying this comment over from https://bugzilla.redhat.com/show_bug.cgi?id=1625254 Jeff Cantrill 2018-10-17 14:17:42 EDT RED HAT CONFIDENTIAL Dan, Reading through the attached customer case seems to indicate the customer is trying to use fluent to forward logs to splunk an was having issues routing messages there (i.e. firewall issue)? The previous cases I have examined where the utf-8 parse issue presents itself seem to be related to available disk. Fluent tries to write buffers to the host filesystem, buffer gets corrupted because it runs out of disk. It is plausible that if fluent is unable to write its messages to the destination that they get pushed back into a buffer for retry. This could eventually lead to disk issues if fluent is filling up the file system. We verified that docker space disk sometimes reached the 88%, but even though, the remaining 12% stands for ~30G of free space. Not really sure if this issue is due to a disk pressure. (In reply to Nicolas Nosenzo from comment #2) > We verified that docker space disk sometimes reached the 88%, but even > though, the remaining 12% stands for ~30G of free space. Not really sure if > this issue is due to a disk pressure. If it isn't due to disk pressure, then we have absolutely no clue as to what the problem could possibly be, and we're going to need a lot of diagnostic help (i.e. changing fluentd configs to dump out the records before this error happens, dumping the fluentd buffer files, etc.) in order to figure out what else it could be. (In reply to Rich Megginson from comment #3) > changing fluentd configs to dump out the records before this > error happens, dumping the fluentd buffer files, etc.) in order to figure > out what else it could be. Is this possible with setting fluentd in trace log level ? and just listing (ls) the buffer file folder ? (In reply to Nicolas Nosenzo from comment #4) > (In reply to Rich Megginson from comment #3) > > > changing fluentd configs to dump out the records before this > > error happens, dumping the fluentd buffer files, etc.) in order to figure > > out what else it could be. > > Is this possible with setting fluentd in trace log level ? No - trace mode will not dump the contents of the record. > and just listing > (ls) the buffer file folder ? The problem is that we want to see the contents of the records _before_ they are written to the file buffers, so that we can rule out the records being corrupted in some other way. @Nicolas, Is the customer still experiencing this issue? If so, would it be possible to get some recent fluent and elasticsearch logs: oc exec $ESPOD -- logs oc exec $FLUENT -- logs which is contingent on their version having: [1] https://github.com/openshift/origin-aggregated-logging/blob/release-3.9/elasticsearch/utils/logs [2] https://github.com/openshift/origin-aggregated-logging/blob/release-3.9/fluentd/utils/logs We have recently seen an issue with records getting rejected by Elasticsearch with a 400 error. My belief is the rejected entries cycle through fluent's buffers indefinitely which may lead to buffers never being flushed. (In reply to Jeff Cantrill from comment #8) > @Nicolas, > > Is the customer still experiencing this issue? If so, would it be possible > to get some recent fluent and elasticsearch logs: > > oc exec $ESPOD -- logs > oc exec $FLUENT -- logs > > which is contingent on their version having: > > [1] > https://github.com/openshift/origin-aggregated-logging/blob/release-3.9/ > elasticsearch/utils/logs > [2] > https://github.com/openshift/origin-aggregated-logging/blob/release-3.9/ > fluentd/utils/logs > > We have recently seen an issue with records getting rejected by > Elasticsearch with a 400 error. My belief is the rejected entries cycle > through fluent's buffers indefinitely which may lead to buffers never being > flushed. Jeff, Haven't got a response from my last update, anyhow I will ask them to gather these logs in case the issue is still there. I defer to the comments from Rich. We are confident these issues are related to lack of buffer space on the node for fluent to write buffer files. Speculating, I would bet the nodes where you see the utf-8 issues are also nodes which have containers that generate high log volumes which would only be confirmed in 3.9 by monitoring the node's disk space over time. Later installations of Openshift (e.g. 3.11) do/can include prometheus to grabe node metrics, one of which is disk usage and capacity. In 3.9 you may have hawkular metrics deployed which might provide you the same information. I can not speak definitively to what is available. Alternatively, you have to introduce a service on the node that periodically dumps information about the disk which you then would have to aggregate and graph to see trends about its usage. *** Bug 1716553 has been marked as a duplicate of this bug. *** Reclosing this issue as there has been no recent activity and marked as a duplicate |