Bug 1691142
| Summary: | Unparsable logs causing continued block on logging stack | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Steven Walter <stwalter> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED DUPLICATE | QA Contact: | Anping Li <anli> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.11.0 | CC: | aos-bugs, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-21 13:04:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Closing as a duplicate to https://bugzilla.redhat.com/show_bug.cgi?id=1668338 Only current resolution is to: * Disable MERGE_JSON_LOG * remove indices causing the issue. *** This bug has been marked as a duplicate of bug 1668338 *** Thanks for all the info. I'm updating https://access.redhat.com/solutions/3986441 later today to keep this knowledge available for CEE and customers (In reply to Steven Walter from comment #7) > Thanks for all the info. I'm updating > https://access.redhat.com/solutions/3986441 later today to keep this > knowledge available for CEE and customers Thanks for the KB - I've added some comments which might be helpful diagnosing and working around this issue. |
Description of problem: Elasticsearch rejecting logs, it seems due to conflicts of data types in the json structure (i.e. one app has a key called "arg0" that's of one data type, another app has a key called "arg0" of a conflicting data type). While this is potentially a problem that we want to address (with, as a workaround in the meantime, having the customer rename the logging fields), I haven't found a way to get fluentd / elasticsaerch to "skip" these logs -- which I thought they were suppoesd to do automatically. (I recall that we've made changes in the past to make fluentd send the non-broken logs separately from the broken logs) But, even after scaling down the apps and restarting fluentd pods, issue persists Fluentd reports: dump an error event: error_class=Fluent::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="output_tag" time=1551903902 record={ . . . } Elasticsearch reports: failed to execute bulk item (create) BulkShardRequest [[project.xxxxxxxxxx][0]] containing [786] requests org.elasticsearch.index.mapper.MapperParsingException: failed to parse [arg1] at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.6.12.redhat-1.jar:5.6.12.redhat-1] . . . Version-Release number of selected component (if applicable): image: registry.access.redhat.com/openshift3/ose-logging-elasticsearch5:v3.11.82 How reproducible: Repeated in customer environment Expected results: Additional info: We've addressed similar issues in past, but that was a different error (too many arguments for ES): https://access.redhat.com/solutions/3868701