Bug 1686947
| Summary: | Allow MERGE_JSON_LOG=true for indexing of JSON payload fields | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Rich Megginson <rmeggins> |
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | anli, aos-bugs, ddelcian, jcantril, jgoulding, pstrick, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openshift3/logging-fluentd:v3.9.73-1 | Doc Type: | Bug Fix |
| Doc Text: |
Cause: Using MERGE_JSON_LOG=true can create fields in the
record which will cause schema and syntax violations in
Elasticsearch. It can also create too many fields for
Elasticsearch to handle without severe performance problems.
Consequence: Fluentd reports error 400 sending records to
Elasticsearch. Elasticsearch performance degrades.
Fix: Allow users who experience these problems to tune their
Fluentd to accomodate their log record fields.
Result: Logs are ingested into Fluentd with no errors.
Elasticsearch performance does not degrade.
|
Story Points: | --- |
| Clone Of: | 1686946 | Environment: | |
| Last Closed: | 2019-06-06 06:56:05 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1685243, 1686946 | ||
| Bug Blocks: | |||
|
Comment 1
Rich Megginson
2019-03-12 18:28:40 UTC
merged upstream https://github.com/openshift/origin-aggregated-logging/commit/c68eea3d43aa6a9f808974883a7c039150349a82 The fix is implemented in rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 - please verify that the fluentd image has this version (or later). There isn't rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 in v3.9.74. gem list |grep fluent fluent-mixin-config-placeholders (0.4.0) fluent-plugin-docker_metadata_filter (0.1.1) fluent-plugin-elasticsearch (1.17.2) fluent-plugin-flatten-hash (0.4.0) fluent-plugin-kubernetes_metadata_filter (1.0.3) fluent-plugin-record-modifier (0.6.2) fluent-plugin-remote-syslog (1.1) fluent-plugin-rewrite-tag-filter (1.5.6) fluent-plugin-secure-forward (0.4.5) fluent-plugin-systemd (0.0.11) fluent-plugin-viaq_data_model (0.0.14) fluentd (0.12.43) Moving to MODIFIED - the bz is fixed - now it is up to ART/RCM to ensure that there is an image built with the right rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 rpm . . . The new JSON log messages handle are merged into version v3.9.78 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0788 |