Bug 1686947 - Allow MERGE_JSON_LOG=true for indexing of JSON payload fields
Summary: Allow MERGE_JSON_LOG=true for indexing of JSON payload fields
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.z
Assignee: Rich Megginson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1685243 1686946
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-08 17:54 UTC by Rich Megginson
Modified: 2019-06-06 06:56 UTC (History)
7 users (show)

Fixed In Version: openshift3/logging-fluentd:v3.9.73-1
Doc Type: Bug Fix
Doc Text:
Cause: Using MERGE_JSON_LOG=true can create fields in the record which will cause schema and syntax violations in Elasticsearch. It can also create too many fields for Elasticsearch to handle without severe performance problems. Consequence: Fluentd reports error 400 sending records to Elasticsearch. Elasticsearch performance degrades. Fix: Allow users who experience these problems to tune their Fluentd to accomodate their log record fields. Result: Logs are ingested into Fluentd with no errors. Elasticsearch performance does not degrade.
Clone Of: 1686946
Environment:
Last Closed: 2019-06-06 06:56:05 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1557 0 None closed add support for new fluentd viaq filter undefined fields 2021-02-17 20:11:36 UTC
Red Hat Product Errata RHBA-2019:0788 0 None None None 2019-06-06 06:56:14 UTC

Comment 1 Rich Megginson 2019-03-12 18:28:40 UTC 
https://github.com/openshift/origin-aggregated-logging/pull/1557
Comment 2 Rich Megginson 2019-03-13 01:48:42 UTC 
merged upstream https://github.com/openshift/origin-aggregated-logging/commit/c68eea3d43aa6a9f808974883a7c039150349a82
Comment 3 Rich Megginson 2019-03-13 02:01:04 UTC 
The fix is implemented in rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 - please verify that the fluentd image has this version (or later).
Comment 5 Anping Li 2019-03-27 01:45:51 UTC 
There isn't rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 in v3.9.74.

gem list |grep fluent 
fluent-mixin-config-placeholders (0.4.0)
fluent-plugin-docker_metadata_filter (0.1.1)
fluent-plugin-elasticsearch (1.17.2)
fluent-plugin-flatten-hash (0.4.0)
fluent-plugin-kubernetes_metadata_filter (1.0.3)
fluent-plugin-record-modifier (0.6.2)
fluent-plugin-remote-syslog (1.1)
fluent-plugin-rewrite-tag-filter (1.5.6)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-systemd (0.0.11)
fluent-plugin-viaq_data_model (0.0.14)
fluentd (0.12.43)
Comment 7 Rich Megginson 2019-04-08 18:28:14 UTC 
Moving to MODIFIED - the bz is fixed - now it is up to ART/RCM to ensure that there is an image built with the right rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 rpm . . .
Comment 9 Anping Li 2019-04-22 13:54:16 UTC 
The new JSON log messages handle are merged into version v3.9.78
Comment 11 errata-xmlrpc 2019-06-06 06:56:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0788
Note You need to log in before you can comment on or make changes to this bug.