Bug 1686947 - Allow MERGE_JSON_LOG=true for indexing of JSON payload fields
Summary: Allow MERGE_JSON_LOG=true for indexing of JSON payload fields
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.z
Assignee: Rich Megginson
QA Contact: Anping Li
Depends On: 1685243 1686946
TreeView+ depends on / blocked
Reported: 2019-03-08 17:54 UTC by Rich Megginson
Modified: 2019-06-06 06:56 UTC (History)
7 users (show)

Fixed In Version: openshift3/logging-fluentd:v3.9.73-1
Doc Type: Bug Fix
Doc Text:
Cause: Using MERGE_JSON_LOG=true can create fields in the record which will cause schema and syntax violations in Elasticsearch. It can also create too many fields for Elasticsearch to handle without severe performance problems. Consequence: Fluentd reports error 400 sending records to Elasticsearch. Elasticsearch performance degrades. Fix: Allow users who experience these problems to tune their Fluentd to accomodate their log record fields. Result: Logs are ingested into Fluentd with no errors. Elasticsearch performance does not degrade.
Clone Of: 1686946
Last Closed: 2019-06-06 06:56:05 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1557 None None None 2019-03-12 18:28:40 UTC
Red Hat Product Errata RHBA-2019:0788 None None None 2019-06-06 06:56:14 UTC

Comment 3 Rich Megginson 2019-03-13 02:01:04 UTC
The fix is implemented in rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 - please verify that the fluentd image has this version (or later).

Comment 5 Anping Li 2019-03-27 01:45:51 UTC
There isn't rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 in v3.9.74.

gem list |grep fluent 
fluent-mixin-config-placeholders (0.4.0)
fluent-plugin-docker_metadata_filter (0.1.1)
fluent-plugin-elasticsearch (1.17.2)
fluent-plugin-flatten-hash (0.4.0)
fluent-plugin-kubernetes_metadata_filter (1.0.3)
fluent-plugin-record-modifier (0.6.2)
fluent-plugin-remote-syslog (1.1)
fluent-plugin-rewrite-tag-filter (1.5.6)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-systemd (0.0.11)
fluent-plugin-viaq_data_model (0.0.14)
fluentd (0.12.43)

Comment 7 Rich Megginson 2019-04-08 18:28:14 UTC
Moving to MODIFIED - the bz is fixed - now it is up to ART/RCM to ensure that there is an image built with the right rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 rpm . . .

Comment 9 Anping Li 2019-04-22 13:54:16 UTC
The new JSON log messages handle are merged into version v3.9.78

Comment 11 errata-xmlrpc 2019-06-06 06:56:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.