Bug 1494912 - metrics indexes should not save the _source
Summary: metrics indexes should not save the _source
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.7.z
Assignee: Rich Megginson
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On:
Blocks: 1512132
TreeView+ depends on / blocked
 
Reported: 2017-09-24 05:33 UTC by Shirly Radco
Modified: 2017-12-18 13:22 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The _source was not disabled for metrics records being stored in Elasticsearch. Consequence: The records were taking up much more resources (cpu, ram, disk) than necessary. Fix: Completely disable the _source for project.ovirt-metrics* records. Result: Metrics records are much smaller and require fewer resources to handle.
Clone Of:
: 1512132 (view as bug list)
Environment:
Last Closed: 2017-12-18 13:22:48 UTC
Target Upstream Version:


Attachments (Terms of Use)
kibana_screenshot (306.82 KB, image/png)
2017-09-24 05:33 UTC, Shirly Radco
no flags Details
kibana_screenshot2 (232.83 KB, image/png)
2017-09-24 05:34 UTC, Shirly Radco
no flags Details


Links
System ID Priority Status Summary Last Updated
Github ViaQ elasticsearch-templates pull 64 'None' closed Bug 1494912 - metrics indexes sould not save the _source 2020-09-22 18:03:15 UTC
Github openshift origin-aggregated-logging pull 780 'None' closed Bug 1494912 - metrics indexes should not save the _source 2020-09-22 18:03:14 UTC
Red Hat Product Errata RHBA-2017:3464 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.7 bug fix and enhancement update 2017-12-18 18:22:05 UTC

Description Shirly Radco 2017-09-24 05:33:38 UTC
Created attachment 1330072 [details]
kibana_screenshot

Description of problem:
As elastic recommend we should remove all _source for metrics index.
Currently only fields under collectd namespace are removed.

Version-Release number of selected component (if applicable):
3.6.1

How reproducible:
100%

Steps to Reproduce:
1.Send metrics data from oVirt to the openshift setup.
2.Check kibana discovery tag
3.

Actual results:
We see the records that are collected

Expected results:
No dana should appear under for metrics.

Additional info:

Comment 1 Shirly Radco 2017-09-24 05:34:06 UTC
Created attachment 1330073 [details]
kibana_screenshot2

Comment 2 Rich Megginson 2017-09-24 22:13:28 UTC
The source is not saved for the collectd fields.

Has something changed?

Comment 3 Shirly Radco 2017-09-26 10:57:51 UTC
(In reply to Rich Megginson from comment #2)
> The source is not saved for the collectd fields.
> 
> Has something changed?

_source and _all fields should all be disabled for the the records in the indexes of project.ovirt-metrics* should be removed.

As recommended by https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html

Currently only collectd namespaces have source disabled.
The fields are enriched with pipline , kubernetes and ovirt metadata fields.
The _source and _all fields should be disabled for them as well.

For testing, You should not see any metrics related data in the discovery tab.
Only an option to create graphs in the visualize tab and dashboards.

See attachement with the fields that are still have _source field.

Comment 4 Lukas Svaty 2017-10-09 06:31:17 UTC
Can we raise priority on this? 

This one is blocking verification process and QA automation regarding integration of oVirt and Viaq and is required by RHV-QA.

Comment 5 Rich Megginson 2017-10-09 21:00:19 UTC
(In reply to Lukas Svaty from comment #4)
> Can we raise priority on this? 
> 
> This one is blocking verification process and QA automation regarding
> integration of oVirt and Viaq and is required by RHV-QA.

What is it blocking?

Comment 6 Lukas Svaty 2017-10-10 06:18:41 UTC
see BZ#1494190 I don't think it is a good way to remove this record if it will cause data not being populated in Discovery tab. If it's a requirement from the elasticsearch side, we should find another way to query the data in Discovery tab.

Comment 7 Rich Megginson 2017-10-10 14:20:24 UTC
@lvlcek - do you know of a good query, using curl and the query api, that can be used to determine if the collectd stats are being stored in Elasticsearch?

Comment 8 Shirly Radco 2017-10-15 09:52:29 UTC
(In reply to Lukas Svaty from comment #6)
> see BZ#1494190 I don't think it is a good way to remove this record if it
> will cause data not being populated in Discovery tab. If it's a requirement
> from the elasticsearch side, we should find another way to query the data in
> Discovery tab.

Same as in other metrics solutions, like prometheus and graphite, the metrics samples are not required to be viewed per sample.

We should find a solution for qa testing metrics. But, we should not save the source and all fields.

Probably the best way would be pre-defined dashboards.

Comment 9 openshift-github-bot 2017-11-12 22:46:00 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/8bf4db038ca9d0a39b499a760ea1891535ab7e84
Bug 1494912 - metrics indexes should not save the _source

https://bugzilla.redhat.com/show_bug.cgi?id=1494912
Completely disable the _source and _all for ovirt-metrics-*

https://github.com/openshift/origin-aggregated-logging/commit/ef1639722a466254077161d5b22923fc6be102b8
Merge pull request #780 from richm/viaq-0.0.12

Automatic merge from submit-queue.

Bug 1494912 - metrics indexes should not save the _source

https://bugzilla.redhat.com/show_bug.cgi?id=1494912
Completely disable the _source and _all for ovirt-metrics-*
/test

Comment 11 Anping Li 2017-11-16 11:02:35 UTC
@Shirly, OCP QE haven no experience to setup  oVirt Env. Is there an oVirt Env for sharing?

Comment 12 Rich Megginson 2017-11-17 02:10:21 UTC
I think RHV QE should handle this bug.

Comment 14 Wei Sun 2017-11-29 02:34:38 UTC
Hi Lukas,
Could you help check if this bug could be verified?

Thanks!

Comment 15 Lukas Svaty 2017-11-29 10:22:19 UTC
verified in openshift-ansible-3.6.173.0.75-1.git.0.0a44128.el7.noarch

Comment 18 errata-xmlrpc 2017-12-18 13:22:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3464


Note You need to log in before you can comment on or make changes to this bug.