Bug 1512132 - metrics indexes should not save the _source
Summary: metrics indexes should not save the _source
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.6.z
Assignee: Rich Megginson
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On: 1494912
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 23:48 UTC by Rich Megginson
Modified: 2017-12-14 21:02 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The _source was not disabled for metrics records being stored in Elasticsearch. Consequence: The records were taking up much more resources (cpu, ram, disk) than necessary. Fix: Completely disable the _source for project.ovirt-metrics* records. Result: Metrics records are much smaller and require fewer resources to handle.
Clone Of: 1494912
Environment:
Last Closed: 2017-12-14 21:02:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ViaQ elasticsearch-templates pull 64 0 None None None 2017-11-10 23:48:10 UTC
Github openshift origin-aggregated-logging pull 783 0 None None None 2017-11-13 01:39:13 UTC
Red Hat Product Errata RHBA-2017:3438 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Rich Megginson 2017-11-10 23:48:11 UTC
+++ This bug was initially created as a clone of Bug #1494912 +++

Description of problem:
As elastic recommend we should remove all _source for metrics index.
Currently only fields under collectd namespace are removed.

Version-Release number of selected component (if applicable):
3.6.1

How reproducible:
100%

Steps to Reproduce:
1.Send metrics data from oVirt to the openshift setup.
2.Check kibana discovery tag
3.

Actual results:
We see the records that are collected

Expected results:
No dana should appear under for metrics.

Additional info:

--- Additional comment from Shirly Radco on 2017-09-24 01:34 EDT ---



--- Additional comment from Rich Megginson on 2017-09-24 18:13:28 EDT ---

The source is not saved for the collectd fields.

Has something changed?

--- Additional comment from Shirly Radco on 2017-09-26 06:57:51 EDT ---

(In reply to Rich Megginson from comment #2)
> The source is not saved for the collectd fields.
> 
> Has something changed?

_source and _all fields should all be disabled for the the records in the indexes of project.ovirt-metrics* should be removed.

As recommended by https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html

Currently only collectd namespaces have source disabled.
The fields are enriched with pipline , kubernetes and ovirt metadata fields.
The _source and _all fields should be disabled for them as well.

For testing, You should not see any metrics related data in the discovery tab.
Only an option to create graphs in the visualize tab and dashboards.

See attachement with the fields that are still have _source field.

--- Additional comment from Lukas Svaty on 2017-10-09 02:31:17 EDT ---

Can we raise priority on this? 

This one is blocking verification process and QA automation regarding integration of oVirt and Viaq and is required by RHV-QA.

--- Additional comment from Rich Megginson on 2017-10-09 17:00:19 EDT ---

(In reply to Lukas Svaty from comment #4)
> Can we raise priority on this? 
> 
> This one is blocking verification process and QA automation regarding
> integration of oVirt and Viaq and is required by RHV-QA.

What is it blocking?

--- Additional comment from Lukas Svaty on 2017-10-10 02:18:41 EDT ---

see BZ#1494190 I don't think it is a good way to remove this record if it will cause data not being populated in Discovery tab. If it's a requirement from the elasticsearch side, we should find another way to query the data in Discovery tab.

--- Additional comment from Rich Megginson on 2017-10-10 10:20:24 EDT ---

@lvlcek - do you know of a good query, using curl and the query api, that can be used to determine if the collectd stats are being stored in Elasticsearch?

--- Additional comment from Shirly Radco on 2017-10-15 05:52:29 EDT ---

(In reply to Lukas Svaty from comment #6)
> see BZ#1494190 I don't think it is a good way to remove this record if it
> will cause data not being populated in Discovery tab. If it's a requirement
> from the elasticsearch side, we should find another way to query the data in
> Discovery tab.

Same as in other metrics solutions, like prometheus and graphite, the metrics samples are not required to be viewed per sample.

We should find a solution for qa testing metrics. But, we should not save the source and all fields.

Probably the best way would be pre-defined dashboards.

Comment 1 Rich Megginson 2017-11-15 18:05:52 UTC
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=625426
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-16739-20171115174931
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.173.0.63
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.173.0.63-11
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6

Comment 2 Wei Sun 2017-11-29 02:34:07 UTC
Hi Lukas,
Could you help check if this bug could be verified?

Thanks!

Comment 3 Lukas Svaty 2017-11-29 10:24:06 UTC
verified in openshift-ansible-3.6.173.0.75-1.git.0.0a44128.el7.noarch

Comment 6 errata-xmlrpc 2017-12-14 21:02:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438


Note You need to log in before you can comment on or make changes to this bug.