Bug 1512132

+++ This bug was initially created as a clone of Bug #1494912 +++

Description of problem:
As elastic recommend we should remove all _source for metrics index.
Currently only fields under collectd namespace are removed.

Version-Release number of selected component (if applicable):
3.6.1

How reproducible:
100%

Steps to Reproduce:
1.Send metrics data from oVirt to the openshift setup.
2.Check kibana discovery tag
3.

Actual results:
We see the records that are collected

Expected results:
No dana should appear under for metrics.

Additional info:

--- Additional comment from Shirly Radco on 2017-09-24 01:34 EDT ---



--- Additional comment from Rich Megginson on 2017-09-24 18:13:28 EDT ---

The source is not saved for the collectd fields.

Has something changed?

--- Additional comment from Shirly Radco on 2017-09-26 06:57:51 EDT ---

(In reply to Rich Megginson from comment #2)
> The source is not saved for the collectd fields.
> 
> Has something changed?

_source and _all fields should all be disabled for the the records in the indexes of project.ovirt-metrics* should be removed.

As recommended by https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html

Currently only collectd namespaces have source disabled.
The fields are enriched with pipline , kubernetes and ovirt metadata fields.
The _source and _all fields should be disabled for them as well.

For testing, You should not see any metrics related data in the discovery tab.
Only an option to create graphs in the visualize tab and dashboards.

See attachement with the fields that are still have _source field.

--- Additional comment from Lukas Svaty on 2017-10-09 02:31:17 EDT ---

Can we raise priority on this? 

This one is blocking verification process and QA automation regarding integration of oVirt and Viaq and is required by RHV-QA.

--- Additional comment from Rich Megginson on 2017-10-09 17:00:19 EDT ---

(In reply to Lukas Svaty from comment #4)
> Can we raise priority on this? 
> 
> This one is blocking verification process and QA automation regarding
> integration of oVirt and Viaq and is required by RHV-QA.

What is it blocking?

--- Additional comment from Lukas Svaty on 2017-10-10 02:18:41 EDT ---

see BZ#1494190 I don't think it is a good way to remove this record if it will cause data not being populated in Discovery tab. If it's a requirement from the elasticsearch side, we should find another way to query the data in Discovery tab.

--- Additional comment from Rich Megginson on 2017-10-10 10:20:24 EDT ---

@lvlcek - do you know of a good query, using curl and the query api, that can be used to determine if the collectd stats are being stored in Elasticsearch?

--- Additional comment from Shirly Radco on 2017-10-15 05:52:29 EDT ---

(In reply to Lukas Svaty from comment #6)
> see BZ#1494190 I don't think it is a good way to remove this record if it
> will cause data not being populated in Discovery tab. If it's a requirement
> from the elasticsearch side, we should find another way to query the data in
> Discovery tab.

Same as in other metrics solutions, like prometheus and graphite, the metrics samples are not required to be viewed per sample.

We should find a solution for qa testing metrics. But, we should not save the source and all fields.

Probably the best way would be pre-defined dashboards.