Bug 1634036

Summary:	Per indice metrics missing in elasticsearch exporter
Product:	OpenShift Container Platform	Reporter:	Simon Reber <sreber>
Component:	Logging	Assignee:	Lukas Vlcek <lvlcek>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, gferrazs, jcantril, lvlcek, rmeggins, rszumski, rushil, vjaypurk
Target Milestone:	---
Target Release:	3.10.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1703118 (view as bug list)		Environment:
Last Closed:	2019-06-18 14:24:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1703118, 1703136

Description Simon Reber 2018-09-28 14:01:41 UTC

Created attachment 1488081 [details]
Output from both command when reproducing the issue. Showing indices available in elasticsearch and metrics exported

Description of problem:

Running `atomic-openshift-3.10.14-1.git.0.ba8ae6d.el7.x86_64` with registry.access.redhat.com/openshift3/logging-elasticsearch:v3.10.14 is failing to export per indice metrics.

As per https://github.com/vvanholl/elasticsearch-prometheus-exporter the exporter should actually include per indice metrics, but as shown in the attached file no data related to the specific indices can be found

Version-Release number of selected component (if applicable):

 - atomic-openshift-3.10.14-1.git.0.ba8ae6d.el7.x86_64

How reproducible:

 - Always

Steps to Reproduce:
1. Install Prometheus according https://docs.openshift.com/container-platform/3.10/install_config/cluster_metrics.html#openshift-prometheus-deploy
2. Deploy EFK stack as per https://docs.openshift.com/container-platform/3.10/install_config/aggregate_logging.html#deploying-the-efk-stack
3. Check indices available in `elasticsearch` with `oc exec $es_pod -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://logging-es:9200/_cat/indices?v`
4. Check Metrics reported by `elasticsearch` exporter: `curl -k -H "Authorization: Bearer $(oc sa get-token prometheus -n openshift-metrics)" https://<es-pod-ip>:4443/_prometheus/metrics`

Actual results:

No indice related metrics available/reported

Expected results:

Per indice metrics reported in order to use them further for charging, planning and monitoring purpose

Additional info:

Comment 1 Frederic Branczyk 2018-09-28 14:06:41 UTC

Changing this to the logging team as Prometheus just collects whatever the exporter gives it, and this seems to have shown already that the exporter is indeed not exporting the metrics.

Comment 2 Lukas Vlcek 2019-01-02 09:44:47 UTC

Indices level metrics are back in Prometheus exporter plugin. See https://github.com/vvanholl/elasticsearch-prometheus-exporter/pull/131
Next step is to upgrade to relevant ES plugin to have it in OpenShift logging.

Comment 12 Jeff Cantrill 2019-06-18 14:24:18 UTC

Closing CURRENTRELEASE as fixed in 3.11 and later.  Prior versions are based on earlier versions of Elasticsearch which we are moving away from.