Description of problem:
This might not even be a problem, but would like to understand if would this proceed as RFE maybe?
Elasticsearch and prometheus are installed with the standardinstallation of openshift. I expect
- prometheus could access the metrics of elasticsearch
- prometheus is configured to scap them
- alerts are configured (e.g. disc is getting full in one week)
- dashboards are present in grafana
For instance this project https://github.com/justwatchcom/elasticsearch_exporter is providing what we expect. We also expect this for fluentd and kibana.
From customer perspective:
I consider this is a bug. What is your roadmap to fix this bug? We/and our clients need the solution now. Can you provide us guidance and advices to work on this topic? How could we provide our solution upstream?
Version-Release number of selected component (if applicable):
Steps to Reproduce:
Each component is responsible for shipping their monitoring, so reassigning this to the logging component. As far as I am aware though the team has shipped scraping and alerting for 4.1, but I'd prefer if they would confirm that.
Metrics are available in 3.11, though during investigation of another issue I discovered we are unable to pull them through our proxy because of a missing switch. I will use this bz to fix that. I believe there may be a second issue, however, which we corrected in 4.x. I believe even if you provide the correct service account that you will not have properly signed certs unless you ignore who signed them. Logging creates its own certs and builds it's own truststore.
1. Define the service account in your inventory file(openshift_prometheus_namespace, openshift_logging_elasticsearch_prometheus_sa) which will be bound to this role: prometheus-metrics-viewer
2. Deploy logging using the 3.11 fix that will be associated with this bz
3. Retrieve metrics like: 'curl -k https://<logging-es-prometheus service>/_prometheus/metrics -H "Authorization : Bearer $sa_token"
I defer to the monitoring team how to configure prometheus as I'm unfamiliar with that end. Following is the documentation we presently have regarding metrics .
The metrics can be fetched using the token of serviceaccount system:serviceaccount:openshift-monitoring:prometheus-k8s. @Frederic, you are correct, to display the elasticsearch metrics and apply the rules, you need to provide rules files to prometheus. that appened automatically in 4.x.
I'd like close this bug as the elasticsearch can expose the metrics via token. For the further requirement, such display metrics in prometheus. please workaround yourself or file a RFE bug.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.