Description of problem:
When deploying Prometheus on OCP 3.9 using openshift-ansible, the router's metrics are not available: the router metrics endpoint is protected and prometheus can't scrape it.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Deploy prometheus metrics on an OCP 3.9 cluster via openshift-ansible:
2. Check the kubernetes-service-endpoints target for the router metrics endpoint
level=debug ts=2018-04-09T11:18:56.431809488Z caller=scrape.go:676 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://192.168.55.143:1936/metrics msg="Scrape failed" err="server returned HTTP status 403 Forbidden"
Router metrics can be scraped by prometheus
This is reported upstream in https://github.com/openshift/origin/issues/17685
The upstream bug is fixed on master (upcoming 3.10).
Doc is LGTM
The Rarget Release is set to 3.11, I think it should be 3.10
We need new prometheus images to test this defect, the following configurations is not in /etc/prometheus/prometheus.yml of prometheus container
# Scrape config for the router
- job_name: 'openshift-router'
- role: endpoints
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
Right, there's a difference between the upstream issue that was focused on 'oc cluster up' + the example Prometheus template  and this BZ which targets OpenShift Ansible. IIUC the existing playbooks don't configure Prometheus to scrape the router endpoint: this is the configuration snippet that you're not getting currently. I'll address this.
That being said, the merged PR  is relevant for both cases.
I've checked further: with the current openshift/origin and openshift-ansible, Prometheus doesn't scrape the router's metrics because the router's service doesn't have the "prometheus.io/scrape: true" annotation anymore.
I've submitted https://github.com/openshift/openshift-ansible/pull/8512 for Prometheus to scrape the metrics.
https://github.com/openshift/openshift-ansible/pull/8512 has been merged.
clusterrole router-metrics is added in prometheus namespace, and router metrics could be accessed
Created attachment 1447321 [details]