prometheus is not showing metrics Description of problem: prometheus is not showing metrics. Version-Release number of selected component (if applicable): OCP 4.3 How reproducible: Will share other details. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I am elaborating the issue: - All pods from openshift-monitoring were up and running. When we try to execute queries on prometheus console it was not showing the values giving no_data as answer. - Login to prometheus console was working successfully, Also checked the prometheus image was latest as per the OCP version 4.3. - Checked the targets page and all targets were showing status as UP. Nothing was unhealthy in targets page. - Checked the prometheus-proxy container logs and there was no error. - Checked thr curl command as below with kubeadmin user token and metrics from the node were exposed and showing data correctly. # curl -vk https://<node-name>:10250/metrics/cadvisor -H "Authorization: Bearer $TOKEN" - Also, checked below commands if there are values coming for the metrics if curl command runs for node-exporter container # oc exec pod/node-exporter-xxx -c node-exporter -- curl 127.0.0.1:9100/metrics | grep -i cpu The above command was working and showing values as output attached to the case. - Checked below command if the metrics are there when we curl to 9091 port from prometheus container. # oc exec prometheus-k8s-0 -c prometheus -- curl http://localhost:9090/metrics Above command worked to show metrics. - Tried restarting all the node-exporter pods and prometheus-k8s-0 and prometheus-k8s-1 pod but there were no metrics for below queries showing no_data. While checking below metrics we tried using private browser window (incognito mode) and logged in again with kubeadmin user. instance:node_cpu_utilisation:rate1m namespace:container_cpu_usage_seconds_total:sum_rate{namespace="openshift-monitoring"} instance:node_cpu:rate:sum instance:node_cpu:ratio