Created attachment 1841338 [details] "Kubernetes / Compute Resources / Namespace (Pods)" dashboard - 4.10 Description of problem: admin console "Kubernetes / Compute Resources / Namespace (Pods)" dashboard, no namespace field in the dashboard compared with 4.9. checked grafana "Kubernetes / Compute Resources / Namespace (Pods)" dashboard, value is None for the namespace field checked from 410 grafana-dashboard-k8s-resources-namespace configmap, it uses kube_namespace_created metrics, but no such metrics in cluster { "allValue": null, "current": { "text": "", "value": "" }, "datasource": "$datasource", "hide": 0, "includeAll": false, "label": null, "multi": false, "name": "namespace", "options": [ ], "query": "label_values(kube_namespace_created{cluster=\"$cluster\"}, namespace)", "refresh": 2, "regex": "", "sort": 1, "tagValuesQuery": "", "tags": [ ], "tagsQuery": "", "type": "query", "useTags": false } ] # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/label/__name__/values' | jq | grep kube_namespace_created no result 4.9 used kube_pod_info { "allValue": null, "current": { "text": "", "value": "" }, "datasource": "$datasource", "hide": 0, "includeAll": false, "label": null, "multi": false, "name": "namespace", "options": [ ], "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)", "refresh": 2, "regex": "", "sort": 1, "tagValuesQuery": "", "tags": [ ], "tagsQuery": "", "type": "query", "useTags": false } ] } Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2021-11-11-170956 grafana version=7.5.11 How reproducible: always Steps to Reproduce: 1. admin console, go to "Observe -> Dashboards", choose "Kubernetes / Compute Resources / Namespace (Pods)" dashboard 2. 3. Actual results: no namespace field in the dashboard Expected results: Additional info:
the kube_*_created metrics aren't exposed by kube-state-metrics since 4.8 [1]. https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/669 is the upstream PR that introduced the use of kube_namespace_created metric (for performance reasons), we should probably to submit a change to use kube_namespace_info instead. Reassigning to Arun since it's in his area of expertise :) [1] https://github.com/openshift/cluster-monitoring-operator/blob/0ac1e0614f9dba6b49c708e82d9ed2abcfd77b9d/assets/kube-state-metrics/deployment.yaml#L41
the error with "Kubernetes / Compute Resources / Pod" dashboard, no Namespace/Pod fileds in the dashboard compared with 4.9.
"Kubernetes / Compute Resources / Pod" dashboard, since namespace value is None, pod value is also None # oc -n openshift-monitoring get cm grafana-dashboard-k8s-resources-pod -oyaml ... "datasource": "$datasource", "hide": 0, "includeAll": false, "label": null, "multi": false, "name": "namespace", "options": [ ], "query": "label_values(kube_namespace_created{cluster=\"$cluster\"}, namespace)", "refresh": 2, "regex": "", "sort": 1, "tagValuesQuery": "", "tags": [ ], "tagsQuery": "", "type": "query", "useTags": false } ... "datasource": "$datasource", "hide": 0, "includeAll": false, "label": null, "multi": false, "name": "pod", "options": [ ], "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=\"$namespace\"}, pod)", "refresh": 2, "regex": "", "sort": 1, "tagValuesQuery": "", "tags": [ ]
(In reply to Junqi Zhao from comment #6) > the error with "Kubernetes / Compute Resources / Pod" dashboard same error with "Kubernetes / Compute Resources / Pod" dashboard
checked, kube_namespace_created is used in the following dashboards, need to change that Dashboard: "Kubernetes / Compute Resources / Namespace (Pods)" , configmap: grafana-dashboard-k8s-resources-namespace Dashboard: "Kubernetes / Compute Resources / Pod" , configmap: grafana-dashboard-k8s-resources-pod Dashboard: "Kubernetes / Compute Resources / Workload" , configmap: grafana-dashboard-k8s-resources-workload
tested with PR, issue in https://bugzilla.redhat.com/show_bug.cgi?id=2022612#c12 is fixed, could see namespace field and other related fields but there is one regression, "Kubernetes / Compute Resources / Namespace (Workloads)" dashboard, configmap grafana-dashboard-k8s-resources-workloads-namespace see from the picture, query is : count(kube_pod_info{job="kube-state-metrics"}, cluster=""}) by (namespace) response is: {"status":"error","errorType":"bad_data","error":"1:55: parse error: unexpected \"=\" in aggregation"} query should be count(kube_pod_info{job="kube-state-metrics", cluster=""}) by (namespace)
checked with 4.10.0-0.nightly-2021-12-01-164437, could see the namespace filed and other related fields, and regression issue in comment 14 is also fixed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056