Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2022612

Summary: no namespace field for "Kubernetes / Compute Resources / Namespace (Pods)" admin console dashboard
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Arunprasad Rajkumar <arajkuma>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.10CC: amuller, anpicker, aos-bugs, erooth, spasquie
Target Milestone: ---Keywords: Regression
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:26:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
"Kubernetes / Compute Resources / Namespace (Pods)" dashboard - 4.10 none

Description Junqi Zhao 2021-11-12 07:44:29 UTC
Created attachment 1841338 [details]
"Kubernetes / Compute Resources / Namespace (Pods)" dashboard - 4.10

Description of problem:
admin console "Kubernetes / Compute Resources / Namespace (Pods)" dashboard, no namespace field in the dashboard compared with 4.9.
checked grafana "Kubernetes / Compute Resources / Namespace (Pods)" dashboard, value is None for the namespace field
checked from 410 grafana-dashboard-k8s-resources-namespace configmap, it uses kube_namespace_created metrics, but no such metrics in cluster
                {
                    "allValue": null,
                    "current": {
                        "text": "",
                        "value": ""
                    },
                    "datasource": "$datasource",
                    "hide": 0,
                    "includeAll": false,
                    "label": null,
                    "multi": false,
                    "name": "namespace",
                    "options": [

                    ],
                    "query": "label_values(kube_namespace_created{cluster=\"$cluster\"}, namespace)",
                    "refresh": 2,
                    "regex": "",
                    "sort": 1,
                    "tagValuesQuery": "",
                    "tags": [

                    ],
                    "tagsQuery": "",
                    "type": "query",
                    "useTags": false
                }
            ]
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/label/__name__/values' | jq | grep kube_namespace_created
no result

4.9 used kube_pod_info
                {
                    "allValue": null,
                    "current": {
                        "text": "",
                        "value": ""
                    },
                    "datasource": "$datasource",
                    "hide": 0,
                    "includeAll": false,
                    "label": null,
                    "multi": false,
                    "name": "namespace",
                    "options": [

                    ],
                    "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
                    "refresh": 2,
                    "regex": "",
                    "sort": 1,
                    "tagValuesQuery": "",
                    "tags": [

                    ],
                    "tagsQuery": "",
                    "type": "query",
                    "useTags": false
                }
            ]
        }

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2021-11-11-170956
grafana version=7.5.11

How reproducible:
always

Steps to Reproduce:
1. admin console, go to "Observe -> Dashboards", choose "Kubernetes / Compute Resources / Namespace (Pods)" dashboard
2.
3.

Actual results:
no namespace field in the dashboard

Expected results:


Additional info:

Comment 5 Simon Pasquier 2021-11-12 09:11:00 UTC
the kube_*_created metrics aren't exposed by kube-state-metrics since 4.8 [1]. https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/669 is the upstream PR that introduced the use of kube_namespace_created metric (for performance reasons), we should probably to submit a change to use kube_namespace_info instead.

Reassigning to Arun since it's in his area of expertise :)

[1] https://github.com/openshift/cluster-monitoring-operator/blob/0ac1e0614f9dba6b49c708e82d9ed2abcfd77b9d/assets/kube-state-metrics/deployment.yaml#L41

Comment 6 Junqi Zhao 2021-11-12 10:06:21 UTC
the error with "Kubernetes / Compute Resources / Pod" dashboard, no Namespace/Pod fileds in the dashboard compared with 4.9.

Comment 8 Junqi Zhao 2021-11-12 10:11:39 UTC
"Kubernetes / Compute Resources / Pod" dashboard,
since namespace value is None, pod value is also None
# oc -n openshift-monitoring get cm grafana-dashboard-k8s-resources-pod -oyaml
...
                    "datasource": "$datasource",
                    "hide": 0,
                    "includeAll": false,
                    "label": null,
                    "multi": false,
                    "name": "namespace",
                    "options": [

                    ],
                    "query": "label_values(kube_namespace_created{cluster=\"$cluster\"}, namespace)",
                    "refresh": 2,
                    "regex": "",
                    "sort": 1,
                    "tagValuesQuery": "",
                    "tags": [

                    ],
                    "tagsQuery": "",
                    "type": "query",
                    "useTags": false
                }
...
                    "datasource": "$datasource",
                    "hide": 0,
                    "includeAll": false,
                    "label": null,
                    "multi": false,
                    "name": "pod",
                    "options": [

                    ],
                    "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=\"$namespace\"}, pod)",
                    "refresh": 2,
                    "regex": "",
                    "sort": 1,
                    "tagValuesQuery": "",
                    "tags": [

                    ]

Comment 9 Junqi Zhao 2021-11-12 10:12:23 UTC
(In reply to Junqi Zhao from comment #6)
> the error with "Kubernetes / Compute Resources / Pod" dashboard
same error with "Kubernetes / Compute Resources / Pod" dashboard

Comment 12 Junqi Zhao 2021-11-12 10:21:58 UTC
checked, kube_namespace_created is used in the following dashboards, need to change that
Dashboard: "Kubernetes / Compute Resources / Namespace (Pods)" , configmap: grafana-dashboard-k8s-resources-namespace
Dashboard: "Kubernetes / Compute Resources / Pod" , configmap: grafana-dashboard-k8s-resources-pod
Dashboard: "Kubernetes / Compute Resources / Workload" , configmap: grafana-dashboard-k8s-resources-workload

Comment 14 Junqi Zhao 2021-11-24 11:26:52 UTC
tested with PR, issue in https://bugzilla.redhat.com/show_bug.cgi?id=2022612#c12 is fixed, could see namespace field and other related fields
but there is one regression,
"Kubernetes / Compute Resources / Namespace (Workloads)" dashboard, configmap grafana-dashboard-k8s-resources-workloads-namespace

see from the picture,

query is : count(kube_pod_info{job="kube-state-metrics"}, cluster=""}) by (namespace)
response is:
{"status":"error","errorType":"bad_data","error":"1:55: parse error: unexpected \"=\" in aggregation"}

query should be 
count(kube_pod_info{job="kube-state-metrics", cluster=""}) by (namespace)

Comment 16 Junqi Zhao 2021-12-02 12:40:08 UTC
checked with 4.10.0-0.nightly-2021-12-01-164437, could see the namespace filed and other related fields, and regression issue in comment 14 is also fixed

Comment 19 errata-xmlrpc 2022-03-10 16:26:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056