Description of problem: After upgrading cluster to 4.8 the `kube-state-metrics` doesn't export `namespace` labels anymore. It was working in previous OCP versions. In the `args` of the pod, there is the following: ~~~ --metric-labels-allowlist=pods=[*],node=[*] ~~~ The `namespaces` are missing in the allowed labels. Version-Release number of selected component (if applicable): OCP 4.8 How reproducible: Always Steps to Reproduce: 1. Check the `kube-state-metrics` deployment: ~~~ $ oc get deployment kube-state-metrics -n openshift-monitoring -o yaml ~~~ 2. `pods` and `node` labels are allowed, but not `namespace` ~~~ --metric-labels-allowlist=pods=[*],node=[*] ~~~ Actual results: Custom Monitoring and Alerting don't work correctly without the `namespaces` labels. Expected results: Export the `namespace` labels in OCP 4.8 as in previous releases. Additional info:
tested with 4.10.0-0.nightly-2021-10-08-050801, issue is fixed for example, openshift-console has namespace labels # oc get ns openshift-console -o jsonpath="{.metadata.labels"} {"kubernetes.io/metadata.name":"openshift-console","network.openshift.io/policy-group":"console"} seach with "kube_namespace_labels{namespace="openshift-console"}", we can see the namespace's labels # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=kube_namespace_labels%7Bnamespace%3D%22openshift-console%22%7D' | jq { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "kube_namespace_labels", "container": "kube-rbac-proxy-main", "endpoint": "https-main", "job": "kube-state-metrics", "label_kubernetes_io_metadata_name": "openshift-console", "label_network_openshift_io_policy_group": "console", "namespace": "openshift-console", "service": "kube-state-metrics" }, "value": [ 1633686095.574, "1" ] } ] }
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056