Bug 2011698

Summary: After upgrading cluster to 4.8 the kube-state-metrics service doesn't export namespace labels anymore
Product: OpenShift Container Platform Reporter: oarribas <oarribas>
Component: MonitoringAssignee: Jayapriya Pai <janantha>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.8CC: amuller, anpicker, aos-bugs, cruhm, erooth, mskarbek, oarribas, pnair, soconcar, spasquie
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Added namespace label in metric allow list which went away with KSM v2
 Story Points: ---
Clone Of:
: 2012039 (view as bug list) Environment:
Last Closed: 2022-03-10 16:17:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2012029    

Description oarribas 2021-10-07 06:56:20 UTC 
Description of problem:

After upgrading cluster to 4.8 the `kube-state-metrics` doesn't export `namespace` labels anymore. It was working in previous OCP versions.


In the `args` of the pod, there is the following:
~~~
--metric-labels-allowlist=pods=[*],node=[*]
~~~

The `namespaces` are missing in the allowed labels.



Version-Release number of selected component (if applicable):

OCP 4.8


How reproducible:

Always


Steps to Reproduce:
1. Check the `kube-state-metrics` deployment:
~~~
$ oc get deployment kube-state-metrics -n openshift-monitoring -o yaml
~~~

2. `pods` and `node` labels are allowed, but not `namespace`

~~~
--metric-labels-allowlist=pods=[*],node=[*]
~~~


Actual results:

Custom Monitoring and Alerting don't work correctly without the `namespaces` labels.


Expected results:

Export the `namespace` labels in OCP 4.8 as in previous releases.


Additional info:
Comment 4 Junqi Zhao 2021-10-08 09:42:50 UTC 
tested with 4.10.0-0.nightly-2021-10-08-050801, issue is fixed
for example, openshift-console has namespace labels
# oc get ns openshift-console -o jsonpath="{.metadata.labels"}
{"kubernetes.io/metadata.name":"openshift-console","network.openshift.io/policy-group":"console"}

seach with "kube_namespace_labels{namespace="openshift-console"}", we can see the namespace's labels 
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=kube_namespace_labels%7Bnamespace%3D%22openshift-console%22%7D' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "kube_namespace_labels",
          "container": "kube-rbac-proxy-main",
          "endpoint": "https-main",
          "job": "kube-state-metrics",
          "label_kubernetes_io_metadata_name": "openshift-console",
          "label_network_openshift_io_policy_group": "console",
          "namespace": "openshift-console",
          "service": "kube-state-metrics"
        },
        "value": [
          1633686095.574,
          "1"
        ]
      }
    ]
  }
Comment 8 errata-xmlrpc 2022-03-10 16:17:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056