Description of problem: After upgrading cluster to 4.8 the `kube-state-metrics` doesn't export `persistentvolumeclaim` (kube_persistentvolumeclaim_labels) or `persistentvolume` (kube_persistentvolume_labels) labels anymore. It was working in previous OCP versions. Version-Release number of selected component (if applicable): OCP 4.8 How reproducible: Always Steps to Reproduce: 1. Check the `kube-state-metrics` deployment: ~~~ $ oc get deployment kube-state-metrics -n openshift-monitoring -o yaml ~~~ 2. `pods` and `node` labels are allowed, but not `namespace` ~~~ --metric-labels-allowlist=pods=[*],node=[*] ~~~ Actual results: Custom Monitoring and Alerting don't work correctly without the `persistent volume` or `persistent volume claim` labels. Expected results: Export the `persistent volume` and `persistent volume claim` labels in OCP 4.8 as in previous releases. Additional info: These metrics, in addition to `namespace` labels are used by the cost-management-metrics-operator which is needed for cost management in console.redhat.com
tested with 4.10.0-0.nightly-2021-10-13-001151, the fix is in the payload, bind PVs for prometheus pods, no issue with kube_persistentvolume_labels, but still can not see labels for persistentvolumeclaim. # oc -n openshift-monitoring get deployment kube-state-metrics -o yaml | grep metric-labels-allowlist - --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*] kube_persistentvolume_labels result, could see the volume labels kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-1", label_topology_kubernetes_io_zone="us-east-1b", namespace="openshift-monitoring", persistentvolume="pvc-9af60257-cf1c-4564-826e-ec4ce99268d8", service="kube-state-metrics"} 1 kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-1", label_topology_kubernetes_io_zone="us-east-1c", namespace="openshift-monitoring", persistentvolume="pvc-5ab7c7eb-3ed8-48db-97a1-7aa01f647832", service="kube-state-metrics"} 1 PVS prometheus-prometheus-k8s-0 has following labels # oc -n openshift-monitoring get pvc prometheus-prometheus-k8s-0 -o jsonpath="{.metadata.labels"} {"app":"prometheus","app.kubernetes.io/instance":"k8s","app.kubernetes.io/managed-by":"prometheus-operator","app.kubernetes.io/name":"prometheus","operator.prometheus.io/name":"k8s","operator.prometheus.io/shard":"0","prometheus":"k8s"} kube_persistentvolumeclaim_labels result, did not see the PVC labels kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-0", service="kube-state-metrics"} 1 kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-1", service="kube-state-metrics"} 1
Reassigning to Filip for investigation since the change seems correct at first sight but it looks like kube-state-metrics doesn't honor the "--metric-labels-allowlist=persistentvolumeclaims=[*]" flag.
KSM 2.2.1 has the fix[1] to enable full label metrics on PVC. [1] https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.1
tested with 4.10.0-0.nightly-2021-10-23-225921, could see PV/PVC labels from prometheus # oc -n openshift-monitoring exec -c kube-state-metrics kube-state-metrics-fdc48544d-m65cr -- kube-state-metrics --version kube-state-metrics, version v2.2.3 (branch: rhaos-4.10-rhel-8, revision: c9136b4) build user: root@9ec9a187e4e3 build date: 2021-10-21T14:28:54Z go version: go1.16.6 platform: linux/amd64 # oc -n openshift-monitoring get deployment kube-state-metrics -o yaml | grep metric-labels-allowlist - --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*] PV labels # oc get pv pvc-7e82f8ee-7b9f-4405-b00e-5f537fc81556 -o jsonpath="{.metadata.labels"} {"topology.kubernetes.io/region":"us-east-2","topology.kubernetes.io/zone":"us-east-2a"} kube_persistentvolume_labels result in prometheus kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-2", label_topology_kubernetes_io_zone="us-east-2a", namespace="openshift-monitoring", persistentvolume="pvc-7e82f8ee-7b9f-4405-b00e-5f537fc81556", service="kube-state-metrics"} 1 PVC labels # oc -n openshift-monitoring get pvc prometheus-prometheus-k8s-0 -o jsonpath="{.metadata.labels"} {"app":"prometheus","app.kubernetes.io/instance":"k8s","app.kubernetes.io/managed-by":"prometheus-operator","app.kubernetes.io/name":"prometheus","operator.prometheus.io/name":"k8s","operator.prometheus.io/shard":"0","prometheus":"k8s"} kube_persistentvolumeclaim_labels result in prometheus kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_app="prometheus", label_app_kubernetes_io_instance="k8s", label_app_kubernetes_io_managed_by="prometheus-operator", label_app_kubernetes_io_name="prometheus", label_operator_prometheus_io_name="k8s", label_operator_prometheus_io_shard="0", label_prometheus="k8s", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-0", service="kube-state-metrics"} 1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056