Bug 2012915 - kube_persistentvolumeclaim_labels and kube_persistentvolume_labels are missing in OCP 4.8 monitoring stack
Summary: kube_persistentvolumeclaim_labels and kube_persistentvolume_labels are missin...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.10.0
Assignee: Arunprasad Rajkumar
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 2015571
TreeView+ depends on / blocked
 
Reported: 2021-10-11 15:27 UTC by Michael Skarbek
Modified: 2022-10-18 03:24 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2015571 (view as bug list)
Environment:
Last Closed: 2022-03-10 16:18:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1424 0 None open Bug 2012915: add kube_persistentvolumeclaim_labels and kube_persistentvolume_labels 2021-10-12 05:50:52 UTC
Github openshift cluster-monitoring-operator pull 1452 0 None open Bug 2012915: Bump kube-state-metrics to v2.2.3 2021-10-21 15:04:43 UTC
Github openshift kube-state-metrics pull 62 0 None open Bug 2012915: Bump to v2.2.3 2021-10-21 11:10:17 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:19:09 UTC

Description Michael Skarbek 2021-10-11 15:27:19 UTC
Description of problem:

After upgrading cluster to 4.8 the `kube-state-metrics` doesn't export `persistentvolumeclaim` (kube_persistentvolumeclaim_labels) or `persistentvolume` (kube_persistentvolume_labels) labels anymore. It was working in previous OCP versions.


Version-Release number of selected component (if applicable):

OCP 4.8


How reproducible:

Always


Steps to Reproduce:
1. Check the `kube-state-metrics` deployment:
~~~
$ oc get deployment kube-state-metrics -n openshift-monitoring -o yaml
~~~

2. `pods` and `node` labels are allowed, but not `namespace`

~~~
--metric-labels-allowlist=pods=[*],node=[*]
~~~

Actual results:

Custom Monitoring and Alerting don't work correctly without the `persistent volume` or `persistent volume claim` labels.


Expected results:

Export the `persistent volume` and `persistent volume claim` labels in OCP 4.8 as in previous releases.


Additional info:

These metrics, in addition to `namespace` labels are used by the cost-management-metrics-operator which is needed for cost management in console.redhat.com

Comment 5 Junqi Zhao 2021-10-13 04:54:22 UTC
tested with 4.10.0-0.nightly-2021-10-13-001151, the fix is in the payload, bind PVs for prometheus pods, no issue with kube_persistentvolume_labels, but still can not see labels for persistentvolumeclaim.
# oc -n openshift-monitoring get deployment kube-state-metrics -o yaml | grep metric-labels-allowlist
        - --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*]

kube_persistentvolume_labels result, could see the volume labels
kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-1", label_topology_kubernetes_io_zone="us-east-1b", namespace="openshift-monitoring", persistentvolume="pvc-9af60257-cf1c-4564-826e-ec4ce99268d8", service="kube-state-metrics"} 1
kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-1", label_topology_kubernetes_io_zone="us-east-1c", namespace="openshift-monitoring", persistentvolume="pvc-5ab7c7eb-3ed8-48db-97a1-7aa01f647832", service="kube-state-metrics"} 1

PVS prometheus-prometheus-k8s-0 has following labels
# oc -n openshift-monitoring get pvc prometheus-prometheus-k8s-0 -o jsonpath="{.metadata.labels"}
{"app":"prometheus","app.kubernetes.io/instance":"k8s","app.kubernetes.io/managed-by":"prometheus-operator","app.kubernetes.io/name":"prometheus","operator.prometheus.io/name":"k8s","operator.prometheus.io/shard":"0","prometheus":"k8s"}

kube_persistentvolumeclaim_labels result, did not see the PVC labels
kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-0", service="kube-state-metrics"} 1
kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-1", service="kube-state-metrics"} 1

Comment 6 Simon Pasquier 2021-10-13 09:25:00 UTC
Reassigning to Filip for investigation since the change seems correct at first sight but it looks like kube-state-metrics doesn't honor the "--metric-labels-allowlist=persistentvolumeclaims=[*]" flag.

Comment 8 Arunprasad Rajkumar 2021-10-19 14:18:38 UTC
KSM 2.2.1 has the fix[1] to enable full label metrics on PVC.

[1] https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.1

Comment 13 Junqi Zhao 2021-10-25 03:33:36 UTC
tested with 4.10.0-0.nightly-2021-10-23-225921, could see PV/PVC labels from prometheus
# oc -n openshift-monitoring exec -c kube-state-metrics kube-state-metrics-fdc48544d-m65cr -- kube-state-metrics --version
kube-state-metrics, version v2.2.3 (branch: rhaos-4.10-rhel-8, revision: c9136b4)
  build user:       root@9ec9a187e4e3
  build date:       2021-10-21T14:28:54Z
  go version:       go1.16.6
  platform:         linux/amd64

# oc -n openshift-monitoring get deployment kube-state-metrics -o yaml | grep metric-labels-allowlist
        - --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*]

PV labels
# oc get pv pvc-7e82f8ee-7b9f-4405-b00e-5f537fc81556 -o jsonpath="{.metadata.labels"}
{"topology.kubernetes.io/region":"us-east-2","topology.kubernetes.io/zone":"us-east-2a"}

kube_persistentvolume_labels result in prometheus
kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-2", label_topology_kubernetes_io_zone="us-east-2a", namespace="openshift-monitoring", persistentvolume="pvc-7e82f8ee-7b9f-4405-b00e-5f537fc81556", service="kube-state-metrics"} 1

PVC labels
# oc -n openshift-monitoring get pvc prometheus-prometheus-k8s-0 -o jsonpath="{.metadata.labels"}
{"app":"prometheus","app.kubernetes.io/instance":"k8s","app.kubernetes.io/managed-by":"prometheus-operator","app.kubernetes.io/name":"prometheus","operator.prometheus.io/name":"k8s","operator.prometheus.io/shard":"0","prometheus":"k8s"}

kube_persistentvolumeclaim_labels result in prometheus
kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_app="prometheus", label_app_kubernetes_io_instance="k8s", label_app_kubernetes_io_managed_by="prometheus-operator", label_app_kubernetes_io_name="prometheus", label_operator_prometheus_io_name="k8s", label_operator_prometheus_io_shard="0", label_prometheus="k8s", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-0", service="kube-state-metrics"}  1

Comment 16 errata-xmlrpc 2022-03-10 16:18:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.