Bug 2012915

Summary:	kube_persistentvolumeclaim_labels and kube_persistentvolume_labels are missing in OCP 4.8 monitoring stack
Product:	OpenShift Container Platform	Reporter:	Michael Skarbek <mskarbek>
Component:	Monitoring	Assignee:	Arunprasad Rajkumar <arajkuma>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.8	CC:	amuller, anpicker, aos-bugs, arajkuma, cruhm, erooth, janantha, ncarmich, soconcar, spasquie, sthaha
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2015571 (view as bug list)		Environment:
Last Closed:	2022-03-10 16:18:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2015571

Description Michael Skarbek 2021-10-11 15:27:19 UTC

Description of problem:

After upgrading cluster to 4.8 the `kube-state-metrics` doesn't export `persistentvolumeclaim` (kube_persistentvolumeclaim_labels) or `persistentvolume` (kube_persistentvolume_labels) labels anymore. It was working in previous OCP versions.


Version-Release number of selected component (if applicable):

OCP 4.8


How reproducible:

Always


Steps to Reproduce:
1. Check the `kube-state-metrics` deployment:
~~~
$ oc get deployment kube-state-metrics -n openshift-monitoring -o yaml
~~~

2. `pods` and `node` labels are allowed, but not `namespace`

~~~
--metric-labels-allowlist=pods=[*],node=[*]
~~~

Actual results:

Custom Monitoring and Alerting don't work correctly without the `persistent volume` or `persistent volume claim` labels.


Expected results:

Export the `persistent volume` and `persistent volume claim` labels in OCP 4.8 as in previous releases.


Additional info:

These metrics, in addition to `namespace` labels are used by the cost-management-metrics-operator which is needed for cost management in console.redhat.com

Comment 5 Junqi Zhao 2021-10-13 04:54:22 UTC

tested with 4.10.0-0.nightly-2021-10-13-001151, the fix is in the payload, bind PVs for prometheus pods, no issue with kube_persistentvolume_labels, but still can not see labels for persistentvolumeclaim.
# oc -n openshift-monitoring get deployment kube-state-metrics -o yaml | grep metric-labels-allowlist
        - --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*]

kube_persistentvolume_labels result, could see the volume labels
kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-1", label_topology_kubernetes_io_zone="us-east-1b", namespace="openshift-monitoring", persistentvolume="pvc-9af60257-cf1c-4564-826e-ec4ce99268d8", service="kube-state-metrics"} 1
kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-1", label_topology_kubernetes_io_zone="us-east-1c", namespace="openshift-monitoring", persistentvolume="pvc-5ab7c7eb-3ed8-48db-97a1-7aa01f647832", service="kube-state-metrics"} 1

PVS prometheus-prometheus-k8s-0 has following labels
# oc -n openshift-monitoring get pvc prometheus-prometheus-k8s-0 -o jsonpath="{.metadata.labels"}
{"app":"prometheus","app.kubernetes.io/instance":"k8s","app.kubernetes.io/managed-by":"prometheus-operator","app.kubernetes.io/name":"prometheus","operator.prometheus.io/name":"k8s","operator.prometheus.io/shard":"0","prometheus":"k8s"}

kube_persistentvolumeclaim_labels result, did not see the PVC labels
kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-0", service="kube-state-metrics"} 1
kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-1", service="kube-state-metrics"} 1

Comment 6 Simon Pasquier 2021-10-13 09:25:00 UTC

Reassigning to Filip for investigation since the change seems correct at first sight but it looks like kube-state-metrics doesn't honor the "--metric-labels-allowlist=persistentvolumeclaims=[*]" flag.

Comment 8 Arunprasad Rajkumar 2021-10-19 14:18:38 UTC

KSM 2.2.1 has the fix[1] to enable full label metrics on PVC.

[1] https://github.com/kubernetes/kube-state-metrics/releases/tag/v2.2.1

Comment 13 Junqi Zhao 2021-10-25 03:33:36 UTC

tested with 4.10.0-0.nightly-2021-10-23-225921, could see PV/PVC labels from prometheus
# oc -n openshift-monitoring exec -c kube-state-metrics kube-state-metrics-fdc48544d-m65cr -- kube-state-metrics --version
kube-state-metrics, version v2.2.3 (branch: rhaos-4.10-rhel-8, revision: c9136b4)
  build user:       root@9ec9a187e4e3
  build date:       2021-10-21T14:28:54Z
  go version:       go1.16.6
  platform:         linux/amd64

# oc -n openshift-monitoring get deployment kube-state-metrics -o yaml | grep metric-labels-allowlist
        - --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*]

PV labels
# oc get pv pvc-7e82f8ee-7b9f-4405-b00e-5f537fc81556 -o jsonpath="{.metadata.labels"}
{"topology.kubernetes.io/region":"us-east-2","topology.kubernetes.io/zone":"us-east-2a"}

kube_persistentvolume_labels result in prometheus
kube_persistentvolume_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_topology_kubernetes_io_region="us-east-2", label_topology_kubernetes_io_zone="us-east-2a", namespace="openshift-monitoring", persistentvolume="pvc-7e82f8ee-7b9f-4405-b00e-5f537fc81556", service="kube-state-metrics"} 1

PVC labels
# oc -n openshift-monitoring get pvc prometheus-prometheus-k8s-0 -o jsonpath="{.metadata.labels"}
{"app":"prometheus","app.kubernetes.io/instance":"k8s","app.kubernetes.io/managed-by":"prometheus-operator","app.kubernetes.io/name":"prometheus","operator.prometheus.io/name":"k8s","operator.prometheus.io/shard":"0","prometheus":"k8s"}

kube_persistentvolumeclaim_labels result in prometheus
kube_persistentvolumeclaim_labels{container="kube-rbac-proxy-main", endpoint="https-main", job="kube-state-metrics", label_app="prometheus", label_app_kubernetes_io_instance="k8s", label_app_kubernetes_io_managed_by="prometheus-operator", label_app_kubernetes_io_name="prometheus", label_operator_prometheus_io_name="k8s", label_operator_prometheus_io_shard="0", label_prometheus="k8s", namespace="openshift-monitoring", persistentvolumeclaim="prometheus-prometheus-k8s-0", service="kube-state-metrics"}  1

Comment 16 errata-xmlrpc 2022-03-10 16:18:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056