Bug 2030347 - kube-state-metrics exposes metrics about resource annotations
Summary: kube-state-metrics exposes metrics about resource annotations
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.10.0
Assignee: Haoyu Sun
QA Contact: Junqi Zhao
Depends On:
TreeView+ depends on / blocked
Reported: 2021-12-08 14:23 UTC by Simon Pasquier
Modified: 2022-03-10 16:33 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-03-10 16:32:46 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1544 0 None open [WIP] Bug 2030347: kube-state-metrics exposes metrics about resource annotations 2022-01-24 17:04:03 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:33:11 UTC

Description Simon Pasquier 2021-12-08 14:23:02 UTC
Description of problem:
In OCP 4.10, kube-state-metrics exposes kube_.*_annotations timeseries for every resource. This increases the number of series stored by Prometheus in memory while providing little to no value (by default the series only contain the namespace and name labels).

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Check for kube_pod_annotations metric for instance in the OCP console's metrics page

Actual results:
It returns data.

Expected results:
No result.

Additional info:
Annotation metrics should have been disabled by https://github.com/openshift/cluster-monitoring-operator/pull/1426 but the regex had a typo (e.g. "kube_*_annotations" -> "kube_.+_annotations"). It would be good to have an e2e test to avoid future regressions.

Comment 5 Junqi Zhao 2022-01-27 08:23:18 UTC
checked with 4.10.0-0.nightly-2022-01-26-234447, kube_.*_annotations is dropped
# oc -n openshift-monitoring get deploy kube-state-metrics -oyaml | grep "metric-denylist"
        - --metric-denylist=kube_secret_labels,kube_.*_annotations
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/label/__name__/values' | jq | grep kube_.*_annotations
no result

# oc -n openshift-monitoring logs kube-state-metrics-79db5d9694-ppn4v | grep kube_.*_annotations
I0127 06:55:57.960214       1 main.go:133] metric allow-denylisting: Excluding the following lists that were on denylist: kube_pod_container_status_running, kube_pod_completion_time, kube_pod_status_scheduled, kube_secret_labels, kube_.+_metadata_resource_version, kube_pod_init_container_status_running, kube_.+_created, kube_replicaset_metadata_generation, kube_pod_init_container_status_terminated, kube_pod_container_status_terminated, kube_.*_annotations, kube_replicaset_status_observed_generation, kube_pod_restart_policy

Comment 8 errata-xmlrpc 2022-03-10 16:32:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.