Bug 1966126 - root_ca_cert_publisher_sync_duration_seconds metric can have an excessive cardinality
Summary: root_ca_cert_publisher_sync_duration_seconds metric can have an excessive car...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Sergiusz Urbaniak
QA Contact: liyao
Depends On:
TreeView+ depends on / blocked
Reported: 2021-05-31 12:39 UTC by Simon Pasquier
Modified: 2021-07-27 23:10 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-07-27 23:10:35 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-controller-manager-operator pull 526 0 None open Bug 1966126: manifests: drop root_ca_cert_publisher_sync_duration_seconds metric 2021-06-01 13:13:19 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:10:48 UTC

Description Simon Pasquier 2021-05-31 12:39:39 UTC
Description of problem:
The root_ca_cert_publisher_sync_duration_seconds metric tracks the sync duration in the root CA cert publisher per code and namespace [1]. The namespace label is problematic because series for a given namespace will continue to be exposed even after the namespace has been deleted. On clusters with high churn of projects/namespaces (e.g. CI cluster [2]), it can lead kube-controller-manager to expose more than 100,000 series to Prometheus which is in the .

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Execute the following PromQL query in the Prometheus UI
2. Create and delete hundred projects 
  for i in $(seq 0 99); do oc new-project "project-${i}"; done
  for i in $(seq 0 99); do oc delete "project-${i}"; done
3. Execute the same PromQL query.
4. Execute the following PromQL query:

Actual results:
The count of series for root_ca_cert_publisher_sync_duration_seconds_bucket has increased and stays the same even though the projects have been deleted.

The last PromQL query returns result for a project that no longer exists.

Expected results:

Series for projects/namespaces that no longer exist shouldn't be exposed.

Additional info:

The cardinality issue was discussed in the upstream PR but AFAICT it wasn't flagged as a big concern because other metrics have higher numbers of metrics and nobody noticed the impact caused by namespace churn.

[1] https://github.com/kubernetes/kubernetes/pull/98731
[2] https://prometheus-k8s-openshift-monitoring.apps.build01.ci.devcluster.openshift.com/tsdb-status

Comment 2 Maciej Szulik 2021-05-31 15:02:11 UTC
This one falls under the sig-auth, so sending over to auth team to investigate.

Comment 3 Sergiusz Urbaniak 2021-06-01 14:56:07 UTC
as discussed OOB:
- shortterm: we'll disable the metric downstream for the time being
- upstream: we'll suggest dropping the namespace label to reduce series churn.

Comment 9 errata-xmlrpc 2021-07-27 23:10:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.