Bug 1995699 - Get insights on series churn during upgrades
Summary: Get insights on series churn during upgrades
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.z
Assignee: Arunprasad Rajkumar
QA Contact: Junqi Zhao
: 1998969 (view as bug list)
Depends On: 1995695
TreeView+ depends on / blocked
Reported: 2021-08-19 15:53 UTC by Damien Grisonnet
Modified: 2021-09-20 13:38 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1995695
Last Closed: 2021-09-14 06:57:48 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1349 0 None None None 2021-08-31 06:04:47 UTC
Red Hat Product Errata RHBA-2021:3429 0 None None None 2021-09-14 06:58:07 UTC

Description Damien Grisonnet 2021-08-19 15:53:15 UTC
+++ This bug was initially created as a clone of Bug #1995695 +++

Description of problem:

We want to gather information about series churn during upgrades since it will be helpful to know the additional memory usage that Prometheus will have to handle after the update. This may also be helpful to detect high-cardinality metrics.

It would be great to gather this information via Telemetry to have this insight on customer clusters.

To do that, we could use the `scrape_series_added` and `scrape_samples_scraped` metrics from Prometheus. More information about this can be found in this blog post: https://www.robustperception.io/finding-churning-targets-in-prometheus-with-scrape_series_added.

In addition, this information will be very useful to future resiliency improvements since based on the data gathered via telemetry, we will be able to set sane limits to Prometheus that would prevent any malicious target to cause trouble.

Comment 1 Arunprasad Rajkumar 2021-08-31 06:01:05 UTC
*** Bug 1998969 has been marked as a duplicate of this bug. ***

Comment 9 errata-xmlrpc 2021-09-14 06:57:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.