Bug 1995695

Summary:	Get insights on series churn during upgrades
Product:	OpenShift Container Platform	Reporter:	Damien Grisonnet <dgrisonn>
Component:	Monitoring	Assignee:	Arunprasad Rajkumar <arajkuma>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.9	CC:	amuller, anpicker, aos-bugs, erooth, spasquie
Target Milestone:	---	Flags:	arajkuma: needinfo-
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1995699 1998969 (view as bug list)		Environment:
Last Closed:	2021-10-18 17:47:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1995699, 1998969

Description Damien Grisonnet 2021-08-19 15:51:28 UTC

Description of problem:

We want to gather information about series churn during upgrades since it will be helpful to know the additional memory usage that Prometheus will have to handle after the update. This may also be helpful to detect high-cardinality metrics.

It would be great to gather this information via Telemetry to have this insight on customer clusters.

To do that, we could use the `scrape_series_added` and `scrape_samples_scraped` metrics from Prometheus. More information about this can be found in this blog post: https://www.robustperception.io/finding-churning-targets-in-prometheus-with-scrape_series_added.

In addition, this information will be very useful to future resiliency improvements since based on the data gathered via telemetry, we will be able to set sane limits to Prometheus that would prevent any malicious target to cause trouble.

Comment 1 Arunprasad Rajkumar 2021-08-20 08:06:49 UTC

@spasquie suggested to go with `scrape_samples_post_metric_relabeling` instead of `scrape_samples_scraped`

For more details refer,

https://github.com/openshift/cluster-monitoring-operator/pull/1313#issuecomment-902490242

Comment 11 errata-xmlrpc 2021-10-18 17:47:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759