Bug 1995699

Summary: Get insights on series churn during upgrades
Product: OpenShift Container Platform Reporter: Damien Grisonnet <dgrisonn>
Component: MonitoringAssignee: Arunprasad Rajkumar <arajkuma>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.8CC: alegrand, amuller, anpicker, aos-bugs, arajkuma, erooth, juzhao, kakkoyun
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1995695 Environment:
Last Closed: 2021-09-14 06:57:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1995695    
Bug Blocks:    

Description Damien Grisonnet 2021-08-19 15:53:15 UTC 
+++ This bug was initially created as a clone of Bug #1995695 +++

Description of problem:

We want to gather information about series churn during upgrades since it will be helpful to know the additional memory usage that Prometheus will have to handle after the update. This may also be helpful to detect high-cardinality metrics.

It would be great to gather this information via Telemetry to have this insight on customer clusters.

To do that, we could use the `scrape_series_added` and `scrape_samples_scraped` metrics from Prometheus. More information about this can be found in this blog post: https://www.robustperception.io/finding-churning-targets-in-prometheus-with-scrape_series_added.

In addition, this information will be very useful to future resiliency improvements since based on the data gathered via telemetry, we will be able to set sane limits to Prometheus that would prevent any malicious target to cause trouble.
Comment 1 Arunprasad Rajkumar 2021-08-31 06:01:05 UTC 
*** Bug 1998969 has been marked as a duplicate of this bug. ***
Comment 9 errata-xmlrpc 2021-09-14 06:57:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3429