Description of problem: We want to gather information about series churn during upgrades since it will be helpful to know the additional memory usage that Prometheus will have to handle after the update. This may also be helpful to detect high-cardinality metrics. It would be great to gather this information via Telemetry to have this insight on customer clusters. To do that, we could use the `scrape_series_added` and `scrape_samples_scraped` metrics from Prometheus. More information about this can be found in this blog post: https://www.robustperception.io/finding-churning-targets-in-prometheus-with-scrape_series_added. In addition, this information will be very useful to future resiliency improvements since based on the data gathered via telemetry, we will be able to set sane limits to Prometheus that would prevent any malicious target to cause trouble.
@spasquie suggested to go with `scrape_samples_post_metric_relabeling` instead of `scrape_samples_scraped` For more details refer, https://github.com/openshift/cluster-monitoring-operator/pull/1313#issuecomment-902490242
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759