Bug 1999057 - Drop high cardinality cAdvisor metrics
Summary: Drop high cardinality cAdvisor metrics
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.8
Hardware: All
OS: All
high
high
Target Milestone: ---
: 4.8.z
Assignee: Philip Gough
QA Contact: hongyan li
URL:
Whiteboard:
Depends On: 1997034
Blocks: 1999072
TreeView+ depends on / blocked
 
Reported: 2021-08-30 10:55 UTC by Philip Gough
Modified: 2021-11-10 10:55 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Drops selected set of high cardinality metrics from cAdvisor. Reason: To reduce the overall memory footprint of Promtheus. Result: Lowers the overall count of time series stored by Prometheus.
Clone Of: 1997034
: 1999072 (view as bug list)
Environment:
Last Closed: 2021-11-10 10:55:06 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1360 0 None Merged Bug 1999057: jsonnet: Sync with kube-prometheus 2021-09-27 15:22:34 UTC
Github openshift cluster-monitoring-operator pull 1394 0 None open Revert "Bug 1999057: jsonnet: Sync with kube-prometheus" 2021-09-27 10:40:59 UTC
Github prometheus-operator kube-prometheus pull 1353 0 None None None 2021-08-30 11:39:04 UTC

Description Philip Gough 2021-08-30 10:55:54 UTC 
+++ This bug was initially created as a clone of Bug #1997034 +++

Description of problem:

To reduce the overall memory consumption of Prometheus, we can drop unused cAdvisor metrics with high cardinality.

Version-Release number of selected component (if applicable):

4.8


Additional info:

This can be verified by a before and after query: count({metrics_path="/metrics/cadvisor"})

--- Additional comment from Junqi Zhao on 2021-08-30 08:49:53 UTC ---

tested with 4.9.0-0.nightly-2021-08-29-010334
with the fix, the count of the cadvisor metrics is less than that without the fix
count({metrics_path="/metrics/cadvisor"})
{} 46166

without the fix
count({metrics_path="/metrics/cadvisor"})
{} 56388
Note You need to log in before you can comment on or make changes to this bug.