Bug 1913543 - backport: cadvisor machine metrics are missing in k8s 1.19
Summary: backport: cadvisor machine metrics are missing in k8s 1.19
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.6.z
Assignee: Elana Hashman
QA Contact: Weinan Liu
Depends On: 1913096
TreeView+ depends on / blocked
Reported: 2021-01-07 03:49 UTC by OpenShift BugZilla Robot
Modified: 2021-03-09 20:16 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-03-09 20:16:08 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 511 0 None open [release-4.6] Bug 1913543: UPSTREAM: 97006: kubelet: Fix cadvisor machine metrics 2021-02-09 14:20:00 UTC
Red Hat Product Errata RHBA-2021:0674 0 None None None 2021-03-09 20:16:26 UTC

Description OpenShift BugZilla Robot 2021-01-07 03:49:10 UTC
+++ This bug was initially created as a clone of Bug #1913096 +++

See upstream bug: https://github.com/kubernetes/kubernetes/issues/95204

Description of problem:

Machine metrics from cadvisor are missing in Kubernetes 1.19+ (OpenShift 4.6+).

I believe OpenShift does not use the machine_* metrics to calculate machine resource stats, instead relying on the stable metrics provided by kube-state-metrics: https://github.com/kubernetes/kubernetes/issues/95204#issuecomment-719445180

However, it is possible our customers are using these metrics directly.

Version-Release number of selected component (if applicable): 4.6+

How reproducible:

Expected output from a metrics query:

# kubectl get --raw "/api/v1/nodes/NODE_NAME/proxy/metrics/cadvisor" | grep -i machine_cpu_cores
# HELP machine_cpu_cores Number of CPU cores on the machine.
# TYPE machine_cpu_cores gauge
machine_cpu_cores 4

Actual output: 

No matching metrics, as these metrics are not produced.

Additional info:

Patch here: https://github.com/kubernetes/kubernetes/pull/97006

Backport for 1.19: https://github.com/kubernetes/kubernetes/pull/97692
Backport for 1.20: https://github.com/kubernetes/kubernetes/pull/97691

Comment 3 Elana Hashman 2021-02-25 16:26:48 UTC
Bumped severity - cadvisor machine metrics will be totally missing on a 4.5 -> 4.6 upgrade without this.

Comment 11 errata-xmlrpc 2021-03-09 20:16:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.20 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.