Bug 1907998

Summary:	Gather kube_pod_resource_request/limit metrics as exposed in upstream KEP 1916
Product:	OpenShift Container Platform	Reporter:	Clayton Coleman <ccoleman>
Component:	kube-scheduler	Assignee:	Clayton Coleman <ccoleman>
Status:	CLOSED ERRATA	QA Contact:	RamaKasturi <knarra>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.7	CC:	alegrand, anpicker, aos-bugs, erooth, kakkoyun, lcosic, mfojtik, pkrupa, surbania
Target Milestone:	---	Keywords:	UpcomingSprint
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-24 15:44:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Clayton Coleman 2020-12-15 16:25:36 UTC

We want to more accurately reflect to administrators the resource requests and limits in use on the cluster. As part of upstream KEP 1916 we added a O(pods) (constant factor ~1 series/pod) set of metrics that capture non-terminal (pod,resource,limits/request) values as a sum as interpreted by the kube model (init containers are max).  This will in the future allow a large number of capacity planning dashboards to be implemented.

We want to report total requested / limits metrics to telemetry to assess the rough fleet level use of requests and limits which will be a separate recording rule.  That will allow us to evaluate use of vertical autoscaling and HPA autoscaling in the fleet and whether that leads to higher utilization.

While this is an alpha stability metric, the implementation has low resource cost and scraping by default will allow us to improve overall admin experience on cluster in the UI starting in 4.8 with custom dashboards / better graphs in pod UI, as well as improving the requested limits on the main console dashboard.

Comment 7 Clayton Coleman 2021-01-12 15:09:08 UTC

test with 4.7.0-0.nightly-2020-12-20-031835,there are 3 endpoints under openshift-kube-scheduler, but only find "kube_pod_resource_request" and "kube_pod_resource_limit" under https://192.168.2.82:10259/metrics/resources
please change to VERIFIED if it is not a bug

That is intentional and discussed here: https://github.com/kubernetes/enhancements/pull/1916/files#diff-d4bcd9dbd0f7f963d2fdfe8202d93742bae278c771607805080b45a6a14d1cf2R288 (we don't necessarily want to have multiple components reporting the same high cardinality metrics and the view is the same from all three).

Comment 10 errata-xmlrpc 2021-02-24 15:44:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633