Bug 2014614 - Metrics scraping requests should be assigned to exempt priority level
Summary: Metrics scraping requests should be assigned to exempt priority level
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.10.0
Assignee: Stefan Schimanski
QA Contact: Rahul Gangwar
URL:
Whiteboard:
Depends On:
Blocks: 2014615
TreeView+ depends on / blocked
 
Reported: 2021-10-15 15:48 UTC by Ben Luddy
Modified: 2022-03-12 04:39 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:39:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:39:32 UTC

Description Ben Luddy 2021-10-15 15:48:19 UTC
Description of problem:

Requests from Prometheus to kube-apiserver's /metrics endpoint aren't exempted from Priority and Fairness. When their assigned priority level (currently workload-high) becomes saturated, rejected metrics requests may result in missing samples.


Version-Release number of selected component (if applicable): 4.10


How reproducible: Always


Steps to Reproduce:

1. Note the UID of the "exempt" prioritylevel:

$ oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{'\n'}"

2. Query the metrics endpoint while impersonating the Prometheus ServiceAccount:

$ oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid


Actual results:

The /metrics request isn't assigned to the exempt priority level.

$ oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{ '\n' }"
afd92c7f-65df-48e3-ac6b-48d0a063cc33
$ oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid
I1015 11:41:54.224598  163641 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: 00e7fd91-9201-4ee6-8cfd-99fb0d6c0cc3


Expected results:

The /metrics request is assigned to the exempt priority level.

Comment 2 Rahul Gangwar 2021-10-20 05:13:05 UTC
 oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-10-19-210507   True        False         14m     Cluster version is 4.10.0-0.nightly-2021-10-19-210507

oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{ '\n' }"
c6370b8b-2f97-4a07-9ca6-448551bc8e3e

oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid
I1020 10:15:17.560268   44362 round_trippers.go:454]     X-Kubernetes-Pf-Prioritylevel-Uid: c6370b8b-2f97-4a07-9ca6-448551bc8e3e

The /metrics request is assigned to the exempt priority level.

Comment 5 errata-xmlrpc 2022-03-12 04:39:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.