+++ This bug was initially created as a clone of Bug #2014615 +++ +++ This bug was initially created as a clone of Bug #2014614 +++ Description of problem: Requests from Prometheus to kube-apiserver's /metrics endpoint aren't exempted from Priority and Fairness. When their assigned priority level (currently workload-high) becomes saturated, rejected metrics requests may result in missing samples. Version-Release number of selected component (if applicable): 4.10 How reproducible: Always Steps to Reproduce: 1. Note the UID of the "exempt" prioritylevel: $ oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{'\n'}" 2. Query the metrics endpoint while impersonating the Prometheus ServiceAccount: $ oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid Actual results: The /metrics request isn't assigned to the exempt priority level. $ oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{ '\n' }" afd92c7f-65df-48e3-ac6b-48d0a063cc33 $ oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid I1015 11:41:54.224598 163641 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: 00e7fd91-9201-4ee6-8cfd-99fb0d6c0cc3 Expected results: The /metrics request is assigned to the exempt priority level.
oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-10-29-221623 True False 12m Cluster version is 4.8.0-0.nightly-2021-10-29-221623 oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{ '\n' }" d57f6d8b-996a-423d-8bb8-4e3a6b8a77df oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid I1030 21:22:21.492301 50747 round_trippers.go:454] X-Kubernetes-Pf-Prioritylevel-Uid: d57f6d8b-996a-423d-8bb8-4e3a6b8a77df The /metrics request is assigned to the exempt priority level.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.8.19 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4109