Hide Forgot
+++ This bug was initially created as a clone of Bug #2014614 +++ Description of problem: Requests from Prometheus to kube-apiserver's /metrics endpoint aren't exempted from Priority and Fairness. When their assigned priority level (currently workload-high) becomes saturated, rejected metrics requests may result in missing samples. Version-Release number of selected component (if applicable): 4.10 How reproducible: Always Steps to Reproduce: 1. Note the UID of the "exempt" prioritylevel: $ oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{'\n'}" 2. Query the metrics endpoint while impersonating the Prometheus ServiceAccount: $ oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid Actual results: The /metrics request isn't assigned to the exempt priority level. $ oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{ '\n' }" afd92c7f-65df-48e3-ac6b-48d0a063cc33 $ oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid I1015 11:41:54.224598 163641 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: 00e7fd91-9201-4ee6-8cfd-99fb0d6c0cc3 Expected results: The /metrics request is assigned to the exempt priority level.
oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-10-22-102153 True False 16m Cluster version is 4.9.0-0.nightly-2021-10-22-102153 oc get prioritylevelconfiguration exempt -o=jsonpath="{.metadata.uid}{ '\n' }" 4496da66-a970-42a3-87b6-1e919b74a59d oc --as=system:serviceaccount:openshift-monitoring:prometheus-k8s get --raw '/metrics' -v10 2>&1 | grep X-Kubernetes-Pf-Prioritylevel-Uid I1025 12:17:01.708499 10980 round_trippers.go:454] X-Kubernetes-Pf-Prioritylevel-Uid: 4496da66-a970-42a3-87b6-1e919b74a59d The /metrics request is assigned to the exempt priority level.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.9.5 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4005