Description of problem: # oc -n openshift-monitoring logs node-exporter-2dm9b -c kube-rbac-proxy I0506 23:28:27.287583 20410 main.go:186] Valid token audiences: I0506 23:28:27.287684 20410 main.go:248] Reading certificate files I0506 23:28:27.287882 20410 main.go:281] Starting TCP socket on [10.0.164.159]:9100 I0506 23:28:27.288101 20410 main.go:288] Listening securely on [10.0.164.159]:9100 E0507 02:21:39.915104 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 02:21:39.915130 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 02:36:39.917788 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 02:36:39.917815 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 02:51:39.916397 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 02:51:39.916426 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 03:06:39.915206 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 03:06:39.915262 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 03:42:54.919036 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 03:42:54.919059 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 03:57:54.918380 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 03:57:54.918405 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:12:54.929356 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:12:54.929381 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:14:09.912921 20410 webhook.go:197] Failed to make webhook authorizer request: subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope E0507 04:14:09.912945 20410 proxy.go:96] Authorization error (user=system:serviceaccount:openshift-monitoring:prometheus-k8s, verb=get, resource=, subresource=)%!(EXTRA *errors.StatusError=subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope) E0507 04:27:54.916531 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:27:54.916556 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:42:54.921523 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:42:54.921546 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:57:54.919979 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 04:57:54.920005 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:12:54.917455 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:12:54.917478 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:27:54.921037 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:27:54.921061 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:42:54.914027 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:42:54.914056 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:57:54.923614 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 05:57:54.923641 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 10:12:52.002061 20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0507 10:12:52.002091 20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
4.5.0-0.nightly-2020-05-06-003431 node_exporter v0.18.1
reproduced with 4.5.0-0.nightly-2020-05-19-041951 # oc -n openshift-monitoring logs node-exporter-c2rk7 -c kube-rbac-proxy I0519 23:32:01.233030 32217 main.go:186] Valid token audiences: I0519 23:32:01.233133 32217 main.go:248] Reading certificate files I0519 23:32:01.233367 32217 main.go:281] Starting TCP socket on [10.0.190.192]:9100 I0519 23:32:01.233547 32217 main.go:288] Listening securely on [10.0.190.192]:9100 E0520 04:44:49.266432 32217 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0520 04:44:49.266537 32217 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0520 04:59:49.267742 32217 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope E0520 04:59:49.267772 32217 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope # oc get tokenreviews -A Error from server (MethodNotAllowed): the server does not allow this method on the requested resource # oc get subjectaccessreviews -A Error from server (MethodNotAllowed): the server does not allow this method on the requested resource
Created attachment 1690279 [details] kube-apiserver logs
Created attachment 1690280 [details] sum by(code) (rate(apiserver_request_total{resource="tokenreviews",version="v1"}[5m]))
Created attachment 1690281 [details] up{job="node-exporter"}
Created attachment 1690285 [details] openshift-state-metrics logs
Created attachment 1690286 [details] kube-state-metrics logs
I've investigated more deeply this issue. Note that the same class of failures had already been reported for the other monitoring components using oauth-proxy and/or kube-rbac-proxy: https://bugzilla.redhat.com/show_bug.cgi?id=1832830 https://bugzilla.redhat.com/show_bug.cgi?id=1836836 Prometheus scrapes node-exporter through kube-rbac-proxy. When this happens, kube-rbac-proxy sends a POST request to /apis/authentication.k8s.io/v1/tokenreviews to validate the bearer token sent by Prometheus. Once in a while, the API server replies with "403 Forbidden" because it evaluates that the node-exporter's service account can't create tokenreviews (e.g. it's not the token presented by Prometheus that is rejected). Before and after this event, the server will happily authorize the same token for tokenreviews. As a result, Prometheus considers that node-exporter is down for this scrape event (see the dips in attachment 1690281 [details]). I've enabled the TraceAll log level in the API server (oc patch kubeapiservers/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "TraceAll" }]'). I can see the kube-rbac-proxy request being rejected (search for "RBAC DENY" in attachment 1690279 [details]) but there's no explanation why. I've seen similar failures for prometheus-adapter's kube-rbac-proxy and oauth-proxy's alertmanager but also for prometheus-operator, kube-state-metrics and openshift-metrics when they query the Kubernetes API (see attachment 1690284 [details], attachment 1690285 [details], attachment 1690286 [details]). Looking at the audit logs, only the monitoring components experience those random failures (not sure if it's because they rely heavily on kube-rbac-proxy/auth-proxy or because of something else). Finally the 'sum by(code) (rate(apiserver_request_total{resource="tokenreviews",version="v1"}[5m]))' query returns data only for status code 201, not 403 (see attachment 1690280 [details]).
*** Bug 1832830 has been marked as a duplicate of this bug. ***
*** Bug 1836836 has been marked as a duplicate of this bug. ***
*** Bug 1836087 has been marked as a duplicate of this bug. ***
Note that this also happens for the prometheus-adapter pod (https://bugzilla.redhat.com/show_bug.cgi?id=1836087)
Hitting this in 4.5, with prometheus-adapter: Warning FailedGetResourceMetric 110m horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/namespace-name/pods?labelSelector=app%3Dapp-cem%2example%3D\": subjectaccessreviews.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-adapter\" cannot create resource \"subjectaccessreviews\" in API group \"authorization.k8s.io\" at the cluster scope") has prevented the request from succeeding (get pods.metrics.k8s.io) Since it at first appears to be a permissions error we tried to add a view clusterrole to the mentioned service account: $ oc adm policy add-cluster-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-adapter It didn't work - this is preventing most (but not all) HPA objects from scraping metrics. Is there a workaround to the issue described in c#12?
closing out as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1863011, the symptoms here are the same, namely RBAC issues despite having the correct assets deployed by cluster-monitoring-operator. *** This bug has been marked as a duplicate of bug 1863011 ***