Bug 1832825 - "cannot create resource subjectaccessreviews/tokenreviews at the cluster scope" error info in node-exporter pod's kube-rbac-proxy container
Summary: "cannot create resource subjectaccessreviews/tokenreviews at the cluster scop...
Keywords:
Status: CLOSED DUPLICATE of bug 1863011
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.5
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.7.0
Assignee: Simon Pasquier
QA Contact: Junqi Zhao
URL:
Whiteboard:
: 1832830 1836087 1836836 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-07 10:35 UTC by Junqi Zhao
Modified: 2024-03-25 15:54 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-01 05:50:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
kube-apiserver logs (2.25 MB, application/gzip)
2020-05-20 15:06 UTC, Simon Pasquier
no flags Details
sum by(code) (rate(apiserver_request_total{resource="tokenreviews",version="v1"}[5m])) (75.67 KB, image/png)
2020-05-20 15:12 UTC, Simon Pasquier
no flags Details
openshift-state-metrics logs (1.24 KB, text/plain)
2020-05-20 15:37 UTC, Simon Pasquier
no flags Details
kube-state-metrics logs (1.86 KB, text/plain)
2020-05-20 15:37 UTC, Simon Pasquier
no flags Details

Description Junqi Zhao 2020-05-07 10:35:04 UTC
Description of problem:
# oc -n openshift-monitoring logs node-exporter-2dm9b -c kube-rbac-proxy
I0506 23:28:27.287583   20410 main.go:186] Valid token audiences: 
I0506 23:28:27.287684   20410 main.go:248] Reading certificate files
I0506 23:28:27.287882   20410 main.go:281] Starting TCP socket on [10.0.164.159]:9100
I0506 23:28:27.288101   20410 main.go:288] Listening securely on [10.0.164.159]:9100
E0507 02:21:39.915104   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 02:21:39.915130   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 02:36:39.917788   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 02:36:39.917815   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 02:51:39.916397   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 02:51:39.916426   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 03:06:39.915206   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 03:06:39.915262   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 03:42:54.919036   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 03:42:54.919059   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 03:57:54.918380   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 03:57:54.918405   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:12:54.929356   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:12:54.929381   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:14:09.912921   20410 webhook.go:197] Failed to make webhook authorizer request: subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope
E0507 04:14:09.912945   20410 proxy.go:96] Authorization error (user=system:serviceaccount:openshift-monitoring:prometheus-k8s, verb=get, resource=, subresource=)%!(EXTRA *errors.StatusError=subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope)
E0507 04:27:54.916531   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:27:54.916556   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:42:54.921523   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:42:54.921546   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:57:54.919979   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 04:57:54.920005   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:12:54.917455   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:12:54.917478   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:27:54.921037   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:27:54.921061   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:42:54.914027   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:42:54.914056   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:57:54.923614   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:57:54.923641   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 10:12:52.002061   20410 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 10:12:52.002091   20410 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Junqi Zhao 2020-05-07 10:39:35 UTC
4.5.0-0.nightly-2020-05-06-003431
node_exporter v0.18.1

Comment 5 Junqi Zhao 2020-05-20 08:19:24 UTC
reproduced with 4.5.0-0.nightly-2020-05-19-041951
# oc -n openshift-monitoring logs node-exporter-c2rk7 -c kube-rbac-proxy
I0519 23:32:01.233030   32217 main.go:186] Valid token audiences: 
I0519 23:32:01.233133   32217 main.go:248] Reading certificate files
I0519 23:32:01.233367   32217 main.go:281] Starting TCP socket on [10.0.190.192]:9100
I0519 23:32:01.233547   32217 main.go:288] Listening securely on [10.0.190.192]:9100
E0520 04:44:49.266432   32217 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 04:44:49.266537   32217 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 04:59:49.267742   32217 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 04:59:49.267772   32217 proxy.go:73] Unable to authenticate the request due to an error: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:node-exporter" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope

# oc get tokenreviews -A
Error from server (MethodNotAllowed): the server does not allow this method on the requested resource
# oc get subjectaccessreviews -A
Error from server (MethodNotAllowed): the server does not allow this method on the requested resource

Comment 6 Simon Pasquier 2020-05-20 15:06:36 UTC
Created attachment 1690279 [details]
kube-apiserver logs

Comment 7 Simon Pasquier 2020-05-20 15:12:27 UTC
Created attachment 1690280 [details]
sum by(code) (rate(apiserver_request_total{resource="tokenreviews",version="v1"}[5m]))

Comment 8 Simon Pasquier 2020-05-20 15:13:10 UTC
Created attachment 1690281 [details]
up{job="node-exporter"}

Comment 10 Simon Pasquier 2020-05-20 15:37:17 UTC
Created attachment 1690285 [details]
openshift-state-metrics logs

Comment 11 Simon Pasquier 2020-05-20 15:37:38 UTC
Created attachment 1690286 [details]
kube-state-metrics logs

Comment 12 Simon Pasquier 2020-05-20 15:42:46 UTC
I've investigated more deeply this issue. Note that the same class of failures had already been reported for the other monitoring components using oauth-proxy and/or kube-rbac-proxy:

https://bugzilla.redhat.com/show_bug.cgi?id=1832830
https://bugzilla.redhat.com/show_bug.cgi?id=1836836

Prometheus scrapes node-exporter through kube-rbac-proxy. When this happens, kube-rbac-proxy sends a POST request to /apis/authentication.k8s.io/v1/tokenreviews to validate the bearer token sent by Prometheus.
Once in a while, the API server replies with "403 Forbidden" because it evaluates that the node-exporter's service account can't create tokenreviews (e.g. it's not the token presented by Prometheus that is rejected). Before and after this event, the server will happily authorize the same token for tokenreviews. As a result, Prometheus considers that node-exporter is down for this scrape event (see the dips in attachment 1690281 [details]).

I've enabled the TraceAll log level in the API server (oc patch kubeapiservers/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "TraceAll" }]'). I can see the kube-rbac-proxy request being rejected (search for "RBAC DENY" in attachment 1690279 [details]) but there's no explanation why. I've seen similar failures for prometheus-adapter's kube-rbac-proxy and oauth-proxy's alertmanager but also for prometheus-operator, kube-state-metrics and openshift-metrics when they query the Kubernetes API (see attachment 1690284 [details], attachment 1690285 [details], attachment 1690286 [details]).

Looking at the audit logs, only the monitoring components experience those random failures (not sure if it's because they rely heavily on kube-rbac-proxy/auth-proxy or because of something else).

Finally the 'sum by(code) (rate(apiserver_request_total{resource="tokenreviews",version="v1"}[5m]))' query returns data only for status code 201, not 403 (see attachment 1690280 [details]).

Comment 13 Simon Pasquier 2020-05-20 15:44:37 UTC
*** Bug 1832830 has been marked as a duplicate of this bug. ***

Comment 14 Simon Pasquier 2020-05-20 15:46:02 UTC
*** Bug 1836836 has been marked as a duplicate of this bug. ***

Comment 15 Sergiusz Urbaniak 2020-05-25 08:51:34 UTC
*** Bug 1836087 has been marked as a duplicate of this bug. ***

Comment 16 Sergiusz Urbaniak 2020-05-25 08:52:22 UTC
Note that this also happens for the prometheus-adapter pod (https://bugzilla.redhat.com/show_bug.cgi?id=1836087)

Comment 26 Caden Marchese 2020-09-30 14:32:02 UTC
Hitting this in 4.5, with prometheus-adapter:

  Warning  FailedGetResourceMetric       110m                    horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/namespace-name/pods?labelSelector=app%3Dapp-cem%2example%3D\": subjectaccessreviews.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-monitoring:prometheus-adapter\" cannot create resource \"subjectaccessreviews\" in API group \"authorization.k8s.io\" at the cluster scope") has prevented the request from succeeding (get pods.metrics.k8s.io)

Since it at first appears to be a permissions error we tried to add a view clusterrole to the mentioned service account:

  $ oc adm policy add-cluster-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-adapter

It didn't work - this is preventing most (but not all) HPA objects from scraping metrics. Is there a workaround to the issue described in c#12?

Comment 29 Sergiusz Urbaniak 2020-10-01 05:50:10 UTC
closing out as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1863011, the symptoms here are the same, namely RBAC issues despite having the correct assets deployed by cluster-monitoring-operator.

*** This bug has been marked as a duplicate of bug 1863011 ***


Note You need to log in before you can comment on or make changes to this bug.