Bug 1832830

Summary: "cannot create resource subjectaccessreviews/tokenreviews at the cluster scope" error info in alertmanager pod's alertmanager-proxy container logs
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Simon Pasquier <spasquie>
Status: CLOSED DUPLICATE QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: low    
Version: 4.5CC: alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, spasquie, surbania
Target Milestone: ---Keywords: Regression, Reopened
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-20 15:44:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2020-05-07 10:56:11 UTC
Description of problem:
# oc -n openshift-monitoring logs alertmanager-main-1 -c alertmanager-proxy
2020/05/07 02:10:50 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main
2020/05/07 02:10:50 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2020/05/07 02:10:50 provider.go:312: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2020/05/07 02:10:50 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"
2020/05/07 02:10:50 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"
2020/05/07 02:10:50 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main
2020/05/07 02:10:50 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled
2020/05/07 02:10:50 http.go:107: HTTPS: listening on [::]:9095
I0507 02:10:50.174462       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
E0507 05:44:43.464720       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 05:44:43 oauthproxy.go:782: requestauth: 10.129.2.18:54220 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 05:44:43.464792       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 05:44:43 oauthproxy.go:782: requestauth: 10.128.2.21:56888 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 08:00:19.386611       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 08:00:19 oauthproxy.go:782: requestauth: 10.130.0.51:50544 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 09:42:54.529180       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 09:42:54 oauthproxy.go:782: requestauth: 10.129.2.18:54396 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 09:49:33 provider.go:394: authorizer reason: 
2020/05/07 09:52:11 provider.go:394: authorizer reason: 
2020/05/07 09:55:10 provider.go:394: authorizer reason: 
2020/05/07 09:57:25 provider.go:394: authorizer reason: 
2020/05/07 09:57:40 provider.go:394: authorizer reason: 
2020/05/07 09:57:57 provider.go:394: authorizer reason: 
2020/05/07 09:58:12 provider.go:394: authorizer reason: 
2020/05/07 09:58:27 provider.go:394: authorizer reason: 
2020/05/07 09:59:24 provider.go:394: authorizer reason: 
E0507 10:30:19.424134       1 webhook.go:197] Failed to make webhook authorizer request: subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope
2020/05/07 10:30:19 oauthproxy.go:782: requestauth: 10.129.0.52:54082 subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope
E0507 10:37:18.511056       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 10:37:18 oauthproxy.go:782: requestauth: 10.128.2.21:36442 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0507 10:37:18.515126       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/07 10:37:18 oauthproxy.go:782: requestauth: 10.129.2.18:54396 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope


Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-05-06-003431
alertmanager v0.20.0

How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 Junqi Zhao 2020-05-20 08:13:01 UTC
reproduced with 4.5.0-0.nightly-2020-05-18-225907
# oc -n openshift-monitoring logs alertmanager-main-0 -c alertmanager-proxy
2020/05/19 23:40:27 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main
2020/05/19 23:40:27 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2020/05/19 23:40:27 provider.go:312: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2020/05/19 23:40:27 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"
2020/05/19 23:40:27 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"
2020/05/19 23:40:27 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main
2020/05/19 23:40:27 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled
2020/05/19 23:40:27 http.go:107: HTTPS: listening on [::]:9095
I0519 23:40:27.265549       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
2020/05/20 03:31:54 provider.go:394: authorizer reason: 
E0520 06:00:30.171891       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 06:00:30.171913       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 06:00:30 oauthproxy.go:782: requestauth: 10.128.2.10:59668 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 06:00:30 oauthproxy.go:782: requestauth: 10.129.2.5:52020 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 06:45:30.170023       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 06:45:30 oauthproxy.go:782: requestauth: 10.128.2.10:59668 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 06:45:30.170052       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 06:45:30 oauthproxy.go:782: requestauth: 10.129.2.5:52020 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 07:30:30.171702       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 07:30:30 oauthproxy.go:782: requestauth: 10.129.2.5:52020 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 07:30:30.171713       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 07:30:30 oauthproxy.go:782: requestauth: 10.128.2.10:59668 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 08:04:13.466773       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 08:04:13 oauthproxy.go:782: requestauth: 10.128.2.10:59668 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0520 08:04:13.466804       1 webhook.go:109] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
2020/05/20 08:04:13 oauthproxy.go:782: requestauth: 10.129.2.5:52020 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-monitoring:alertmanager-main" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope

# oc get tokenreviews -A
Error from server (MethodNotAllowed): the server does not allow this method on the requested resource
# oc get subjectaccessreviews -A
Error from server (MethodNotAllowed): the server does not allow this method on the requested resource

Comment 5 Simon Pasquier 2020-05-20 15:44:37 UTC
Closing as a duplicate because this is exactly the same error than returned by the Kubernetes API in bug 1832825.

*** This bug has been marked as a duplicate of bug 1832825 ***