Bug 1970828

Description of problem:
login any route under openshift-monitoring will get "500 Internal Error"
# oc -n openshift-monitoring get route
NAME                HOST/PORT                                                                                PATH   SERVICES            PORT    TERMINATION          WILDCARD
alertmanager-main   alertmanager-main-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com          alertmanager-main   web     reencrypt/Redirect   None
grafana             grafana-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com                    grafana             https   reencrypt/Redirect   None
prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com             prometheus-k8s      web     reencrypt/Redirect   None
thanos-querier      thanos-querier-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com             thanos-querier      web     reencrypt/Redirect   None

example:
# oc -n openshift-monitoring logs -c grafana-proxy grafana-864b6bff47-ffl6h
2021/06/11 08:03:52 provider.go:120: Defaulting client-id to system:serviceaccount:openshift-monitoring:grafana
2021/06/11 08:03:52 provider.go:125: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2021/06/11 08:03:52 provider.go:314: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2021/06/11 08:03:52 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:3001/"
2021/06/11 08:03:52 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:grafana
2021/06/11 08:03:52 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled
2021/06/11 08:03:52 http.go:107: HTTPS: listening on [::]:3000
I0611 08:03:52.540434       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
2021/06/11 09:35:20 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/11 09:35:20 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/11 09:35:23 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/11 09:35:23 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/11 09:35:24 oauthproxy.go:656: error redeeming code (client:10.131.0.15:36282): got 400 from "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token" {"error":"unauthorized_client","error_description":"The client is not authorized to request a token using this method."}
2021/06/11 09:35:24 oauthproxy.go:445: ErrorPage 500 Internal Error Internal Error

also found unauthorized_client in openshift-authentication
# oc -n openshift-authentication logs oauth-openshift-999bcf96f-8q298 | grep unauthorized_client
E0611 09:00:16.695488       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s
E0611 09:34:42.825853       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:alertmanager-main
E0611 09:35:24.056607       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:grafana
E0611 09:35:49.436008       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s
E0611 09:35:51.401434       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:thanos-querier

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-06-11-024306

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: