Bug 1972898

Tried to log into prometheus UI on build02.  Get a 500 internal server error.  Logs for the prometheus-proxy have:

2021/06/16 20:03:12 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/16 20:03:12 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.build02.gcp.ci.openshift.org",
  "authorization_endpoint": "https://oauth-openshift.apps.build02.gcp.ci.openshift.org/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.build02.gcp.ci.openshift.org/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/16 20:03:12 oauthproxy.go:656: error redeeming code (client:10.129.40.5:43122): got 400 from "https://oauth-openshift.apps.build02.gcp.ci.openshift.org/oauth/token" {"error":"unauthorized_client","error_description":"The client is not authorized to request a token using this method."}
2021/06/16 20:03:12 oauthproxy.go:445: ErrorPage 500 Internal Error Internal Error


Looks like the client for on cluster oauth is broken.  Not being able to log into things on cluster with delegated auth is urgent blocker+ to me.