Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1970828

Summary: "500 Internal Error" for all openshift-monitoring routes
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: oauth-proxyAssignee: Sergiusz Urbaniak <surbania>
Status: CLOSED ERRATA QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.8CC: aos-bugs, ccoleman, hongkliu, mfojtik, surbania, wking
Target Milestone: ---Keywords: Regression
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:12:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2021-06-11 09:56:48 UTC 
Description of problem:
login any route under openshift-monitoring will get "500 Internal Error"
# oc -n openshift-monitoring get route
NAME                HOST/PORT                                                                                PATH   SERVICES            PORT    TERMINATION          WILDCARD
alertmanager-main   alertmanager-main-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com          alertmanager-main   web     reencrypt/Redirect   None
grafana             grafana-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com                    grafana             https   reencrypt/Redirect   None
prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com             prometheus-k8s      web     reencrypt/Redirect   None
thanos-querier      thanos-querier-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com             thanos-querier      web     reencrypt/Redirect   None

example:
# oc -n openshift-monitoring logs -c grafana-proxy grafana-864b6bff47-ffl6h
2021/06/11 08:03:52 provider.go:120: Defaulting client-id to system:serviceaccount:openshift-monitoring:grafana
2021/06/11 08:03:52 provider.go:125: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2021/06/11 08:03:52 provider.go:314: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2021/06/11 08:03:52 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:3001/"
2021/06/11 08:03:52 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:grafana
2021/06/11 08:03:52 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled
2021/06/11 08:03:52 http.go:107: HTTPS: listening on [::]:3000
I0611 08:03:52.540434       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
2021/06/11 09:35:20 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/11 09:35:20 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/11 09:35:23 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/11 09:35:23 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/11 09:35:24 oauthproxy.go:656: error redeeming code (client:10.131.0.15:36282): got 400 from "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token" {"error":"unauthorized_client","error_description":"The client is not authorized to request a token using this method."}
2021/06/11 09:35:24 oauthproxy.go:445: ErrorPage 500 Internal Error Internal Error

also found unauthorized_client in openshift-authentication
# oc -n openshift-authentication logs oauth-openshift-999bcf96f-8q298 | grep unauthorized_client
E0611 09:00:16.695488       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s
E0611 09:34:42.825853       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:alertmanager-main
E0611 09:35:24.056607       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:grafana
E0611 09:35:49.436008       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s
E0611 09:35:51.401434       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:thanos-querier

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-06-11-024306

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Standa Laznicka 2021-06-11 10:23:23 UTC 
This is a blocker because it will prevent valid users from being able to log in to services that are using the OpenShift authentication to gate access, where the service is using a Service Account as its OAuth2 client according to https://docs.openshift.com/container-platform/4.7/authentication/using-service-accounts-as-oauth-client.html. Among others, this would apply to all the monitoring tools in the openshift-monitoring namespace, supplied by default in every OCP installation.
Comment 6 Simon Pasquier 2021-06-16 08:08:25 UTC 
*** Bug 1972314 has been marked as a duplicate of this bug. ***
Comment 7 W. Trevor King 2021-06-16 20:50:13 UTC 
*** Bug 1972898 has been marked as a duplicate of this bug. ***
Comment 10 errata-xmlrpc 2021-07-27 23:12:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438