Bug 1970828 - "500 Internal Error" for all openshift-monitoring routes
Summary: "500 Internal Error" for all openshift-monitoring routes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oauth-proxy
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.8.0
Assignee: Sergiusz Urbaniak
QA Contact:
URL:
Whiteboard:
: 1972314 1972898 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-11 09:56 UTC by Junqi Zhao
Modified: 2021-07-27 23:12 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:12:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oauth-server pull 80 0 None closed Bug 1970828: fix projected service accounts as oauth-client 2021-06-16 08:09:10 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:12:57 UTC

Description Junqi Zhao 2021-06-11 09:56:48 UTC
Description of problem:
login any route under openshift-monitoring will get "500 Internal Error"
# oc -n openshift-monitoring get route
NAME                HOST/PORT                                                                                PATH   SERVICES            PORT    TERMINATION          WILDCARD
alertmanager-main   alertmanager-main-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com          alertmanager-main   web     reencrypt/Redirect   None
grafana             grafana-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com                    grafana             https   reencrypt/Redirect   None
prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com             prometheus-k8s      web     reencrypt/Redirect   None
thanos-querier      thanos-querier-openshift-monitoring.apps.***.qe.gcp.devcluster.openshift.com             thanos-querier      web     reencrypt/Redirect   None

example:
# oc -n openshift-monitoring logs -c grafana-proxy grafana-864b6bff47-ffl6h
2021/06/11 08:03:52 provider.go:120: Defaulting client-id to system:serviceaccount:openshift-monitoring:grafana
2021/06/11 08:03:52 provider.go:125: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2021/06/11 08:03:52 provider.go:314: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2021/06/11 08:03:52 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:3001/"
2021/06/11 08:03:52 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:grafana
2021/06/11 08:03:52 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled
2021/06/11 08:03:52 http.go:107: HTTPS: listening on [::]:3000
I0611 08:03:52.540434       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
2021/06/11 09:35:20 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/11 09:35:20 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/11 09:35:23 provider.go:587: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2021/06/11 09:35:23 provider.go:627: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2021/06/11 09:35:24 oauthproxy.go:656: error redeeming code (client:10.131.0.15:36282): got 400 from "https://oauth-openshift.apps.***.qe.gcp.devcluster.openshift.com/oauth/token" {"error":"unauthorized_client","error_description":"The client is not authorized to request a token using this method."}
2021/06/11 09:35:24 oauthproxy.go:445: ErrorPage 500 Internal Error Internal Error

also found unauthorized_client in openshift-authentication
# oc -n openshift-authentication logs oauth-openshift-999bcf96f-8q298 | grep unauthorized_client
E0611 09:00:16.695488       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s
E0611 09:34:42.825853       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:alertmanager-main
E0611 09:35:24.056607       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:grafana
E0611 09:35:49.436008       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s
E0611 09:35:51.401434       1 access.go:177] osin: error=unauthorized_client, internal_error=<nil> get_client=client check failed, client_id=system:serviceaccount:openshift-monitoring:thanos-querier

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-06-11-024306

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Standa Laznicka 2021-06-11 10:23:23 UTC
This is a blocker because it will prevent valid users from being able to log in to services that are using the OpenShift authentication to gate access, where the service is using a Service Account as its OAuth2 client according to https://docs.openshift.com/container-platform/4.7/authentication/using-service-accounts-as-oauth-client.html. Among others, this would apply to all the monitoring tools in the openshift-monitoring namespace, supplied by default in every OCP installation.

Comment 6 Simon Pasquier 2021-06-16 08:08:25 UTC
*** Bug 1972314 has been marked as a duplicate of this bug. ***

Comment 7 W. Trevor King 2021-06-16 20:50:13 UTC
*** Bug 1972898 has been marked as a duplicate of this bug. ***

Comment 10 errata-xmlrpc 2021-07-27 23:12:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.