Bug 1684169 - "x509: certificate signed by unknown authority" in oauthproxy blocks user from logging monitoring routes
Summary: "x509: certificate signed by unknown authority" in oauthproxy blocks user fro...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Matt Rogers
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-28 15:42 UTC by Junqi Zhao
Modified: 2019-06-04 10:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:44:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
prometheus routes are accessible (147.42 KB, application/gzip)
2019-03-04 06:36 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:44:57 UTC

Description Junqi Zhao 2019-02-28 15:42:28 UTC
Description of problem:
500 Internal Error for grafana/prometheus/alertmanager route is fixed with 4.0.0-0.nightly-2019-02-27-213933, but new error "x509: certificate signed by unknown authority" in oauthproxy blocks user from logging monitoring routes, this is first seen in 4.0.0-0.nightly-2019-02-28-054829


# oc -n openshift-monitoring logs alertmanager-main-0 -c alertmanager-proxy
2019/02/28 11:38:48 provider.go:102: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main
2019/02/28 11:38:48 provider.go:107: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2019/02/28 11:38:48 provider.go:288: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2019/02/28 11:38:48 oauthproxy.go:201: mapping path "/" => upstream "http://localhost:9093/"
2019/02/28 11:38:48 oauthproxy.go:222: compiled skip-auth-regex => "^/metrics"
2019/02/28 11:38:48 oauthproxy.go:228: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main
2019/02/28 11:38:48 oauthproxy.go:238: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled
2019/02/28 11:38:48 http.go:96: HTTPS: listening on [::]:9094
2019/02/28 11:54:55 provider.go:370: authorizer reason: 
2019/02/28 11:54:58 provider.go:370: authorizer reason: 
2019/02/28 11:55:05 provider.go:530: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2019/02/28 11:55:05 provider.go:570: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2019/02/28 11:55:12 provider.go:530: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2019/02/28 11:55:12 provider.go:570: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2019/02/28 11:55:12 oauthproxy.go:646: error redeeming code (client:10.131.0.7:58830): Post https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/token: x509: certificate signed by unknown authority
2019/02/28 11:55:12 oauthproxy.go:439: ErrorPage 500 Internal Error Internal Error
2019/02/28 11:55:13 provider.go:370: authorizer reason: 

Version-Release number of selected component (if applicable):
4.0.0-0.nightly-2019-02-28-054829

How reproducible:
Always

Steps to Reproduce:
1. Login all cluster monitoring routes(grafana/prometheus/alertmanager routes)
2.
3.

Actual results:
500 error for all cluster monitoring routes

Expected results:
Routes could be accessed.

Additional info:

Comment 4 Standa Laznicka 2019-03-01 08:25:16 UTC
Upstream patch was merged, moving to modified

Comment 5 Standa Laznicka 2019-03-01 08:52:27 UTC
Potentially found a bug, moving back to assigned until fixed or disproved

Comment 6 Standa Laznicka 2019-03-01 11:42:19 UTC
Fixed upstream.

Comment 7 Junqi Zhao 2019-03-04 06:35:42 UTC
all routes could be accessed with 4.0.0-0.nightly-2019-03-04-033148, see attached file

Comment 8 Junqi Zhao 2019-03-04 06:36:14 UTC
Created attachment 1540522 [details]
prometheus routes are accessible

Comment 11 errata-xmlrpc 2019-06-04 10:44:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.