Bug 1684169

Summary: "x509: certificate signed by unknown authority" in oauthproxy blocks user from logging monitoring routes
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: apiserver-authAssignee: Matt Rogers <mrogers>
Status: CLOSED ERRATA QA Contact: Chuan Yu <chuyu>
Severity: high Docs Contact:
Priority: high    
Version: 4.1.0CC: anusaxen, aos-bugs, mkhan, nagrawal, slaznick, sponnaga
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:44:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
prometheus routes are accessible none

Description Junqi Zhao 2019-02-28 15:42:28 UTC 
Description of problem:
500 Internal Error for grafana/prometheus/alertmanager route is fixed with 4.0.0-0.nightly-2019-02-27-213933, but new error "x509: certificate signed by unknown authority" in oauthproxy blocks user from logging monitoring routes, this is first seen in 4.0.0-0.nightly-2019-02-28-054829


# oc -n openshift-monitoring logs alertmanager-main-0 -c alertmanager-proxy
2019/02/28 11:38:48 provider.go:102: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main
2019/02/28 11:38:48 provider.go:107: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2019/02/28 11:38:48 provider.go:288: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2019/02/28 11:38:48 oauthproxy.go:201: mapping path "/" => upstream "http://localhost:9093/"
2019/02/28 11:38:48 oauthproxy.go:222: compiled skip-auth-regex => "^/metrics"
2019/02/28 11:38:48 oauthproxy.go:228: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main
2019/02/28 11:38:48 oauthproxy.go:238: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled
2019/02/28 11:38:48 http.go:96: HTTPS: listening on [::]:9094
2019/02/28 11:54:55 provider.go:370: authorizer reason: 
2019/02/28 11:54:58 provider.go:370: authorizer reason: 
2019/02/28 11:55:05 provider.go:530: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2019/02/28 11:55:05 provider.go:570: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2019/02/28 11:55:12 provider.go:530: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2019/02/28 11:55:12 provider.go:570: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2019/02/28 11:55:12 oauthproxy.go:646: error redeeming code (client:10.131.0.7:58830): Post https://openshift-authentication-openshift-authentication.apps.**.qe.devcluster.openshift.com/oauth/token: x509: certificate signed by unknown authority
2019/02/28 11:55:12 oauthproxy.go:439: ErrorPage 500 Internal Error Internal Error
2019/02/28 11:55:13 provider.go:370: authorizer reason: 

Version-Release number of selected component (if applicable):
4.0.0-0.nightly-2019-02-28-054829

How reproducible:
Always

Steps to Reproduce:
1. Login all cluster monitoring routes(grafana/prometheus/alertmanager routes)
2.
3.

Actual results:
500 error for all cluster monitoring routes

Expected results:
Routes could be accessed.

Additional info:
Comment 4 Standa Laznicka 2019-03-01 08:25:16 UTC 
Upstream patch was merged, moving to modified
Comment 5 Standa Laznicka 2019-03-01 08:52:27 UTC 
Potentially found a bug, moving back to assigned until fixed or disproved
Comment 6 Standa Laznicka 2019-03-01 11:42:19 UTC 
Fixed upstream.
Comment 7 Junqi Zhao 2019-03-04 06:35:42 UTC 
all routes could be accessed with 4.0.0-0.nightly-2019-03-04-033148, see attached file
Comment 8 Junqi Zhao 2019-03-04 06:36:14 UTC 
Created attachment 1540522 [details]
prometheus routes are accessible
Comment 11 errata-xmlrpc 2019-06-04 10:44:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758