Bug 1671290

Summary: Can not login prometheus/alertmanager/grafana routes with Google IDP Authorization
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: apiserver-authAssignee: Erica von Buelow <evb>
Status: CLOSED ERRATA QA Contact: Chuan Yu <chuyu>
Severity: high Docs Contact:
Priority: high    
Version: 4.1.0CC: aos-bugs, juzhao, mkhan, nagrawal
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:42:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
"Application is not available" in page
none
"Application is not available" in page
none
"Application is not available" in page - 503 error
none
4.0.0-0.nightly-2019-01-31-184459 build, it is 500 error now
none
error in prometheus-proxy container log none

Description Junqi Zhao 2019-01-31 10:19:01 UTC 
Created attachment 1525303 [details]
"Application is not available" in page

Description of problem:
Authorization method is Google IDP, then login prometheus/alertmanager/grafana will meet "Application is not available" error.

actually the application is available, since test with kubeadmin temp user, all the routes could be accessed

Take prometheus-k8s route as example
https://prometheus-k8s-openshift-monitoring.apps.qe-chuan.qe.devcluster.openshift.com/

click "Log in with OpenShift" in the page, it navigates to
https://o-openshift-authentication.apps.qe-chuan.qe.devcluster.openshift.com/oauth/authorize?approval_prompt=force&client_id=system%3Aserviceaccount%3Aopenshift-monitoring%3Aprometheus-k8s&redirect_uri=https%3A%2F%2Fprometheus-k8s-openshift-monitoring.apps.qe-chuan.qe.devcluster.openshift.com%2Foauth%2Fcallback&response_type=code&scope=user%3Ainfo+user%3Acheck-access&state=ee2765b51fe28107845bb5b815a7d9f0%3A%2F

decode the url, it is the same as
https://o-openshift-authentication.apps.qe-chuan.qe.devcluster.openshift.com/oauth/authorize?approval_prompt=force&client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s&redirect_uri=https://prometheus-k8s-openshift-monitoring.apps.qe-chuan.qe.devcluster.openshift.com/oauth/callback&response_type=code&scope=user:info user:check-access&state=ee2765b51fe28107845bb5b815a7d9f0:/

then the page shows error "Application is not available"
Dubug with webconsole tool, error shows:
The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol.

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-01-30-174704   True        False         2h      Cluster version is 4.0.0-0.nightly-2019-01-30-174704

$ oc version
oc v4.0.0-0.150.0
kubernetes v1.12.4+f39ab668d3
features: Basic-Auth GSSAPI Kerberos SPNEGO


How reproducible:
Always

Steps to Reproduce:
1. See the description part
2.
3.

Actual results:
Can not login prometheus/alertmanager/grafana routes 

Expected results:
Can login prometheus/alertmanager/grafana routes 

Additional info:
Comment 1 Junqi Zhao 2019-01-31 10:20:41 UTC 
Blocks monitoring UI test with Google IDP
Comment 2 Junqi Zhao 2019-01-31 10:23:55 UTC 
Created attachment 1525304 [details]
"Application is not available" in page
Comment 3 Neelesh Agrawal 2019-01-31 17:01:58 UTC 
Can you send kubeconfig and a way to access the cluster? This will help us diagnose the issue.
Comment 4 Neelesh Agrawal 2019-01-31 21:10:31 UTC 
Junqi, looks like accessing QE cluster may be difficult for developers. Please attach kubeconfig and any relevant logs to this bug.
Comment 5 Junqi Zhao 2019-02-01 06:52:40 UTC 
Created attachment 1525723 [details]
"Application is not available" in page - 503 error
Comment 7 Junqi Zhao 2019-02-01 07:58:27 UTC 
Created attachment 1525762 [details]
4.0.0-0.nightly-2019-01-31-184459 build, it is 500 error now
Comment 8 Junqi Zhao 2019-02-01 08:02:59 UTC 
Created attachment 1525763 [details]
error in prometheus-proxy container log

2019/02/01 06:59:12 oauthproxy.go:635: error redeeming code (client:10.131.0.5:57256): unable to retrieve email address for user from token: got 404 {
  "paths": [
    "/apis",
    "/healthz",
    "/healthz/log",
    "/healthz/ping",
    "/healthz/poststarthook/oauth.openshift.io-startoauthclientsbootstrapping",
    "/metrics"
  ]
}

2019/02/01 06:20:18 oauthproxy.go:635: error redeeming code (client:10.131.0.5:32912): Post https://o.apps.juzhao.qe.devcluster.openshift.com/oauth/token: x509: certificate signed by unknown authority
2019/02/01 06:20:18 oauthproxy.go:434: ErrorPage 500 Internal Error Internal Error
Comment 14 Junqi Zhao 2019-02-12 10:44:03 UTC 
prometheus/alertmanager/grafana routes could be accessed with
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-12-005016   True        False         1h      Cluster version is 4.0.0-0.nightly-2019-02-12-005016


Images:
  NAME                                          PULL SPEC
  cluster-monitoring-operator                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51c040a9e285083d1f31083ac48479edce1552e53ff2bb8b3f91c62718b99fb2
  grafana                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:788cb9461d4b38c4a8ef5cdac7cbdf46befea56c20f310ec4bf0c127428b908e
  k8s-prometheus-adapter                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2986bc4db9ea270dc77aa1d896e3f76c98dc7bf90f1b4217e8c577a5aa6c1447
  kube-rbac-proxy                               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee82c2882ccddc2605dc92f35b3cc2b2fb5ef76e4b5a5abd9c246b9a0f988d9b
  kube-state-metrics                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed6b9bea8c80d114b4adbaf27c2dd08f1d04957d0c6aa5afb22830616a7d2642
  oauth-proxy                                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:59aa49bd4992ea5a792eb178a4de32743040ff12299e507f669a241b0a0f6ae4
  prom-label-proxy                              quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cc3449b501b7a769cb5e60759a59bc57bcc46f03ad75875ac3efa59bb98782e
  prometheus                                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab4c12ab38d078ed4c9f98a72e7ec4e20c3230d2606dbcd31f5bdf41626b7c35
  prometheus-alertmanager                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d545f54ba476d1efe9c3209f0aec5b39c83e4098c2c8a44694301ce0d029d24
  prometheus-config-reloader                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58f065cb228dce82a21224df06062c99332972a2129ad6b40f778f840d469f84
  prometheus-node-exporter                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fb8fbb8e955e8b3e2174e32063e40fc6762afd02c5c63b117ffcbd8cca10812
  prometheus-operator                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3a6a9b30150beae1235ea52e45dc302883bda35213b204aeab98165216ad8f9
  telemeter                                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c736368f8135cafa3f806c8953edf6d4e33dac13a3de5ba413b37067b41af54
Comment 17 errata-xmlrpc 2019-06-04 10:42:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758