Bug 1671290 - Can not login prometheus/alertmanager/grafana routes with Google IDP Authorization
Summary: Can not login prometheus/alertmanager/grafana routes with Google IDP Authoriz...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Erica von Buelow
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-31 10:19 UTC by Junqi Zhao
Modified: 2019-06-04 10:43 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:42:30 UTC
Target Upstream Version:


Attachments (Terms of Use)
"Application is not available" in page (94.46 KB, image/png)
2019-01-31 10:19 UTC, Junqi Zhao
no flags Details
"Application is not available" in page (80.79 KB, image/png)
2019-01-31 10:23 UTC, Junqi Zhao
no flags Details
"Application is not available" in page - 503 error (286.24 KB, image/png)
2019-02-01 06:52 UTC, Junqi Zhao
no flags Details
4.0.0-0.nightly-2019-01-31-184459 build, it is 500 error now (268.30 KB, image/png)
2019-02-01 07:58 UTC, Junqi Zhao
no flags Details
error in prometheus-proxy container log (6.91 KB, text/plain)
2019-02-01 08:02 UTC, Junqi Zhao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:43:52 UTC

Description Junqi Zhao 2019-01-31 10:19:01 UTC
Created attachment 1525303 [details]
"Application is not available" in page

Description of problem:
Authorization method is Google IDP, then login prometheus/alertmanager/grafana will meet "Application is not available" error.

actually the application is available, since test with kubeadmin temp user, all the routes could be accessed

Take prometheus-k8s route as example
https://prometheus-k8s-openshift-monitoring.apps.qe-chuan.qe.devcluster.openshift.com/

click "Log in with OpenShift" in the page, it navigates to
https://o-openshift-authentication.apps.qe-chuan.qe.devcluster.openshift.com/oauth/authorize?approval_prompt=force&client_id=system%3Aserviceaccount%3Aopenshift-monitoring%3Aprometheus-k8s&redirect_uri=https%3A%2F%2Fprometheus-k8s-openshift-monitoring.apps.qe-chuan.qe.devcluster.openshift.com%2Foauth%2Fcallback&response_type=code&scope=user%3Ainfo+user%3Acheck-access&state=ee2765b51fe28107845bb5b815a7d9f0%3A%2F

decode the url, it is the same as
https://o-openshift-authentication.apps.qe-chuan.qe.devcluster.openshift.com/oauth/authorize?approval_prompt=force&client_id=system:serviceaccount:openshift-monitoring:prometheus-k8s&redirect_uri=https://prometheus-k8s-openshift-monitoring.apps.qe-chuan.qe.devcluster.openshift.com/oauth/callback&response_type=code&scope=user:info user:check-access&state=ee2765b51fe28107845bb5b815a7d9f0:/

then the page shows error "Application is not available"
Dubug with webconsole tool, error shows:
The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol.

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-01-30-174704   True        False         2h      Cluster version is 4.0.0-0.nightly-2019-01-30-174704

$ oc version
oc v4.0.0-0.150.0
kubernetes v1.12.4+f39ab668d3
features: Basic-Auth GSSAPI Kerberos SPNEGO


How reproducible:
Always

Steps to Reproduce:
1. See the description part
2.
3.

Actual results:
Can not login prometheus/alertmanager/grafana routes 

Expected results:
Can login prometheus/alertmanager/grafana routes 

Additional info:

Comment 1 Junqi Zhao 2019-01-31 10:20:41 UTC
Blocks monitoring UI test with Google IDP

Comment 2 Junqi Zhao 2019-01-31 10:23:55 UTC
Created attachment 1525304 [details]
"Application is not available" in page

Comment 3 Neelesh Agrawal 2019-01-31 17:01:58 UTC
Can you send kubeconfig and a way to access the cluster? This will help us diagnose the issue.

Comment 4 Neelesh Agrawal 2019-01-31 21:10:31 UTC
Junqi, looks like accessing QE cluster may be difficult for developers. Please attach kubeconfig and any relevant logs to this bug.

Comment 5 Junqi Zhao 2019-02-01 06:52:40 UTC
Created attachment 1525723 [details]
"Application is not available" in page - 503 error

Comment 7 Junqi Zhao 2019-02-01 07:58:27 UTC
Created attachment 1525762 [details]
4.0.0-0.nightly-2019-01-31-184459 build, it is 500 error now

Comment 8 Junqi Zhao 2019-02-01 08:02:59 UTC
Created attachment 1525763 [details]
error in prometheus-proxy container log

2019/02/01 06:59:12 oauthproxy.go:635: error redeeming code (client:10.131.0.5:57256): unable to retrieve email address for user from token: got 404 {
  "paths": [
    "/apis",
    "/healthz",
    "/healthz/log",
    "/healthz/ping",
    "/healthz/poststarthook/oauth.openshift.io-startoauthclientsbootstrapping",
    "/metrics"
  ]
}

2019/02/01 06:20:18 oauthproxy.go:635: error redeeming code (client:10.131.0.5:32912): Post https://o.apps.juzhao.qe.devcluster.openshift.com/oauth/token: x509: certificate signed by unknown authority
2019/02/01 06:20:18 oauthproxy.go:434: ErrorPage 500 Internal Error Internal Error

Comment 14 Junqi Zhao 2019-02-12 10:44:03 UTC
prometheus/alertmanager/grafana routes could be accessed with
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-12-005016   True        False         1h      Cluster version is 4.0.0-0.nightly-2019-02-12-005016


Images:
  NAME                                          PULL SPEC
  cluster-monitoring-operator                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51c040a9e285083d1f31083ac48479edce1552e53ff2bb8b3f91c62718b99fb2
  grafana                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:788cb9461d4b38c4a8ef5cdac7cbdf46befea56c20f310ec4bf0c127428b908e
  k8s-prometheus-adapter                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2986bc4db9ea270dc77aa1d896e3f76c98dc7bf90f1b4217e8c577a5aa6c1447
  kube-rbac-proxy                               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee82c2882ccddc2605dc92f35b3cc2b2fb5ef76e4b5a5abd9c246b9a0f988d9b
  kube-state-metrics                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed6b9bea8c80d114b4adbaf27c2dd08f1d04957d0c6aa5afb22830616a7d2642
  oauth-proxy                                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:59aa49bd4992ea5a792eb178a4de32743040ff12299e507f669a241b0a0f6ae4
  prom-label-proxy                              quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cc3449b501b7a769cb5e60759a59bc57bcc46f03ad75875ac3efa59bb98782e
  prometheus                                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab4c12ab38d078ed4c9f98a72e7ec4e20c3230d2606dbcd31f5bdf41626b7c35
  prometheus-alertmanager                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d545f54ba476d1efe9c3209f0aec5b39c83e4098c2c8a44694301ce0d029d24
  prometheus-config-reloader                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58f065cb228dce82a21224df06062c99332972a2129ad6b40f778f840d469f84
  prometheus-node-exporter                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fb8fbb8e955e8b3e2174e32063e40fc6762afd02c5c63b117ffcbd8cca10812
  prometheus-operator                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3a6a9b30150beae1235ea52e45dc302883bda35213b204aeab98165216ad8f9
  telemeter                                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c736368f8135cafa3f806c8953edf6d4e33dac13a3de5ba413b37067b41af54

Comment 17 errata-xmlrpc 2019-06-04 10:42:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.