Bug 1674346 - 403 error for grafana/prometheus route
Summary: 403 error for grafana/prometheus route
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Frederic Branczyk
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-11 04:32 UTC by Junqi Zhao
Modified: 2019-06-04 10:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:42:43 UTC
Target Upstream Version:

Attachments (Terms of Use)
403 error for grafana route (270.29 KB, image/png)
2019-02-11 04:32 UTC, Junqi Zhao 		no flags Details
grafana UI could be accessed (138.95 KB, image/png)
2019-02-12 10:40 UTC, Junqi Zhao 		no flags Details
403 error for prometheus route (247.14 KB, image/png)
2019-02-15 05:37 UTC, Junqi Zhao 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:43:51 UTC

Description Junqi Zhao 2019-02-11 04:32:50 UTC 
Created attachment 1528857 [details]
403 error for grafana route

Description of problem:
Cloned from https://jira.coreos.com/browse/MON-550

Version-Release number of selected component (if applicable):
$ oc version
oc v4.0.0-0.168.0
kubernetes v1.12.4+bdfe8e3f3a
features: Basic-Auth GSSAPI Kerberos SPNEGO

Images:
quay.io/openshift/origin-grafana:latest
quay.io/openshift/origin-oauth-proxy:latest

How reproducible:
Always

Steps to Reproduce:
1. login grafana route
2.
3.

Actual results:
403 error for grafana route

Expected results:
Be able to login grafana route

Additional info:
Comment 1 Junqi Zhao 2019-02-11 04:34:20 UTC 
There is not such issue for prometheus and alertmanager routes
Comment 2 Junqi Zhao 2019-02-12 10:40:39 UTC 
grafana UI could be accessed now, see attached picture
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-12-005016   True        False         1h      Cluster version is 4.0.0-0.nightly-2019-02-12-005016


Images:
  NAME                                          PULL SPEC
  cluster-monitoring-operator                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51c040a9e285083d1f31083ac48479edce1552e53ff2bb8b3f91c62718b99fb2
  grafana                                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:788cb9461d4b38c4a8ef5cdac7cbdf46befea56c20f310ec4bf0c127428b908e
  k8s-prometheus-adapter                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2986bc4db9ea270dc77aa1d896e3f76c98dc7bf90f1b4217e8c577a5aa6c1447
  kube-rbac-proxy                               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee82c2882ccddc2605dc92f35b3cc2b2fb5ef76e4b5a5abd9c246b9a0f988d9b
  kube-state-metrics                            quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed6b9bea8c80d114b4adbaf27c2dd08f1d04957d0c6aa5afb22830616a7d2642
  oauth-proxy                                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:59aa49bd4992ea5a792eb178a4de32743040ff12299e507f669a241b0a0f6ae4
  prom-label-proxy                              quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cc3449b501b7a769cb5e60759a59bc57bcc46f03ad75875ac3efa59bb98782e
  prometheus                                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab4c12ab38d078ed4c9f98a72e7ec4e20c3230d2606dbcd31f5bdf41626b7c35
  prometheus-alertmanager                       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d545f54ba476d1efe9c3209f0aec5b39c83e4098c2c8a44694301ce0d029d24
  prometheus-config-reloader                    quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:58f065cb228dce82a21224df06062c99332972a2129ad6b40f778f840d469f84
  prometheus-node-exporter                      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fb8fbb8e955e8b3e2174e32063e40fc6762afd02c5c63b117ffcbd8cca10812
  prometheus-operator                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3a6a9b30150beae1235ea52e45dc302883bda35213b204aeab98165216ad8f9
  telemeter                                     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c736368f8135cafa3f806c8953edf6d4e33dac13a3de5ba413b37067b41af54
Comment 3 Junqi Zhao 2019-02-12 10:40:57 UTC 
Created attachment 1533990 [details]
grafana UI could be accessed
Comment 4 minden 2019-02-12 11:50:11 UTC 
> grafana UI could be accessed now, see attached picture

Thereby this bug is resolved?

I am not aware of any bug-fixes on our side that would relate to this.
Comment 5 Junqi Zhao 2019-02-13 02:55:53 UTC 
yes,grafana UI could be accessed, not sure what is the cause
Comment 6 Junqi Zhao 2019-02-15 05:37:31 UTC 
re-open it,
403 error for grafana/prometheus route, alertmanager route works well

$ oc version
oc v4.0.0-0.171.0
kubernetes v1.12.4+a532756e37
features: Basic-Auth GSSAPI Kerberos SPNEGO

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-14-160606   True        False         3h36m   Cluster version is 4.0.0-0.nightly-2019-02-14-160606
Comment 7 Junqi Zhao 2019-02-15 05:37:58 UTC 
Created attachment 1535042 [details]
403 error for prometheus route
Comment 8 Junqi Zhao 2019-02-15 06:06:37 UTC 
errors maybe related to oauth-proxy
# oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus-proxy
2019/02/15 03:21:14 oauthproxy.go:764: 10.128.2.11:33376 invalid Authorization header Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJ0ZWxlbWV0ZXItY2xpZW50LXRva2VuLWcydGN2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InRlbGVtZXRlci1jbGllbnQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2NGJlYjQ2NC0zMGM1LTExZTktOTBmZS0wYWQ4ODUwMmMwZTAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6b3BlbnNoaWZ0LW1vbml0b3Jpbmc6dGVsZW1ldGVyLWNsaWVudCJ9.cqOleEo5LcMqQwULcJmAIql-pulA4Ntm3_0FZwaufZniUphgQpjg6aBxEBa33AX9f2b8l5tIN4zjh3kBVMxFJ5nHEgRu6T9zQuDToRZdDgcHkUsgJjwsliWvznQfIfq8v1-40ca0azLkNZ3Wc4hn72JMKzh-P0aNYYSpJFEntJZCKdV4DDGHFf2IWpZOcET2vYZpjBoMu4FetqufnTs9dqA2ec2xO6KZzgNmxcsRVVd3PAGj0wwe5GT00YAVPKK_qvkpV-hjPpPrNxpuynLw2JLL0kheB6TjvQwp4-ZzqA-Bd44ZYJNDxGnbOvRSEJxsdvZA9Uhpga89VPO7K2x7qA
Comment 10 Junqi Zhao 2019-02-18 06:45:07 UTC 
grafana and prometheus routes could be accessed with
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-02-17-182259   True        False         3h17m     Cluster version is 4.0.0-0.nightly-2019-02-17-182259
Comment 15 errata-xmlrpc 2019-06-04 10:42:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758
Note You need to log in before you can comment on or make changes to this bug.