Created attachment 1655177 [details] 403 errors The console backend proxies requests to the Prometheus and Alertmanager services for metrics. We're starting to see 403 responses (see screenshot), which breaks metrics in console. This appears to have surfaced with the changes in https://openshift-release.svc.ci.openshift.org/releasestream/4.4.0-0.nightly/release/4.4.0-0.nightly-2020-01-24-172700 aws-console tests were passing in 4.4.0-0.nightly-2020-01-24-141203 and earlier.
Created attachment 1655178 [details] Metrics and alerts fail to load in console
Created attachment 1655179 [details] prometheus-proxy container logs There are some suspicious auth errors in the prometheus-proxy container logs
*** Bug 1796912 has been marked as a duplicate of this bug. ***
*** Bug 1797497 has been marked as a duplicate of this bug. ***
Created attachment 1657750 [details] still 403/Forbidden error
Reproduced. The login to the prometheus dashboard actually works fine but apparently in the monitoring/alerts section console is trying to communicate with it using a token and this triggers the bug.
Tested with 4.4.0-0.nightly-2020-02-06-170203 and checked Prometheus API in admin console "Home -> Overview" "Workloads -> Pods" "Monitoring -> Alerting", "Monitoring -> Metrics" "Compute -> Nodes" and developer console Could show metrics data/diagram now
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days