Description of problem: normal user view project metrics on Home -> Project -> Dashboard, it reports No datapoints found and all GET requests return 403 Forbidden. The issue is not reproduced by cluster-admin user Version-Release number of selected component (if applicable): 4.4.0-0.nightly-2020-01-12-221811 How reproducible: Always Steps to Reproduce: 1. normal user create a project and add application, make sure some pods are running $ oc get pods -n ui1-project1 | grep Running perl-1-bpks9 1/1 Running 0 95m php-659cf5c84b-qgqbk 1/1 Running 0 36m ruby-8486cb7467-5thrp 1/1 Running 0 36m 2. Check project status at Home -> Projects -> Dashboard Actual results: 2. metrics in Utilization all report No datapoints found, GET request returns 403 Forbidden Request URL: https://<console_route>/api/prometheus/api/v1/query_range?start=1578903086.444&end=1578906686.444&step=60&query=sum%28pod%3Acontainer_fs_usage_bytes%3Asum%7Bcontainer%3D%22%22%2Cpod%21%3D%22%22%2Cnamespace%3D%27ui1-project2%27%7D%29+BY+%28namespace%29 Expected results: 2. normal user should have permission to view metrics Additional info:
Created attachment 1651795 [details] 403 Forbidden
Created attachment 1657918 [details] 403 errors when logged in as test user
Created attachment 1657919 [details] 404 errors when logged in as kube:admin
I was able to reproduce this using a 4.3 cluster which is needed at this time as workaround to: https://bugzilla.redhat.com/show_bug.cgi?id=1794885 Prometheus and Alertmanager services returning 403 errors, breaking console metrics Logged in as kube:admin, Projects -> Project Details, Utilization dashboard card shows graphs/data Logged in as test:test, Projects -> Project Details, Utilization dashboard card shows 'Not available' & 'No datapoints found.' Logged in as kube:admin, I see only 2 404 errors (see attached) Logged in as test:test, I see several 403 errors (see attached) - Not sure if errors due to running 4.4 code on top of 4.3 cluster, or part of the root cause Debugging the error I see: "Error: Prometheus URL is not available at http://0.0.0.0:9000/static/main-0a3c6a98c951...." Agree that normal user should be able to access '/api/prometheus/api/v1/query_range', as Prometheus docs states: "It is presumed that untrusted users have access to the Prometheus HTTP endpoint and logs. They have access to all time series information contained in the database, plus a variety of operational/debugging information. It is also presumed that only trusted users have the ability to change the command line, configuration file, rule files and other aspects of the runtime environment of Prometheus and other components."
Issue seems to be here: https://github.com/openshift/console/blob/master/frontend/public/actions/dashboards.ts#L100 When logged in as test:test, window.SERVER_FLAGS.prometheusTenancyBaseURL and window.SERVER_FLAGS.prometheusBaseURL are empty strings When logged in as kube:admin, these window.SERVER_FLAGS are set Notice they are being set in server/server.go.
I don't believe that the project dashboard is passing the namespace with the query, so we're not hitting the prometheus tenancy endpoint.
Note that metrics are entirely broken by bug 1794885, but there is an additional problem specific to the project dashboard for normal users.
the namespace passing got lost in https://github.com/openshift/console/pull/3790
Now normal user can view project metrics successfully, charts in Utilization are shown correctly. Verified on 4.4.0-0.nightly-2020-02-06-230833
Moving to VERIFIED and opened a new bug to track this different issue
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581