Bug 2015806 - Metrics view in Deployment reports "Forbidden" when not cluster-admin
Summary: Metrics view in Deployment reports "Forbidden" when not cluster-admin
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Management Console
Version: 4.8
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 4.10.0
Assignee: Jon Jackson
QA Contact: Xiyun Zhao
Depends On:
Blocks: 2020000
TreeView+ depends on / blocked
Reported: 2021-10-20 07:05 UTC by Simon Reber
Modified: 2023-09-15 01:16 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Resource details page metrics tab metrics requests were hitting the cluster-scoped Thanos endpoint Consequence: Users without authorization for this endpoint would receive a 401 response for all queries Fix: Update to use the Thanos tenancy endpoints and remove redundant namespace query arguments Result: Users with correct RBAC can now see data in resource metrics tabs.
Clone Of:
: 2020000 (view as bug list)
Last Closed: 2022-03-10 16:20:39 UTC
Target Upstream Version:

Attachments (Terms of Use)
Screenshot from the Console, showing the problem (75.70 KB, image/png)
2021-10-20 07:05 UTC, Simon Reber
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift console pull 10344 0 None open Bug 2015806: Fix resource metrics 403 errors for project admin users 2021-10-28 00:27:17 UTC
Red Hat Knowledge Base (Solution) 6440081 0 None None None 2021-10-20 11:17:55 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:20:52 UTC

Description Simon Reber 2021-10-20 07:05:24 UTC
Created attachment 1834956 [details]
Screenshot from the Console, showing the problem

Description of problem:

The newly available Metrics View in the Deployment Details is not able to show the expected graph when working with project admin role.

The graph is perfectly shown, when logged in with a user having cluster-admin. But every user that does not have cluster-admin role is not able to see the metrics and instead is seeing the Forbidden message

Version-Release number of selected component (if applicable):

 - OpenShift Container Platform 4.8.13

How reproducible:

 - Always

Steps to Reproduce:
1. Create a Deployment in the Management Console with a user having project admin role
2. Check Metrics of the Deployment

Actual results:

Metrics are not shown and instead an error is reported that it's Forbidden to View the Metrics.

Expected results:

Metrics should be visible for every default project role (admin,edit and view)

Additional info:

Comment 4 Jakub Hadvig 2021-10-21 09:13:08 UTC
@Simon ˆˆ

Comment 8 Xiyun Zhao 2021-11-02 02:34:04 UTC
This bug has been verified and fixed on payload 4.10.0-0.nightly-2021-11-01-163833

Verification steps:
1. Login in OCP as a normal user
2. Change user to 'Administrator' perspective, create a Deployment with the user that has project admin role
3. Navigate to Deployment -> Deployment details page -> Metrics tab
4. Verify if user is able to check Metrics on deployment details page without any error
5. Update perspective from administrator to developer mode
6. Navigate to Topology ->  Deployment, select the deployment that was created on step2, then goto Observe tab
7. Verify if user is able to check Metrics on Observe tab without any error
8. Re-login OCP as cluster-admin
9. Repeat step3-7

4,7 No error message found on Metrics, user is able to check Metrics normally and correctly

Comment 16 Victor Medina 2022-02-07 09:36:38 UTC

Just to be sure... I have this ticket 03143162 with this very same issue. CU had a version OCP4.8 and upgraded to the most recent 4.9.17 recently but still has the same issue. 

Is there anything else need to be done to fix this issue? CU reports issue is still present after upgrading to lastest v4.9.

Comment 17 Jon Jackson 2022-02-07 16:22:02 UTC
@vmedina They shouldn't need to do anything else as long as the user who is viewing the workload metrics tab has the correct permissions to make requests to the Thanos tenancy endpoint. It would be helpful to see screen captures and/or a har file (https://support.google.com/admanager/answer/10358597?hl=en) to further investigate whether this is the same issue or not.

Comment 18 Victor Medina 2022-02-08 12:01:27 UTC
@jonjacks Thanks, sure I will request screenshots and a har file and share it as soon as I get them from CU via a private post to this BZ.

Comment 19 Victor Medina 2022-02-10 09:04:46 UTC
@jonjacks I am attaching the requested files

Comment 31 Xiyun Zhao 2022-02-23 14:16:57 UTC
Hi Victor & Ashwinl,

After investigation and summarize all the comment upon, please check if below response is helpful. 
The customer error is failed on API that is to query sum and range of charts, which means user has no sufficient permission to get the chart basic information. So I agreed with Jon's suggestion, please help to confirm 
  1) Whether deployment is being created successfully or not.
  2) Could try to let customer create a deployment with the example YAML, and check if the Metric/Observe could be shown normally.

Here are the result for all mentioned issue, both of them have been re-checked on OCP4.9.17
1. The customer mentioned issue as below shown(Attached file:webconsole error), currently it could not be reproduced on OCP4.9.17. After create a deployment with sample YAML, all Metrics could be shown on page correctly both on administrator and developer perspective, and without listed error. (Attached file: Testing_OCP_4.9.17)
   /api/v1/query?query=sum%........ 403 (Forbidden)
   /api/vi/query?query_range? star....timout=30 403(Forbidden)

2. The original error for this bug is for two issues as below shown, and currently it has been fixed, and not related to customer issue.
   1) Metrics could not be shown with an error 'An error occurred Forbidden' (Attached file: Screenshot from the Console, showing the problem) on administrator perspective on deployment page.
   2) An error 'workload_type%3D27deployment%27%7D%29+by+%28pod%29timout=30s 403 (Forbidden)' can be tracked on console of browser which cause loading chart failed(Attached file: Screenshot with view of Browser developer tools)

3. Two existing 403 forbidden issue that mentioned on discussion shown on below. It's a known issue, and could be tracking on ticket https://bugzilla.redhat.com/show_bug.cgi?id=2042683 
   /api/prometheus/api/v1/rules 403 (Forbidden)
   /api/alertmanager/api/v2/silences 403 (Forbidden)

4. A new error /api/prometheus-tenancy is also mentioned on discussion. It could be reproduced, and need more information from developers. 
   But it could not impact user to loading metrics on development page on administrator perspective, and also could not impact metrics loading on topology page on developer perspective. (Attached file: Testing_OCP_4.9.17)
   main-chunk-240037e69e403788db78.min.js:1 GET https://console-openshift-console.apps.ci-ln-vmp9d4t-72292.origin-ci-int-gce.dev.rhcloud.com/api/prometheus-tenancy/api/v1/rules?namespace=test 403 (Forbidden)
   main-chunk-240037e69e403788db78.min.js:1 Error polling URL: r: Forbidden

Comment 36 errata-xmlrpc 2022-03-10 16:20:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Comment 37 Red Hat Bugzilla 2023-09-15 01:16:26 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days

Note You need to log in before you can comment on or make changes to this bug.