Bug 1477223 - Ordinary user can not view their newly created pods' metrics within their project in web console, at the same time, their old pods' metrics data is still visible
Ordinary user can not view their newly created pods' metrics within their pro...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metrics (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: 3.6.z
Assigned To: Matt Wringe
Junqi Zhao
: NeedsTestCase, Regression
Depends On:
Blocks: 1477868
  Show dependency treegraph
 
Reported: 2017-08-01 09:51 EDT by Junqi Zhao
Modified: 2017-09-07 23:15 EDT (History)
7 users (show)

See Also:
Fixed In Version: 3.6.0
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1477868 (view as bug list)
Environment:
Last Closed: 2017-09-07 23:15:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ordinary user can not view their pod's metrics, no diagram in web console (100.66 KB, image/png)
2017-08-01 09:51 EDT, Junqi Zhao
no flags Details
metrics route could be accessed (83.86 KB, image/png)
2017-08-01 09:52 EDT, Junqi Zhao
no flags Details
clster admin can view user's metrics diagram (109.74 KB, image/png)
2017-08-01 09:53 EDT, Junqi Zhao
no flags Details
events and hawkular_metrics pod log (97.68 KB, text/plain)
2017-08-01 21:06 EDT, Junqi Zhao
no flags Details
network diagnostics snapshot (197.91 KB, image/png)
2017-08-01 21:06 EDT, Junqi Zhao
no flags Details
network XHR snapshot (331.82 KB, image/png)
2017-08-01 21:45 EDT, Junqi Zhao
no flags Details
CORS Blocked Error (152.71 KB, image/png)
2017-08-01 23:04 EDT, yapei
no flags Details
Issue is fixed, ordinary user could view their pod's metrics (109.52 KB, image/png)
2017-08-02 21:27 EDT, Junqi Zhao
no flags Details

  None (edit)
Description Junqi Zhao 2017-08-01 09:51:33 EDT
Created attachment 1307605 [details]
ordinary user can not view their pod's metrics, no diagram in web console

Description of problem:
Ordinary user, not cluster admin, create one project and deploy pods within the project, when logging in web console, under Metrics tab, there is "Metrics are not available" error:
********************************************************************************
Metrics are not available.
An error occurred getting metrics for container java-mainclass from https://hawkular-metrics.0801-ob0.qe.rhcloud.com/hawkular/metrics. 
********************************************************************************

But when you click the metrics route, it could be accessed, and cluster admin can view the metrics diagram of Ordinary user's project pods, see the attached pictures.


Version-Release number of selected component (if applicable):
# openshift version
openshift v3.6.173.0.1
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

Images from brew
metrics-hawkular-metrics:v3.6.173.0.1-1
metrics-cassandra:v3.6.173.0.1-1
metrics-heapster:v3.6.173.0.1-1

How reproducible:
Always

Steps to Reproduce:
1. Create one project, and deploy pod in it.
such as:
$ oc new-project java
$ oc new-app --docker-image=docker.io/chunyunchen/java-mainclass:2.2.94-SNAPSHOT

2. Check the metrics in web console after the pod change to running status.
3.

Actual results:
pods' metrics can not be viewed in web console

Expected results:
pods' metrics should be viewed in web console

Additional info:
Comment 1 Junqi Zhao 2017-08-01 09:52 EDT
Created attachment 1307606 [details]
metrics route could be accessed
Comment 2 Junqi Zhao 2017-08-01 09:53 EDT
Created attachment 1307607 [details]
clster admin can view user's metrics diagram
Comment 3 Samuel Padgett 2017-08-01 12:22:11 EDT
Status code -1 might mean the browser blocked the HTTP request because the CORS preflight check failed. Is it possible that Hawkular was not ready when you first tested as the ordinary user?

If this is reproducible, I'd like to see what's in the browser developer tools network tab (Tools -> Developer Tools -> Network and refresh the page). Please take a screenshot of what you see.

Also check the events in the openshift-infra namespace and attach the logs for the hawkular-metrics pod (also in the openshift-infra namespace).
Comment 4 Junqi Zhao 2017-08-01 21:05:17 EDT
(In reply to Samuel Padgett from comment #3)
> Status code -1 might mean the browser blocked the HTTP request because the
> CORS preflight check failed. Is it possible that Hawkular was not ready when
> you first tested as the ordinary user?

I did the testing after all the pods became ready, and the browser did not block the HTTP request because I tested metrics 3.4.1, it did not have this issue.

> If this is reproducible, I'd like to see what's in the browser developer
> tools network tab (Tools -> Developer Tools -> Network and refresh the
> page). Please take a screenshot of what you see.
Status code: 304, not modified, see the attached picture

> Also check the events in the openshift-infra namespace and attach the logs
> for the hawkular-metrics pod (also in the openshift-infra namespace).

See the attached file
Comment 5 Junqi Zhao 2017-08-01 21:06 EDT
Created attachment 1307852 [details]
events and hawkular_metrics pod log
Comment 6 Junqi Zhao 2017-08-01 21:06 EDT
Created attachment 1307853 [details]
network diagnostics snapshot
Comment 7 Samuel Padgett 2017-08-01 21:28:23 EDT
Can you make sure the XHR tab is selected when you check the network requests? Thanks!
Comment 8 Samuel Padgett 2017-08-01 21:42:23 EDT
(In reply to Junqi Zhao from comment #4)

> I did the testing after all the pods became ready, and the browser did not
> block the HTTP request because I tested metrics 3.4.1, it did not have this
> issue.

The browser will block the request if the HTTP OPTIONS preflight check does not have the right CORS response headers even if it worked in 3.4. When this happens, you usually see status -1. This is why I'm hoping to see the network tab for XHR specifically.

You might also check to see if there are any errors in the Firefox JavaScript console.

Thank you for the events and logs.
Comment 9 Junqi Zhao 2017-08-01 21:45 EDT
Created attachment 1307856 [details]
network XHR snapshot
Comment 11 yapei 2017-08-01 23:00:14 EDT
Yeah, checked on Junqi's testing environment and there is COR blocked error:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://hawkular-metrics.0801-m9s.qe.rhcloud.com/hawkular/metrics/gauges/dctest-1%2F240bee90-771e-11e7-9dea-fa163e197345%2Fcpu%2Fusage_rate/data?bucketDuration=120000ms&start=-60mn. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)

Details attached in screenshot
Comment 12 yapei 2017-08-01 23:04:09 EDT
Typo in my comments, s/COR/CORS/g
Comment 13 yapei 2017-08-01 23:04 EDT
Created attachment 1307883 [details]
CORS Blocked Error
Comment 14 Matt Wringe 2017-08-02 12:40:10 EDT
Its not cors, the subjectaccessreview that Hawkular Metrics uses also returns back the expected results.

We did have a recent change to add in another filter to Hawkular Metrics, but if that is affecting things, you should be getting a 500 error and not 403.

Investigating further
Comment 15 Matt Wringe 2017-08-02 12:46:50 EDT
Can you please attach the logs for hawkular metrics, cassandra, and heapster. As well as the output of 'oc get pods -o yaml -n openshift-infra'?
Comment 16 Matt Wringe 2017-08-02 13:07:58 EDT
I can reproduce and I think I know what the problem is. Hopefully I will have an update soon.
Comment 20 Junqi Zhao 2017-08-02 21:26:00 EDT
Issue is fixed, ordinary user can not view their pods' metrics within their project. Please change the status to ON_QA.

Images from brew
metrics-hawkular-metrics:v3.6.173.0.3-2
metrics-cassandra:v3.6.173.0.3-1
metrics-heapster:v3.6.173.0.3-1
Comment 21 Junqi Zhao 2017-08-02 21:27 EDT
Created attachment 1308538 [details]
Issue is fixed, ordinary user could view their pod's metrics
Comment 23 Junqi Zhao 2017-08-03 01:29:42 EDT
Close it based on Comment 20
Comment 26 errata-xmlrpc 2017-09-07 23:15:23 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2642

Note You need to log in before you can comment on or make changes to this bug.