Bug 1477223

Summary: Ordinary user can not view their newly created pods' metrics within their project in web console, at the same time, their old pods' metrics data is still visible
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, juzhao, qitang, spadgett, trankin, xiazhao, xtian, yapei
Target Milestone: ---Keywords: Regression
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 3.6.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1477868 (view as bug list) Environment:
Last Closed: 2017-09-08 03:15:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1477868    
Attachments:
Description Flags
ordinary user can not view their pod's metrics, no diagram in web console
none
metrics route could be accessed
none
clster admin can view user's metrics diagram
none
events and hawkular_metrics pod log
none
network diagnostics snapshot
none
network XHR snapshot
none
CORS Blocked Error
none
Issue is fixed, ordinary user could view their pod's metrics none

Description Junqi Zhao 2017-08-01 13:51:33 UTC
Created attachment 1307605 [details]
ordinary user can not view their pod's metrics, no diagram in web console

Description of problem:
Ordinary user, not cluster admin, create one project and deploy pods within the project, when logging in web console, under Metrics tab, there is "Metrics are not available" error:
********************************************************************************
Metrics are not available.
An error occurred getting metrics for container java-mainclass from https://hawkular-metrics.0801-ob0.qe.rhcloud.com/hawkular/metrics. 
********************************************************************************

But when you click the metrics route, it could be accessed, and cluster admin can view the metrics diagram of Ordinary user's project pods, see the attached pictures.


Version-Release number of selected component (if applicable):
# openshift version
openshift v3.6.173.0.1
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

Images from brew
metrics-hawkular-metrics:v3.6.173.0.1-1
metrics-cassandra:v3.6.173.0.1-1
metrics-heapster:v3.6.173.0.1-1

How reproducible:
Always

Steps to Reproduce:
1. Create one project, and deploy pod in it.
such as:
$ oc new-project java
$ oc new-app --docker-image=docker.io/chunyunchen/java-mainclass:2.2.94-SNAPSHOT

2. Check the metrics in web console after the pod change to running status.
3.

Actual results:
pods' metrics can not be viewed in web console

Expected results:
pods' metrics should be viewed in web console

Additional info:

Comment 1 Junqi Zhao 2017-08-01 13:52:54 UTC
Created attachment 1307606 [details]
metrics route could be accessed

Comment 2 Junqi Zhao 2017-08-01 13:53:31 UTC
Created attachment 1307607 [details]
clster admin can view user's metrics diagram

Comment 3 Samuel Padgett 2017-08-01 16:22:11 UTC
Status code -1 might mean the browser blocked the HTTP request because the CORS preflight check failed. Is it possible that Hawkular was not ready when you first tested as the ordinary user?

If this is reproducible, I'd like to see what's in the browser developer tools network tab (Tools -> Developer Tools -> Network and refresh the page). Please take a screenshot of what you see.

Also check the events in the openshift-infra namespace and attach the logs for the hawkular-metrics pod (also in the openshift-infra namespace).

Comment 4 Junqi Zhao 2017-08-02 01:05:17 UTC
(In reply to Samuel Padgett from comment #3)
> Status code -1 might mean the browser blocked the HTTP request because the
> CORS preflight check failed. Is it possible that Hawkular was not ready when
> you first tested as the ordinary user?

I did the testing after all the pods became ready, and the browser did not block the HTTP request because I tested metrics 3.4.1, it did not have this issue.

> If this is reproducible, I'd like to see what's in the browser developer
> tools network tab (Tools -> Developer Tools -> Network and refresh the
> page). Please take a screenshot of what you see.
Status code: 304, not modified, see the attached picture

> Also check the events in the openshift-infra namespace and attach the logs
> for the hawkular-metrics pod (also in the openshift-infra namespace).

See the attached file

Comment 5 Junqi Zhao 2017-08-02 01:06:08 UTC
Created attachment 1307852 [details]
events and hawkular_metrics pod log

Comment 6 Junqi Zhao 2017-08-02 01:06:45 UTC
Created attachment 1307853 [details]
network diagnostics snapshot

Comment 7 Samuel Padgett 2017-08-02 01:28:23 UTC
Can you make sure the XHR tab is selected when you check the network requests? Thanks!

Comment 8 Samuel Padgett 2017-08-02 01:42:23 UTC
(In reply to Junqi Zhao from comment #4)

> I did the testing after all the pods became ready, and the browser did not
> block the HTTP request because I tested metrics 3.4.1, it did not have this
> issue.

The browser will block the request if the HTTP OPTIONS preflight check does not have the right CORS response headers even if it worked in 3.4. When this happens, you usually see status -1. This is why I'm hoping to see the network tab for XHR specifically.

You might also check to see if there are any errors in the Firefox JavaScript console.

Thank you for the events and logs.

Comment 9 Junqi Zhao 2017-08-02 01:45:15 UTC
Created attachment 1307856 [details]
network XHR snapshot

Comment 11 Yadan Pei 2017-08-02 03:00:14 UTC
Yeah, checked on Junqi's testing environment and there is COR blocked error:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://hawkular-metrics.0801-m9s.qe.rhcloud.com/hawkular/metrics/gauges/dctest-1%2F240bee90-771e-11e7-9dea-fa163e197345%2Fcpu%2Fusage_rate/data?bucketDuration=120000ms&start=-60mn. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)

Details attached in screenshot

Comment 12 Yadan Pei 2017-08-02 03:04:09 UTC
Typo in my comments, s/COR/CORS/g

Comment 13 Yadan Pei 2017-08-02 03:04:38 UTC
Created attachment 1307883 [details]
CORS Blocked Error

Comment 14 Matt Wringe 2017-08-02 16:40:10 UTC
Its not cors, the subjectaccessreview that Hawkular Metrics uses also returns back the expected results.

We did have a recent change to add in another filter to Hawkular Metrics, but if that is affecting things, you should be getting a 500 error and not 403.

Investigating further

Comment 15 Matt Wringe 2017-08-02 16:46:50 UTC
Can you please attach the logs for hawkular metrics, cassandra, and heapster. As well as the output of 'oc get pods -o yaml -n openshift-infra'?

Comment 16 Matt Wringe 2017-08-02 17:07:58 UTC
I can reproduce and I think I know what the problem is. Hopefully I will have an update soon.

Comment 20 Junqi Zhao 2017-08-03 01:26:00 UTC
Issue is fixed, ordinary user can not view their pods' metrics within their project. Please change the status to ON_QA.

Images from brew
metrics-hawkular-metrics:v3.6.173.0.3-2
metrics-cassandra:v3.6.173.0.3-1
metrics-heapster:v3.6.173.0.3-1

Comment 21 Junqi Zhao 2017-08-03 01:27:00 UTC
Created attachment 1308538 [details]
Issue is fixed, ordinary user could view their pod's metrics

Comment 23 Junqi Zhao 2017-08-03 05:29:42 UTC
Close it based on Comment 20

Comment 26 errata-xmlrpc 2017-09-08 03:15:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2642