Bug 1351730 - Access to Hawkular-Metrics yields 403 Forbidden
Summary: Access to Hawkular-Metrics yields 403 Forbidden
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.2.1
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Matt Wringe
QA Contact: chunchen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-30 16:45 UTC by Wolfram Richter
Modified: 2016-09-30 02:16 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-27 09:39:06 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Description Wolfram Richter 2016-06-30 16:45:06 UTC
Description of problem:

In my test setup with OSE 3.2.1.1, I have trouble getting hawkular-metrics to work. I can access the service home page just fine, but displaying values via the OpenShift console yields a 403 forbidden. Using the Browser’s Developer Tools, I can see that two OPTIONS requests are made which yield a 200 OK, but then the two GET requests yield 403 forbidden. 


Version-Release number of selected component (if applicable):

[root@ose3-master1 ~]# oc version
oc v3.2.1.1-1-g33fa4ea
kubernetes v1.2.0-36-g4a3f9c5
[root@ose3-master1 ~]# oc describe pods -n openshift-infra | grep Image
    Image:	registry.access.redhat.com/openshift3/metrics-cassandra:latest
    Image ID:	docker://sha256:afeae5fccd3ff19e691d6edc67e826de6c4b294948b19564aa0542b23786e03a
    Image:	registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest
    Image ID:	docker://sha256:219e26f45297076a110390153f59f12f76dbb082d7ecab01bfae9c4495a1b55e
    Image:	registry.access.redhat.com/openshift3/metrics-heapster:latest
    Image ID:	docker://sha256:eac7eb4e46c4b8f23b032388c21896ac0f3aab0ad10300f373fdcfb254b04f7a
    Image:	registry.access.redhat.com/openshift3/metrics-deployer:latest
    Image ID:	docker://sha256:6b7f57e2c4963a858ac6792eb0dba30ec3cb34f211b3ee4990c5b5c92a96c289
[root@ose3-master1 ~]#


How reproducible:

100% in my environment

Steps to Reproduce:
1. Deploy hawkular-metrics using the ansible script at: https://github.com/wrichter/hailstorm/blob/master/ansible/roles/layerX_openshift_installer/tasks/deploy_metrics.yml
2. Create test project in OSE
3. Create test pod in project
4. Access Metrics tab for pod
5. Click on the hawkular-metrics link and accept browser certificate
6. Go back to metrics tab and try to view metrics

Actual results:

An error message: " Metrics are not available.

An error occurred getting metrics for container cakephp-example from https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics.

If you have network connectivity, this could indicate a misconfiguration.
Please contact your system administrator.
Forbidden"


Expected results:

Metrics graphs are displayed


Additional info:


What I can see in the browser’s developer tools:

Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/gauges/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fmemory%2Fusage/data?buckets=61&start=1467232969237
Request Method:OPTIONS
Status Code:200 OK
Remote Address:10.32.105.36:443

Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/counters/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fcpu%2Fusage/data?buckets=61&start=1467232969237
Request Method:OPTIONS
Status Code:200 OK
Remote Address:10.32.105.36:443

Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/counters/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fcpu%2Fusage/data?buckets=61&start=1467232969237
Request Method:GET
Status Code:403 Forbidden
Remote Address:10.32.105.36:443

Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/gauges/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fmemory%2Fusage/data?buckets=61&start=1467232969237
Request Method:GET
Status Code:403 Forbidden
Remote Address:10.32.105.36:443


[root@ose3-master1 ~]# export TOKEN="FjpSRsoep20TXl0JmdgBKTwGBfd4jH8ePd3V2v7KJMs"
[root@ose3-master1 ~]# curl --insecure -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"apiVersion":"v1", "kind":"SubjectAccessReview", "esource": "pods", "verb":"get", "namespace":"test"}' https://localhost:8443/oapi/v1/subjectaccessreviews
{
  "kind": "SubjectAccessReviewResponse",
  "apiVersion": "v1",
  "namespace": "test",
  "allowed": true,
  "reason": "allowed by rule in test"
}[root@ose3-master1 ~]# curl -s -k -H "Authorization: Bearer $TOKEN" -H "Hawkular-Tenant: test" https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/metrics?type=counter
<html><head><title>JBWEB000065: HTTP Status 403 - </title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 403 - </h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u></u></p><p><b>JBWEB000069: description</b> <u>JBWEB000123: Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"></body></html>[root@ose3-master1 ~]#

Comment 1 Matt Wringe 2016-06-30 19:40:43 UTC
When you hit the Hawkular Metrics status page (https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status what is the exact version it displays?

Also, using IMAGE_VERSION=latest is highly not recommended. That will bring in the latest version, regardless of the underlying OpenShift version you are running. This can cause a lot of problems if you are using an older version of OpenShift and a newer one is released.

Comment 2 Wolfram Richter 2016-06-30 21:02:38 UTC
The output is 

{"MetricsService":"STARTED","Implementation-Version":"0.8.2.Final-redhat-1","Built-From-Git-SHA1":"9eb775892235e05bb1a7828af33a572e4945416e"}

ACK wrt IMAGE_VERSION=latest (will change this to the specific version number) but if I understand it correctly it should not affect the situation right now since latest is in fact what matches the OpenShift version deployed.

Comment 3 Wolfram Richter 2016-07-05 20:54:22 UTC
With the help of Ivan McKinley I found the root cause of the problem:

Ansible automation code which does NOT work, i.e. yields 403:

oc process metrics-deployer-template -n openshift -v IMAGE_VERSION={{ version_tag.stdout }},MASTER_URL=https://openshift.{{ hailstorm_dns_domain }}:8443/,HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.{{ hailstorm_dns_domain }},USE_PERSISTENT_STORAGE=false | oc create -n openshift-infra -f -


Ansible automation code which DOES work, i.e. metrics are displayed:

oc process metrics-deployer-template -n openshift -v IMAGE_VERSION={{ version_tag.stdout }},MASTER_URL=https://openshift.{{ hailstorm_dns_domain }}:8443,HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.{{ hailstorm_dns_domain }},USE_PERSISTENT_STORAGE=false | oc create -n openshift-infra -f -


--> The difference is in the trailing slash of the MASTER_URL parameter. I have the feeling that the deployer could be enhanced to trim away the trailing slash and/or big warning signs should be added to the docs not to use a trailing slash.

Rereading the docs I can probably skip specifying MASTER_URL and IMAGE_VERSION for my specific use cases.

Comment 4 Matt Wringe 2016-07-20 13:29:43 UTC
We will update the deployer to take into consideration that the MASTER_URL should not have a trailing slash.

Comment 5 Matt Wringe 2016-07-28 19:59:01 UTC
This has been updated in origin-metrics and will be fixed in our next release.

Comment 6 Xia Zhao 2016-07-29 03:35:30 UTC
Tested with the latest origin-metrics images on docker hub, deployed metrics with trailing slash in the end of MASTER_URL (MASTER_URL=https://{master-dns}:8443/), the metrics stacks can be successfully deployed, metrics stacks running fine. Set to verified.

Images tested: 
openshift/origin-metrics-cassandra    c8821a831af2
openshift/origin-metrics-hawkular-metrics    72055177c8f0
openshift/origin-metrics-heapster    8c577a6146f1
openshift/origin-metrics-deployer    9a4f68918761

openshift version:
openshift v1.3.0-alpha.2+214f70b
kubernetes v1.3.0+57fb9ac
etcd 2.3.0+git

# oc get po
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-ravat   1/1       Running     0          43m
hawkular-metrics-c9zgg       1/1       Running     0          43m
heapster-z29wt               1/1       Running     0          43m
metrics-deployer-ron2x       0/1       Completed   0          47m

Comment 7 Xia Zhao 2016-07-29 07:47:43 UTC
I also checked that the metrics charts are displayed with data on webconsole in my env of comment #6

Comment 9 errata-xmlrpc 2016-09-27 09:39:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933


Note You need to log in before you can comment on or make changes to this bug.