Description of problem: In my test setup with OSE 3.2.1.1, I have trouble getting hawkular-metrics to work. I can access the service home page just fine, but displaying values via the OpenShift console yields a 403 forbidden. Using the Browser’s Developer Tools, I can see that two OPTIONS requests are made which yield a 200 OK, but then the two GET requests yield 403 forbidden. Version-Release number of selected component (if applicable): [root@ose3-master1 ~]# oc version oc v3.2.1.1-1-g33fa4ea kubernetes v1.2.0-36-g4a3f9c5 [root@ose3-master1 ~]# oc describe pods -n openshift-infra | grep Image Image: registry.access.redhat.com/openshift3/metrics-cassandra:latest Image ID: docker://sha256:afeae5fccd3ff19e691d6edc67e826de6c4b294948b19564aa0542b23786e03a Image: registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest Image ID: docker://sha256:219e26f45297076a110390153f59f12f76dbb082d7ecab01bfae9c4495a1b55e Image: registry.access.redhat.com/openshift3/metrics-heapster:latest Image ID: docker://sha256:eac7eb4e46c4b8f23b032388c21896ac0f3aab0ad10300f373fdcfb254b04f7a Image: registry.access.redhat.com/openshift3/metrics-deployer:latest Image ID: docker://sha256:6b7f57e2c4963a858ac6792eb0dba30ec3cb34f211b3ee4990c5b5c92a96c289 [root@ose3-master1 ~]# How reproducible: 100% in my environment Steps to Reproduce: 1. Deploy hawkular-metrics using the ansible script at: https://github.com/wrichter/hailstorm/blob/master/ansible/roles/layerX_openshift_installer/tasks/deploy_metrics.yml 2. Create test project in OSE 3. Create test pod in project 4. Access Metrics tab for pod 5. Click on the hawkular-metrics link and accept browser certificate 6. Go back to metrics tab and try to view metrics Actual results: An error message: " Metrics are not available. An error occurred getting metrics for container cakephp-example from https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics. If you have network connectivity, this could indicate a misconfiguration. Please contact your system administrator. Forbidden" Expected results: Metrics graphs are displayed Additional info: What I can see in the browser’s developer tools: Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/gauges/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fmemory%2Fusage/data?buckets=61&start=1467232969237 Request Method:OPTIONS Status Code:200 OK Remote Address:10.32.105.36:443 Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/counters/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fcpu%2Fusage/data?buckets=61&start=1467232969237 Request Method:OPTIONS Status Code:200 OK Remote Address:10.32.105.36:443 Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/counters/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fcpu%2Fusage/data?buckets=61&start=1467232969237 Request Method:GET Status Code:403 Forbidden Remote Address:10.32.105.36:443 Request URL:https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/gauges/cakephp-example%2F990b034f-3e3d-11e6-905c-020000000114%2Fmemory%2Fusage/data?buckets=61&start=1467232969237 Request Method:GET Status Code:403 Forbidden Remote Address:10.32.105.36:443 [root@ose3-master1 ~]# export TOKEN="FjpSRsoep20TXl0JmdgBKTwGBfd4jH8ePd3V2v7KJMs" [root@ose3-master1 ~]# curl --insecure -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"apiVersion":"v1", "kind":"SubjectAccessReview", "esource": "pods", "verb":"get", "namespace":"test"}' https://localhost:8443/oapi/v1/subjectaccessreviews { "kind": "SubjectAccessReviewResponse", "apiVersion": "v1", "namespace": "test", "allowed": true, "reason": "allowed by rule in test" }[root@ose3-master1 ~]# curl -s -k -H "Authorization: Bearer $TOKEN" -H "Hawkular-Tenant: test" https://hawkular-metrics.apps.hailstorm3.coe.muc.redhat.com/hawkular/metrics/metrics?type=counter <html><head><title>JBWEB000065: HTTP Status 403 - </title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 403 - </h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u></u></p><p><b>JBWEB000069: description</b> <u>JBWEB000123: Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"></body></html>[root@ose3-master1 ~]#
When you hit the Hawkular Metrics status page (https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status what is the exact version it displays? Also, using IMAGE_VERSION=latest is highly not recommended. That will bring in the latest version, regardless of the underlying OpenShift version you are running. This can cause a lot of problems if you are using an older version of OpenShift and a newer one is released.
The output is {"MetricsService":"STARTED","Implementation-Version":"0.8.2.Final-redhat-1","Built-From-Git-SHA1":"9eb775892235e05bb1a7828af33a572e4945416e"} ACK wrt IMAGE_VERSION=latest (will change this to the specific version number) but if I understand it correctly it should not affect the situation right now since latest is in fact what matches the OpenShift version deployed.
With the help of Ivan McKinley I found the root cause of the problem: Ansible automation code which does NOT work, i.e. yields 403: oc process metrics-deployer-template -n openshift -v IMAGE_VERSION={{ version_tag.stdout }},MASTER_URL=https://openshift.{{ hailstorm_dns_domain }}:8443/,HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.{{ hailstorm_dns_domain }},USE_PERSISTENT_STORAGE=false | oc create -n openshift-infra -f - Ansible automation code which DOES work, i.e. metrics are displayed: oc process metrics-deployer-template -n openshift -v IMAGE_VERSION={{ version_tag.stdout }},MASTER_URL=https://openshift.{{ hailstorm_dns_domain }}:8443,HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.{{ hailstorm_dns_domain }},USE_PERSISTENT_STORAGE=false | oc create -n openshift-infra -f - --> The difference is in the trailing slash of the MASTER_URL parameter. I have the feeling that the deployer could be enhanced to trim away the trailing slash and/or big warning signs should be added to the docs not to use a trailing slash. Rereading the docs I can probably skip specifying MASTER_URL and IMAGE_VERSION for my specific use cases.
We will update the deployer to take into consideration that the MASTER_URL should not have a trailing slash.
This has been updated in origin-metrics and will be fixed in our next release.
Tested with the latest origin-metrics images on docker hub, deployed metrics with trailing slash in the end of MASTER_URL (MASTER_URL=https://{master-dns}:8443/), the metrics stacks can be successfully deployed, metrics stacks running fine. Set to verified. Images tested: openshift/origin-metrics-cassandra c8821a831af2 openshift/origin-metrics-hawkular-metrics 72055177c8f0 openshift/origin-metrics-heapster 8c577a6146f1 openshift/origin-metrics-deployer 9a4f68918761 openshift version: openshift v1.3.0-alpha.2+214f70b kubernetes v1.3.0+57fb9ac etcd 2.3.0+git # oc get po NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-ravat 1/1 Running 0 43m hawkular-metrics-c9zgg 1/1 Running 0 43m heapster-z29wt 1/1 Running 0 43m metrics-deployer-ron2x 0/1 Completed 0 47m
I also checked that the metrics charts are displayed with data on webconsole in my env of comment #6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1933