Description of problem: Access forbidden when trying to collect metrics from the management-infra service account (cluster-reader). Version-Release number of selected component (if applicable): openshift3/metrics-hawkular-metrics:3.1.1 (5c02894a36cd) openshift 1.1.4-0.git.0.9d8230b.el7 How reproducible: 100% Steps to Reproduce: 1. Deploy metrics 2. Try to collect metrics using the token of the management-infra service account Actual results: Access to collect metrics is forbidden. Expected results: Access to collect metrics is denied. Additional info: TOKEN='...' # curl -s -k -H "Authorization: Bearer $TOKEN" https://localhost:8443/api/v1/pods | jq '.items[].metadata.name' "docker-registry-2-y86su" "management-metrics-2-rkc91" "router-1-2prer" "cakephp-mysql-example-2-build" "cakephp-mysql-example-2-hp38r" "cakephp-mysql-example-2-xrkb9" "mysql-1-805vp" "postgresql-1-11205" "rails-postgresql-example-3-uf4ht" "rails-postgresql-example-4-build" "hawkular-cassandra-1-6a0ii" "hawkular-metrics-byfcy" "heapster-77imc" # curl -s -k -H "Authorization: Bearer $TOKEN" https://localhost:8443/oapi/v1/users/~ | jq '.groups[]' "system:serviceaccounts" "system:serviceaccounts:management-infra" # oc policy who-can list pods Namespace: default Verb: list Resource: pods Users: system:serviceaccount:default:deployer system:serviceaccount:management-infra:management-admin ... Groups: system:cluster-admins system:cluster-readers ... Failure trying to access the metrics using the same token: # curl -s -k -H "Authorization: Bearer $TOKEN" -H "Hawkular-Tenant: default" https://localhost:5000/hawkular/metrics/metrics?type=counter <html><head><title>JBWEB000065: HTTP Status 403 - </title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 403 - </h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u></u></p><p><b>JBWEB000069: description</b> <u>JBWEB000123: Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"></body></html> It works correctly with the hawkular password: PASSWORD='...' # curl -s -k -u hawkular:$PASSWORD -H "Hawkular-Tenant: default" https://localhost:5000/hawkular/metrics/metrics?type=counter | jq '.[].id' "registry/1658a45f-f26c-11e5-b3a1-5254008047f1/cpu/usage" "registry/1658a45f-f26c-11e5-b3a1-5254008047f1/memory/major_page_faults" "registry/1658a45f-f26c-11e5-b3a1-5254008047f1/memory/page_faults" "registry/1658a45f-f26c-11e5-b3a1-5254008047f1/uptime" "router/14c96344-f26c-11e5-b3a1-5254008047f1/cpu/usage" "router/14c96344-f26c-11e5-b3a1-5254008047f1/memory/major_page_faults" "router/14c96344-f26c-11e5-b3a1-5254008047f1/memory/page_faults" "router/14c96344-f26c-11e5-b3a1-5254008047f1/uptime" "router/ee1cc133-f28c-11e5-b3a1-5254008047f1/cpu/usage" "router/ee1cc133-f28c-11e5-b3a1-5254008047f1/memory/major_page_faults" "router/ee1cc133-f28c-11e5-b3a1-5254008047f1/memory/page_faults" "router/ee1cc133-f28c-11e5-b3a1-5254008047f1/uptime"
@Federico which version of metrics are you using and on what version of OpenShift? There is an issue where the OpenShift API interfaces have slightly changed between 3.1 and 3.2, this means that the 3.1 metrics containers will not be able to properly authenticate individual users and will result in a forbidden error message that you are seeing. It does not affect the basic username/password used, only when using the bearer token. This issue has been resolved in the 3.2 images and those should function properly on OSE 3.2
(In reply to Matt Wringe from comment #1) > @Federico which version of metrics are you using and on what version of > OpenShift? The versions were reported in the bug description. Version-Release number of selected component (if applicable): openshift3/metrics-hawkular-metrics:3.1.1 (5c02894a36cd) openshift 1.1.4-0.git.0.9d8230b.el7
Can you see if this command returns 'allowed': true? curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -X POST -d '{"kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://localhost:8443/oapi/v1/subjectaccessreviews
# curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -X POST -d '{"kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://$HOSTNAME:8443/oapi/v1/subjectaccessreviews { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Object 'apiVersion' is missing in '{\"kind\":\"SubjectAccessReview\", \"resource\": \"pods\", \"verb\":\"list\", \"namespace\":\"default\"}'", "reason": "Forbidden", "details": { "kind": "subjectaccessreviews" }, "code": 403 } # curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -X POST -d '{"apiVersion":"v1", "kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://$HOSTNAME:8443/oapi/v1/subjectaccessreviews { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "the body of the request was in an unknown format - accepted media types include: application/json, application/yaml", "reason": "UnsupportedMediaType", "code": 415 } # curl --insecure -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"apiVersion":"v1", "kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://$HOSTNAME:8443/oapi/v1/subjectaccessreviews { "kind": "SubjectAccessReviewResponse", "apiVersion": "v1", "namespace": "default", "allowed": true, "reason": "allowed by cluster rule" } I think you may be missing the apiVersion and the new required Content-Type header in the request.
Are you absolutely sure you are running on OSE 3.1? That is exactly the issue you will be getting if running the 3.1 metric images on OSE 3.2 I can't reproduce with OSE 3.1
Closing as not a bug since its been revealed that this caused by the 3.1 metric images running on 3.2 which is not supported. For running on OSE 3.2 the 3.2 metric images must be used.