Bug 1321346

Summary: Access forbidden when trying to collect metrics
Product: OpenShift Container Platform Reporter: Federico Simoncelli <fsimonce>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED NOTABUG QA Contact: chunchen <chunchen>
Severity: high Docs Contact:
Priority: high    
Version: 3.1.0CC: aos-bugs, fsimonce, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-31 15:32:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Federico Simoncelli 2016-03-25 17:15:28 UTC
Description of problem:
Access forbidden when trying to collect metrics from the management-infra service account (cluster-reader).

Version-Release number of selected component (if applicable):
openshift3/metrics-hawkular-metrics:3.1.1 (5c02894a36cd)
openshift 1.1.4-0.git.0.9d8230b.el7

How reproducible:
100%

Steps to Reproduce:
1. Deploy metrics
2. Try to collect metrics using the token of the management-infra service account

Actual results:
Access to collect metrics is forbidden.

Expected results:
Access to collect metrics is denied.


Additional info:
TOKEN='...'

# curl -s -k -H "Authorization: Bearer $TOKEN" https://localhost:8443/api/v1/pods | jq '.items[].metadata.name'
"docker-registry-2-y86su"
"management-metrics-2-rkc91"
"router-1-2prer"
"cakephp-mysql-example-2-build"
"cakephp-mysql-example-2-hp38r"
"cakephp-mysql-example-2-xrkb9"
"mysql-1-805vp"
"postgresql-1-11205"
"rails-postgresql-example-3-uf4ht"
"rails-postgresql-example-4-build"
"hawkular-cassandra-1-6a0ii"
"hawkular-metrics-byfcy"
"heapster-77imc"

# curl -s -k -H "Authorization: Bearer $TOKEN" https://localhost:8443/oapi/v1/users/~ | jq '.groups[]'
"system:serviceaccounts"
"system:serviceaccounts:management-infra"

# oc policy who-can list pods
Namespace: default
Verb:      list
Resource:  pods

Users:  system:serviceaccount:default:deployer
        system:serviceaccount:management-infra:management-admin
...
Groups: system:cluster-admins
        system:cluster-readers
...


Failure trying to access the metrics using the same token:

# curl -s -k -H "Authorization: Bearer $TOKEN" -H "Hawkular-Tenant: default" https://localhost:5000/hawkular/metrics/metrics?type=counter
<html><head><title>JBWEB000065: HTTP Status 403 - </title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 403 - </h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u></u></p><p><b>JBWEB000069: description</b> <u>JBWEB000123: Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"></body></html>


It works correctly with the hawkular password:

PASSWORD='...'

# curl -s -k -u hawkular:$PASSWORD -H "Hawkular-Tenant: default" https://localhost:5000/hawkular/metrics/metrics?type=counter | jq '.[].id'
"registry/1658a45f-f26c-11e5-b3a1-5254008047f1/cpu/usage"
"registry/1658a45f-f26c-11e5-b3a1-5254008047f1/memory/major_page_faults"
"registry/1658a45f-f26c-11e5-b3a1-5254008047f1/memory/page_faults"
"registry/1658a45f-f26c-11e5-b3a1-5254008047f1/uptime"
"router/14c96344-f26c-11e5-b3a1-5254008047f1/cpu/usage"
"router/14c96344-f26c-11e5-b3a1-5254008047f1/memory/major_page_faults"
"router/14c96344-f26c-11e5-b3a1-5254008047f1/memory/page_faults"
"router/14c96344-f26c-11e5-b3a1-5254008047f1/uptime"
"router/ee1cc133-f28c-11e5-b3a1-5254008047f1/cpu/usage"
"router/ee1cc133-f28c-11e5-b3a1-5254008047f1/memory/major_page_faults"
"router/ee1cc133-f28c-11e5-b3a1-5254008047f1/memory/page_faults"
"router/ee1cc133-f28c-11e5-b3a1-5254008047f1/uptime"

Comment 1 Matt Wringe 2016-03-28 13:58:11 UTC
@Federico which version of metrics are you using and on what version of OpenShift?

There is an issue where the OpenShift API interfaces have slightly changed between 3.1 and 3.2, this means that the 3.1 metrics containers will not be able to properly authenticate individual users and will result in a forbidden error message that you are seeing. It does not affect the basic username/password used, only when using the bearer token.

This issue has been resolved in the 3.2 images and those should function properly on OSE 3.2

Comment 2 Federico Simoncelli 2016-03-29 08:30:18 UTC
(In reply to Matt Wringe from comment #1)
> @Federico which version of metrics are you using and on what version of
> OpenShift?

The versions were reported in the bug description.

Version-Release number of selected component (if applicable):
openshift3/metrics-hawkular-metrics:3.1.1 (5c02894a36cd)
openshift 1.1.4-0.git.0.9d8230b.el7

Comment 3 Matt Wringe 2016-03-30 13:26:23 UTC
Can you see if this command returns 'allowed': true?

curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -X POST -d '{"kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://localhost:8443/oapi/v1/subjectaccessreviews

Comment 4 Federico Simoncelli 2016-03-30 14:45:07 UTC
# curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -X POST -d '{"kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://$HOSTNAME:8443/oapi/v1/subjectaccessreviews
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Object 'apiVersion' is missing in '{\"kind\":\"SubjectAccessReview\", \"resource\": \"pods\", \"verb\":\"list\", \"namespace\":\"default\"}'",
  "reason": "Forbidden",
  "details": {
    "kind": "subjectaccessreviews"
  },
  "code": 403
}

# curl --insecure -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -X POST -d '{"apiVersion":"v1", "kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://$HOSTNAME:8443/oapi/v1/subjectaccessreviews
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "the body of the request was in an unknown format - accepted media types include: application/json, application/yaml",
  "reason": "UnsupportedMediaType",
  "code": 415
}

# curl --insecure -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"apiVersion":"v1", "kind":"SubjectAccessReview", "resource": "pods", "verb":"list", "namespace":"default"}' https://$HOSTNAME:8443/oapi/v1/subjectaccessreviews
{
  "kind": "SubjectAccessReviewResponse",
  "apiVersion": "v1",
  "namespace": "default",
  "allowed": true,
  "reason": "allowed by cluster rule"
}

I think you may be missing the apiVersion and the new required Content-Type header in the request.

Comment 5 Matt Wringe 2016-03-30 15:19:37 UTC
Are you absolutely sure you are running on OSE 3.1? That is exactly the issue you will be getting if running the 3.1 metric images on OSE 3.2

I can't reproduce with OSE 3.1

Comment 6 Matt Wringe 2016-03-31 15:32:07 UTC
Closing as not a bug since its been revealed that this caused by the 3.1 metric images running on 3.2 which is not supported.

For running on OSE 3.2 the 3.2 metric images must be used.