Bug 1493057

Summary: HPA V2 cannot get services/unsafeproxy in the namespace "openshift-infra"
Product: OpenShift Container Platform Reporter: Zhang Cheng <chezhang>
Component: apiserver-authAssignee: Mo <mkhan>
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: amcdermo, aos-bugs, dma, jokerman, mkhan, mmccomas, sjenning, sross
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:11:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 1 Seth Jennings 2017-09-19 14:29:41 UTC
Solly, PTAL.  Seems to be a HPA service account permissions issue.

Comment 2 Solly Ross 2017-10-03 18:34:35 UTC
yeah looks like someone changed the name of the proxy subresource, but didn't update the HPA RBAC rules...

Comment 3 Seth Jennings 2017-10-03 18:47:47 UTC
Rebalancing bugs.  Andrew, PTAL.

Comment 4 Andrew McDermott 2017-10-05 19:44:27 UTC
Debugging and investigating with Solly yields what might be the root cause:

  https://github.com/openshift/origin/issues/16710

Comment 5 Solly Ross 2017-10-05 20:09:20 UTC
We also found this (unrelated) HPA issue (an update to the controller initialization borked RBAC for the HPA controller, again): https://github.com/openshift/origin/pull/16711

Comment 6 Andrew McDermott 2017-10-09 11:07:13 UTC
An easy way to reproduce this is to issue:

 $ curl -k -H "Authorization: Bearer $(oc whoami -t)" \
     https://<<<HOSTNAME>>>:8443/api/v1/namespaces/openshift-infra/services/https:heapster:/proxy/apis/metrics/v1/alpha1/nodes

Comment 7 Mo 2017-10-11 00:52:12 UTC
Fixed in https://github.com/openshift/origin/pull/16741

Comment 8 Andrew McDermott 2017-10-11 18:18:47 UTC
I spent some time validating the tip of Origin @ 9f5c91e050 now that:

 - https://github.com/openshift/origin/pull/16741
 - https://github.com/openshift/origin/pull/16711

have been merged.

Given a deployment I was able to add the HPA and then edit metrics  for CPU Request and Limit to small values. Previously this wasn't possible (fixed by 16711). Given very small request and limit values I saw the pod scale out to 3.

Using curl, I now get: 

curl -H "Authorization: Bearer $(oc whoami -t)" \
  https://fedora-dev-vm-2:8443/api/v1/namespaces/openshift-infra/services/https:heapster:/proxy/apis/metrics/v1alpha1/nodes -k

User "test-admin" cannot get services/proxy in the namespace "openshift-infra".

Comment 9 DeShuai Ma 2017-10-24 02:28:45 UTC
Tested the bug on the env openshift v3.7.0-0.158.0, now hpav2 can get the metrics correctly. verify the bug

Comment 13 errata-xmlrpc 2017-11-28 22:11:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188