Bug 1320441

Summary: Need a way to avoid proxy access or a documentation to setup NO_PROXY on OpenShift Master for Autoscaler
Product: OpenShift Container Platform Reporter: Kenjiro Nakayama <knakayam>
Component: InstallerAssignee: Brenton Leanhardt <bleanhar>
Status: CLOSED ERRATA QA Contact: Gan Huang <ghuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.0CC: aos-bugs, deads, jialiu, jliggitt, jokerman, mmccomas, rhowe
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-12 16:39:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Kenjiro Nakayama 2016-03-23 08:49:40 UTC
Description of problem:
---
Need to avoid proxy access or a documentation to setup NO_PROXY on OpenShift Master for Autoscalar

When we deployed Autoscalr, OSE Master appearently access to the heapster service's API[1]. Thus, we will get following error under the proxy, since it will access to the proxy withoug NO_PROXY setting.

~~~
failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value (109 times in the last 54 minutes, 6 seconds)
~~~

Need a way to avoid this issue, or just simply guide a way to setup NO_PROXY for Master.

[1] https://github.com/openshift/origin/blob/v1.1.2/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/podautoscaler/metrics/metrics_client.go#L167-L169

How reproducible:

- Under the proxy env
- Deploy metrics
- Deploy Autoscaler
- Check oc get event

Actual results:

- Get following error

~~~
failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value (109 times in the last 54 minutes, 6 seconds)
~~~

Expected results:

- Deploy autoscaler without error

Comment 1 Brenton Leanhardt 2016-03-28 17:57:25 UTC
David,

Will https://github.com/kubernetes/kubernetes/pull/23003 will address this problem for OSE 3.2?

Looking at https://github.com/openshift/origin/blob/master/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/podautoscaler/metrics/metrics_client.go#L196 it seems the autoscaler calls the heapster service via the Master proxy API.  Meaning, until your PR ships in OSE we have no reasonable way to set NO_PROXY for purposes of metrics collection.

Comment 2 David Eads 2016-03-28 18:03:02 UTC
> Will https://github.com/kubernetes/kubernetes/pull/23003 will address this problem for OSE 3.2?

Yes.


> Meaning, until your PR ships in OSE we have no reasonable way to set NO_PROXY for purposes of metrics collection.

I don't know of a way.  Jordan?

Comment 3 Jordan Liggitt 2016-03-28 18:16:40 UTC
There's no reasonable way to set NO_PROXY for things doing IP-based proxying over a large range of IPs (services and pods). You could specifically include the IP assigned to the metrics service in the NO_PROXY env, but that requires an API restart, and would need to be updated if the service was deleted/recreated.

Comment 4 Brenton Leanhardt 2016-03-28 18:32:24 UTC
For purposes of testing I'm moving this ON_QA.  QE should be able to set up a metrics deployment and use an autoscaler in an environment that uses a proxy.  The key for 3.2 testing is that you can now set NO_PROXY to include the pod network IP range.  While the ansible work hasn't happened yet you could manually set:

NO_PROXY="10.1.0.0/16,172.30.0.0/16"

In ansible these are the portal_net and sdn_cluster_network_cidr values if you aren't using the default.

Comment 5 Gan Huang 2016-03-29 10:37:02 UTC
Test aganist atomic-openshift-utils-3.0.66-1.git.0.218b6c2.el7.noarch
openshift v3.2.0.8
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

1.Install ose under the proxy env
2.Deploy metrics 
3.Deploy app
4.Deploy hpa(Autoscaler)
5.The pod of the app can't scale
6.Config NO_PROXY="10.1.0.0/16,172.30.0.0/16", restart master service
7.The pod of the app could scale successfully

Move it to verified.

Comment 6 Kenjiro Nakayama 2016-03-29 15:14:01 UTC
It looks like we have to update our doc([1]or[2]) for v3.1, versions ealier than v3.2.0.8, with "NO_PROXY=<HEAPSTER_SERVICE>"

Is anyone working on it?

[1] https://docs.openshift.com/enterprise/3.1/dev_guide/pod_autoscaling.html
[2] https://docs.openshift.com/enterprise/3.1/install_config/http_proxies.html

Comment 7 Brenton Leanhardt 2016-03-29 18:31:16 UTC
Hi Kenjiro, 

The problem is that ultimate the Master's proxy api is used which proxiest based on a pod IP.  The only way to set NO_PROXY correctly today is with 3.2.

Comment 8 Kenjiro Nakayama 2016-04-01 07:00:28 UTC
I see... Thank you, Brenton
So, 3.1.x can't use Autoscalar under the proxy.
Workaround is remove their "http_proxy" setting. But it means that importing imageStream and image from external site (via proxy) will be failed.

Comment 10 errata-xmlrpc 2016-05-12 16:39:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1065