Hide Forgot
Description of problem: --- Need to avoid proxy access or a documentation to setup NO_PROXY on OpenShift Master for Autoscalar When we deployed Autoscalr, OSE Master appearently access to the heapster service's API[1]. Thus, we will get following error under the proxy, since it will access to the proxy withoug NO_PROXY setting. ~~~ failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value (109 times in the last 54 minutes, 6 seconds) ~~~ Need a way to avoid this issue, or just simply guide a way to setup NO_PROXY for Master. [1] https://github.com/openshift/origin/blob/v1.1.2/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/podautoscaler/metrics/metrics_client.go#L167-L169 How reproducible: - Under the proxy env - Deploy metrics - Deploy Autoscaler - Check oc get event Actual results: - Get following error ~~~ failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value (109 times in the last 54 minutes, 6 seconds) ~~~ Expected results: - Deploy autoscaler without error
David, Will https://github.com/kubernetes/kubernetes/pull/23003 will address this problem for OSE 3.2? Looking at https://github.com/openshift/origin/blob/master/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/podautoscaler/metrics/metrics_client.go#L196 it seems the autoscaler calls the heapster service via the Master proxy API. Meaning, until your PR ships in OSE we have no reasonable way to set NO_PROXY for purposes of metrics collection.
> Will https://github.com/kubernetes/kubernetes/pull/23003 will address this problem for OSE 3.2? Yes. > Meaning, until your PR ships in OSE we have no reasonable way to set NO_PROXY for purposes of metrics collection. I don't know of a way. Jordan?
There's no reasonable way to set NO_PROXY for things doing IP-based proxying over a large range of IPs (services and pods). You could specifically include the IP assigned to the metrics service in the NO_PROXY env, but that requires an API restart, and would need to be updated if the service was deleted/recreated.
For purposes of testing I'm moving this ON_QA. QE should be able to set up a metrics deployment and use an autoscaler in an environment that uses a proxy. The key for 3.2 testing is that you can now set NO_PROXY to include the pod network IP range. While the ansible work hasn't happened yet you could manually set: NO_PROXY="10.1.0.0/16,172.30.0.0/16" In ansible these are the portal_net and sdn_cluster_network_cidr values if you aren't using the default.
Test aganist atomic-openshift-utils-3.0.66-1.git.0.218b6c2.el7.noarch openshift v3.2.0.8 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 1.Install ose under the proxy env 2.Deploy metrics 3.Deploy app 4.Deploy hpa(Autoscaler) 5.The pod of the app can't scale 6.Config NO_PROXY="10.1.0.0/16,172.30.0.0/16", restart master service 7.The pod of the app could scale successfully Move it to verified.
It looks like we have to update our doc([1]or[2]) for v3.1, versions ealier than v3.2.0.8, with "NO_PROXY=<HEAPSTER_SERVICE>" Is anyone working on it? [1] https://docs.openshift.com/enterprise/3.1/dev_guide/pod_autoscaling.html [2] https://docs.openshift.com/enterprise/3.1/install_config/http_proxies.html
Hi Kenjiro, The problem is that ultimate the Master's proxy api is used which proxiest based on a pod IP. The only way to set NO_PROXY correctly today is with 3.2.
I see... Thank you, Brenton So, 3.1.x can't use Autoscalar under the proxy. Workaround is remove their "http_proxy" setting. But it means that importing imageStream and image from external site (via proxy) will be failed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1065