Hide Forgot
Description of problem: HPA getting "invalid character 'E' looking for beginning of value" but 3.1.1 metrics appears to be running just fine. Metrics was installed normally following the documentation without any fancy custom settings. The rc for the pod attempting to autoscale includes cpu requests. api/controller logs attached.
The invalid character 'E' is usually because the HPA is expecting a value to be returned by Heapster, but in this case Heapster is returning a message. Most likely it is returning a message that starts with 'Error'. Is the metrics components running in the 'openshift-infra' project or under some other namespace? Only the API proxy is configured to have access to the Heapster api. Can you verify that you can access the Heapster api using the proxy? Something like this should work as a cluster admin user: curl -k -H "Authorization: Bearer `oc whoami -t`" -X GET https://localhost:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/ Note that this is usually unrelated to the metric components. So things like metric graphs in the console can continue to function. Did you make any changes to the Metrics deployer's secret (specifically to the changes about which user can access Heapster)? Is this a new issue (as in did the HPA did used to work before?) or is this the first time the HPA has been used?
I am not 100% certain if all the metrics parts are in the openshift-infra namespace but I believe they are as the customer showed the output of `oc get svc -n openshift-infra heapster -o yaml` and things looked appropriate. Customer confirmed they are a cluster admin and ran the following command which produced the output: curl -X GET -H "Authorization: Bearer <token>" http://localhost:8001/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/<EXAMPLE>/pods/<POD>/metrics/cpu-usage Error: 'Forbidden' Trying to reach: 'https://<HOST>:8082/api/v1/model/namespaces/<EXAMPLE>/pods/<POD>/metrics/cpu-usage' Customer specifically said they deployed basic metrics following the documentation with nothing modified. This is the first time that the HPA is attempting to be used in this cluster.
If you cannot access the Heapster via the API proxy then something is wrong with the setup. Can you please check that the api proxy cert is signed by the system CA. From the master configuration directory: `openssl verify -verbose -CAfile ca.crt master.proxy-client.crt` And can you check that the md5sum of the ca.crt matches the ca.crt that Heapster has? [eg `oc exec ${HEAPSTER_POD_NAME} md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt` Can you also check that the master.proxy-client.crt has the proper CN name. `openssl x509 -noout -subject -in master.proxy-client.crt`
Closing this issue as INSUFFICIENT_DATA. Please re-open if you can provide the requested data or a means or reliably reproducing this issue.