Red Hat Bugzilla – Bug 1309938
The hpa can't get metrics info again
Last modified: 2016-09-29 22:17:15 EDT
+++ This bug was initially created as a clone of Bug #1289503 +++ Description of problem: hpa can't get metrics info from heaster, the hpa current always is "<waiting>" Version-Release number of selected component (if applicable): openshift stage env How reproducible: always Steps to Reproduce: [root@dhcp-128-7 tmp]# oc get hpa NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE mysql-hpa DeploymentConfig/mysql/scale 10% <waiting> 1 10 4m [root@dhcp-128-7 tmp]# oc get dc NAME REVISION REPLICAS TRIGGERED BY mysql 1 1 config [root@dhcp-128-7 tmp]# oc get pod NAME READY STATUS RESTARTS AGE mysql-1-nu4fr 1/1 Running 0 6m [root@dhcp-128-7 tmp]# oc describe hpa/mysql-hpa Name: mysql-hpa Namespace: dma1 Labels: <none> CreationTimestamp: Fri, 19 Feb 2016 11:22:37 +0800 Reference: DeploymentConfig/mysql/scale Target CPU utilization: 10% Current CPU utilization: <not available> Min replicas: 1 Max replicas: 10 [root@dhcp-128-7 tmp]# oc get event FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE 5m 1m 9 mysql-hpa HorizontalPodAutoscaler FailedGetMetrics {horizontal-pod-autoscaler } failed to get CPU consumption and request: some pods do not have request for cpu 5m 1m 9 mysql-hpa HorizontalPodAutoscaler FailedComputeReplicas {horizontal-pod-autoscaler } failed to get cpu utilization: failed to get CPU consumption and request: some pods do not have request for cpu 6m 6m 1 mysql DeploymentConfig DeploymentCreated {deploymentconfig-controller } Created new deployment "mysql-1" for version 1
1. Test on stage env, hpa can't get merics 2. Regression test again on latest origin, hpa can't get metrics again. [root@ip-172-18-5-69 sample-app]# openshift version openshift v1.1.1-21-gbc1a879 kubernetes v1.1.0-origin-1107-g4c8e6f4 etcd 2.2.2 Some error log on origin is: I0219 08:19:53.243852 5471 metrics_client.go:137] Sum of cpu requested: {0.200 DecimalSI} I0219 08:19:53.245464 5471 proxy.go:124] [679a2064a2d04973: /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z] using transport *http.Transport... I0219 08:19:53.245497 5471 proxy.go:176] [679a2064a2d04973] Beginning proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z... I0219 08:19:53.245511 5471 proxy.go:185] [679a2064a2d04973] Not making a reriting transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z... I0219 08:19:53.245532 5471 proxy.go:188] [679a2064a2d04973] making a transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z... I0219 08:19:53.245876 5471 proxy.go:178] [679a2064a2d04973] Proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z finished 380.162µs. W0219 08:19:53.246035 5471 horizontal.go:189] Failed to reconcile php-apache: failed to compute desired number of replicas based on CPU utilization for DeploymentConfig/dma/php-apache: failed to get cpu utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'U' looking for beginning of value
It appears you are using the origin-metrics components? Is this correct? Do you know what the IMAGE_ID is for the heapster docker image?
I cannot reproduce accessing Heapster on v3.1.1.6 with the 3.1.1 metrics components. What happens when you try and run the following command: curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://${HOST}:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model This command will test if the API proxy is able to properly access the Heapster service
1. On origin I build the image from: https://github.com/openshift/origin-metrics/blob/master/hack/build-images.sh [root@ip-172-18-6-122 sample-app]# curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://ec2-52-91-177-211.compute-1.amazonaws.com:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model Unauthorized 2. For stage env, it's deployed by ops, our qe can't change the env. Image info: registry.access.redhat.com/openshift3/metrics-hawkular-metrics 3.1.1 cd0c2b9424bb 3 weeks ago 1.109 GB registry.access.redhat.com/openshift3/metrics-deployer 3.1.1 f840d6881ae4 3 weeks ago 599.9 MB registry.access.redhat.com/openshift3/metrics-cassandra 3.1.1 227b99b94d05 3 weeks ago 472.7 MB registry.access.redhat.com/openshift3/metrics-heapster 3.1.1 32b5f5bab5e7 3 weeks ago 228.4 MB Error is similar with this: https://github.com/openshift/origin/issues/6293#issuecomment-183102390
For origin, are you sure you are pulling in the most recent Heapster base image? Can you also try with the latest from docker hub instead of building locally? For OSE, what is the result of that curl command?
Now on the stage env, hpa can get metrics info. As our qe has no admin permission, we don't know what's happened. I think this bug can be verified. [root@dhcp-128-7 kubernetes]# oc get hpa NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE mysql DeploymentConfig/mysql/scale 80% 0% 1 10 14d Thanks for your help.
Verify this bug works well now, if something error again. I will reopen this bug.