Bug 1309938 - The hpa can't get metrics info again
Summary: The hpa can't get metrics info again
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: unspecified
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Matt Wringe
QA Contact: chunchen
Keywords: Regression
Depends On: 1289503
TreeView+ depends on / blocked
Reported: 2016-02-19 03:28 UTC by DeShuai Ma
Modified: 2018-07-26 19:09 UTC (History)
10 users (show)

Clone Of: 1289503
Last Closed: 2016-05-12 17:12:49 UTC

Attachments (Terms of Use)

Description DeShuai Ma 2016-02-19 03:28:45 UTC
+++ This bug was initially created as a clone of Bug #1289503 +++

Description of problem:
hpa can't get metrics info from heaster, the hpa current always is "<waiting>"

Version-Release number of selected component (if applicable):
openshift stage env

How reproducible:

Steps to Reproduce:
[root@dhcp-128-7 tmp]# oc get hpa
NAME        REFERENCE                      TARGET    CURRENT     MINPODS   MAXPODS   AGE
mysql-hpa   DeploymentConfig/mysql/scale   10%       <waiting>   1         10        4m
[root@dhcp-128-7 tmp]# oc get dc
mysql     1          1          config
[root@dhcp-128-7 tmp]# oc get pod
mysql-1-nu4fr   1/1       Running   0          6m
[root@dhcp-128-7 tmp]# oc describe hpa/mysql-hpa
Name:				mysql-hpa
Namespace:			dma1
Labels:				<none>
CreationTimestamp:		Fri, 19 Feb 2016 11:22:37 +0800
Reference:			DeploymentConfig/mysql/scale
Target CPU utilization:		10%
Current CPU utilization:	<not available>
Min replicas:			1
Max replicas:			10

[root@dhcp-128-7 tmp]# oc get event
FIRSTSEEN   LASTSEEN   COUNT     NAME             KIND                      SUBOBJECT                           TYPE      REASON                  SOURCE                                   MESSAGE
5m          1m         9         mysql-hpa        HorizontalPodAutoscaler                                                 FailedGetMetrics        {horizontal-pod-autoscaler }             failed to get CPU consumption and request: some pods do not have request for cpu
5m          1m         9         mysql-hpa        HorizontalPodAutoscaler                                                 FailedComputeReplicas   {horizontal-pod-autoscaler }             failed to get cpu utilization: failed to get CPU consumption and request: some pods do not have request for cpu
6m          6m         1         mysql            DeploymentConfig                                                        DeploymentCreated       {deploymentconfig-controller }           Created new deployment "mysql-1" for version 1

Comment 1 DeShuai Ma 2016-02-19 08:23:47 UTC
1. Test on stage env, hpa can't get merics 

2. Regression test again on latest origin, hpa can't get metrics again.
[root@ip-172-18-5-69 sample-app]# openshift version
openshift v1.1.1-21-gbc1a879
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.2.2

Some error log on origin is:

I0219 08:19:53.243852    5471 metrics_client.go:137] Sum of cpu requested: {0.200 DecimalSI}
I0219 08:19:53.245464    5471 proxy.go:124] [679a2064a2d04973: /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z] using transport *http.Transport...
I0219 08:19:53.245497    5471 proxy.go:176] [679a2064a2d04973] Beginning proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245511    5471 proxy.go:185] [679a2064a2d04973] Not making a reriting transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245532    5471 proxy.go:188] [679a2064a2d04973] making a transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245876    5471 proxy.go:178] [679a2064a2d04973] Proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z finished 380.162µs.
W0219 08:19:53.246035    5471 horizontal.go:189] Failed to reconcile php-apache: failed to compute desired number of replicas based on CPU utilization for DeploymentConfig/dma/php-apache: failed to get cpu utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'U' looking for beginning of value

Comment 2 Matt Wringe 2016-02-19 14:08:09 UTC
It appears you are using the origin-metrics components? Is this correct?

Do you know what the IMAGE_ID is for the heapster docker image?

Comment 3 Matt Wringe 2016-02-19 17:20:32 UTC
I cannot reproduce accessing Heapster on v3.1.1.6 with the 3.1.1 metrics components.

What happens when you try and run the following command:

curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://${HOST}:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model

This command will test if the API proxy is able to properly access the Heapster service

Comment 4 DeShuai Ma 2016-02-22 05:36:59 UTC
1. On origin I build the image from: https://github.com/openshift/origin-metrics/blob/master/hack/build-images.sh
[root@ip-172-18-6-122 sample-app]# curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://ec2-52-91-177-211.compute-1.amazonaws.com:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model

2. For stage env, it's deployed by ops, our qe can't change the env.
Image info: 
    registry.access.redhat.com/openshift3/metrics-hawkular-metrics               3.1.1               cd0c2b9424bb        3 weeks ago         1.109 GB
    registry.access.redhat.com/openshift3/metrics-deployer                       3.1.1               f840d6881ae4        3 weeks ago         599.9 MB
    registry.access.redhat.com/openshift3/metrics-cassandra                      3.1.1               227b99b94d05        3 weeks ago         472.7 MB
    registry.access.redhat.com/openshift3/metrics-heapster                       3.1.1               32b5f5bab5e7        3 weeks ago         228.4 MB

Error is similar with this: https://github.com/openshift/origin/issues/6293#issuecomment-183102390

Comment 5 Matt Wringe 2016-02-22 16:26:53 UTC
For origin, are you sure you are pulling in the most recent Heapster base image? Can you also try with the latest from docker hub instead of building locally?

For OSE, what is the result of that curl command?

Comment 9 DeShuai Ma 2016-03-09 05:29:24 UTC
Now on the stage env, hpa can get metrics info. As our qe has no admin permission, we don't know what's happened.
I think this bug can be verified.

[root@dhcp-128-7 kubernetes]# oc get hpa
mysql     DeploymentConfig/mysql/scale   80%       0%        1         10        14d

Thanks for your help.

Comment 11 DeShuai Ma 2016-03-10 01:30:05 UTC
Verify this bug works well now, if something error again. I will reopen this bug.

Note You need to log in before you can comment on or make changes to this bug.