Bug 1309938 - The hpa can't get metrics info again
The hpa can't get metrics info again
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Metrics (Show other bugs)
3.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Matt Wringe
chunchen
: Regression
Depends On: 1289503
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-18 22:28 EST by DeShuai Ma
Modified: 2016-09-29 22:17 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1289503
Environment:
Last Closed: 2016-05-12 13:12:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description DeShuai Ma 2016-02-18 22:28:45 EST
+++ This bug was initially created as a clone of Bug #1289503 +++

Description of problem:
hpa can't get metrics info from heaster, the hpa current always is "<waiting>"

Version-Release number of selected component (if applicable):
openshift stage env

How reproducible:
always

Steps to Reproduce:
[root@dhcp-128-7 tmp]# oc get hpa
NAME        REFERENCE                      TARGET    CURRENT     MINPODS   MAXPODS   AGE
mysql-hpa   DeploymentConfig/mysql/scale   10%       <waiting>   1         10        4m
[root@dhcp-128-7 tmp]# oc get dc
NAME      REVISION   REPLICAS   TRIGGERED BY
mysql     1          1          config
[root@dhcp-128-7 tmp]# oc get pod
NAME            READY     STATUS    RESTARTS   AGE
mysql-1-nu4fr   1/1       Running   0          6m
[root@dhcp-128-7 tmp]# oc describe hpa/mysql-hpa
Name:				mysql-hpa
Namespace:			dma1
Labels:				<none>
CreationTimestamp:		Fri, 19 Feb 2016 11:22:37 +0800
Reference:			DeploymentConfig/mysql/scale
Target CPU utilization:		10%
Current CPU utilization:	<not available>
Min replicas:			1
Max replicas:			10

[root@dhcp-128-7 tmp]# oc get event
FIRSTSEEN   LASTSEEN   COUNT     NAME             KIND                      SUBOBJECT                           TYPE      REASON                  SOURCE                                   MESSAGE
5m          1m         9         mysql-hpa        HorizontalPodAutoscaler                                                 FailedGetMetrics        {horizontal-pod-autoscaler }             failed to get CPU consumption and request: some pods do not have request for cpu
5m          1m         9         mysql-hpa        HorizontalPodAutoscaler                                                 FailedComputeReplicas   {horizontal-pod-autoscaler }             failed to get cpu utilization: failed to get CPU consumption and request: some pods do not have request for cpu
6m          6m         1         mysql            DeploymentConfig                                                        DeploymentCreated       {deploymentconfig-controller }           Created new deployment "mysql-1" for version 1
Comment 1 DeShuai Ma 2016-02-19 03:23:47 EST
1. Test on stage env, hpa can't get merics 

2. Regression test again on latest origin, hpa can't get metrics again.
[root@ip-172-18-5-69 sample-app]# openshift version
openshift v1.1.1-21-gbc1a879
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.2.2

Some error log on origin is:

I0219 08:19:53.243852    5471 metrics_client.go:137] Sum of cpu requested: {0.200 DecimalSI}
I0219 08:19:53.245464    5471 proxy.go:124] [679a2064a2d04973: /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z] using transport *http.Transport...
I0219 08:19:53.245497    5471 proxy.go:176] [679a2064a2d04973] Beginning proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245511    5471 proxy.go:185] [679a2064a2d04973] Not making a reriting transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245532    5471 proxy.go:188] [679a2064a2d04973] making a transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245876    5471 proxy.go:178] [679a2064a2d04973] Proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z finished 380.162ยตs.
W0219 08:19:53.246035    5471 horizontal.go:189] Failed to reconcile php-apache: failed to compute desired number of replicas based on CPU utilization for DeploymentConfig/dma/php-apache: failed to get cpu utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'U' looking for beginning of value
Comment 2 Matt Wringe 2016-02-19 09:08:09 EST
It appears you are using the origin-metrics components? Is this correct?

Do you know what the IMAGE_ID is for the heapster docker image?
Comment 3 Matt Wringe 2016-02-19 12:20:32 EST
I cannot reproduce accessing Heapster on v3.1.1.6 with the 3.1.1 metrics components.

What happens when you try and run the following command:

curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://${HOST}:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model

This command will test if the API proxy is able to properly access the Heapster service
Comment 4 DeShuai Ma 2016-02-22 00:36:59 EST
1. On origin I build the image from: https://github.com/openshift/origin-metrics/blob/master/hack/build-images.sh
[root@ip-172-18-6-122 sample-app]# curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://ec2-52-91-177-211.compute-1.amazonaws.com:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model
Unauthorized

2. For stage env, it's deployed by ops, our qe can't change the env.
Image info: 
    registry.access.redhat.com/openshift3/metrics-hawkular-metrics               3.1.1               cd0c2b9424bb        3 weeks ago         1.109 GB
    registry.access.redhat.com/openshift3/metrics-deployer                       3.1.1               f840d6881ae4        3 weeks ago         599.9 MB
    registry.access.redhat.com/openshift3/metrics-cassandra                      3.1.1               227b99b94d05        3 weeks ago         472.7 MB
    registry.access.redhat.com/openshift3/metrics-heapster                       3.1.1               32b5f5bab5e7        3 weeks ago         228.4 MB

Error is similar with this: https://github.com/openshift/origin/issues/6293#issuecomment-183102390
Comment 5 Matt Wringe 2016-02-22 11:26:53 EST
For origin, are you sure you are pulling in the most recent Heapster base image? Can you also try with the latest from docker hub instead of building locally?

For OSE, what is the result of that curl command?
Comment 9 DeShuai Ma 2016-03-09 00:29:24 EST
Now on the stage env, hpa can get metrics info. As our qe has no admin permission, we don't know what's happened.
I think this bug can be verified.

[root@dhcp-128-7 kubernetes]# oc get hpa
NAME      REFERENCE                      TARGET    CURRENT   MINPODS   MAXPODS   AGE
mysql     DeploymentConfig/mysql/scale   80%       0%        1         10        14d

Thanks for your help.
Comment 11 DeShuai Ma 2016-03-09 20:30:05 EST
Verify this bug works well now, if something error again. I will reopen this bug.

Note You need to log in before you can comment on or make changes to this bug.