Bug 1309938

Summary:	The hpa can't get metrics info again
Product:	OpenShift Container Platform	Reporter:	DeShuai Ma <dma>
Component:	Hawkular	Assignee:	Matt Wringe <mwringe>
Status:	CLOSED CURRENTRELEASE	QA Contact:	chunchen <chunchen>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	unspecified	CC:	agoldste, aos-bugs, dmace, dma, jliggitt, mmccomas, mwringe, sdodson, wsun, yinzhou
Target Milestone:	---	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1289503	Environment:
Last Closed:	2016-05-12 17:12:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:	1289503
Bug Blocks:

Description DeShuai Ma 2016-02-19 03:28:45 UTC

+++ This bug was initially created as a clone of Bug #1289503 +++

Description of problem:
hpa can't get metrics info from heaster, the hpa current always is "<waiting>"

Version-Release number of selected component (if applicable):
openshift stage env

How reproducible:
always

Steps to Reproduce:
[root@dhcp-128-7 tmp]# oc get hpa
NAME        REFERENCE                      TARGET    CURRENT     MINPODS   MAXPODS   AGE
mysql-hpa   DeploymentConfig/mysql/scale   10%       <waiting>   1         10        4m
[root@dhcp-128-7 tmp]# oc get dc
NAME      REVISION   REPLICAS   TRIGGERED BY
mysql     1          1          config
[root@dhcp-128-7 tmp]# oc get pod
NAME            READY     STATUS    RESTARTS   AGE
mysql-1-nu4fr   1/1       Running   0          6m
[root@dhcp-128-7 tmp]# oc describe hpa/mysql-hpa
Name:				mysql-hpa
Namespace:			dma1
Labels:				<none>
CreationTimestamp:		Fri, 19 Feb 2016 11:22:37 +0800
Reference:			DeploymentConfig/mysql/scale
Target CPU utilization:		10%
Current CPU utilization:	<not available>
Min replicas:			1
Max replicas:			10

[root@dhcp-128-7 tmp]# oc get event
FIRSTSEEN   LASTSEEN   COUNT     NAME             KIND                      SUBOBJECT                           TYPE      REASON                  SOURCE                                   MESSAGE
5m          1m         9         mysql-hpa        HorizontalPodAutoscaler                                                 FailedGetMetrics        {horizontal-pod-autoscaler }             failed to get CPU consumption and request: some pods do not have request for cpu
5m          1m         9         mysql-hpa        HorizontalPodAutoscaler                                                 FailedComputeReplicas   {horizontal-pod-autoscaler }             failed to get cpu utilization: failed to get CPU consumption and request: some pods do not have request for cpu
6m          6m         1         mysql            DeploymentConfig                                                        DeploymentCreated       {deploymentconfig-controller }           Created new deployment "mysql-1" for version 1

Comment 1 DeShuai Ma 2016-02-19 08:23:47 UTC

1. Test on stage env, hpa can't get merics 

2. Regression test again on latest origin, hpa can't get metrics again.
[root@ip-172-18-5-69 sample-app]# openshift version
openshift v1.1.1-21-gbc1a879
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.2.2

Some error log on origin is:

I0219 08:19:53.243852    5471 metrics_client.go:137] Sum of cpu requested: {0.200 DecimalSI}
I0219 08:19:53.245464    5471 proxy.go:124] [679a2064a2d04973: /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z] using transport *http.Transport...
I0219 08:19:53.245497    5471 proxy.go:176] [679a2064a2d04973] Beginning proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245511    5471 proxy.go:185] [679a2064a2d04973] Not making a reriting transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245532    5471 proxy.go:188] [679a2064a2d04973] making a transport for proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z...
I0219 08:19:53.245876    5471 proxy.go:178] [679a2064a2d04973] Proxy /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/namespaces/dma/pod-list/php-apache-2-82h22/metrics/cpu-usage?start=2016-02-19T08%3A14%3A53Z finished 380.162µs.
W0219 08:19:53.246035    5471 horizontal.go:189] Failed to reconcile php-apache: failed to compute desired number of replicas based on CPU utilization for DeploymentConfig/dma/php-apache: failed to get cpu utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'U' looking for beginning of value

Comment 2 Matt Wringe 2016-02-19 14:08:09 UTC

It appears you are using the origin-metrics components? Is this correct?

Do you know what the IMAGE_ID is for the heapster docker image?

Comment 3 Matt Wringe 2016-02-19 17:20:32 UTC

I cannot reproduce accessing Heapster on v3.1.1.6 with the 3.1.1 metrics components.

What happens when you try and run the following command:

curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://${HOST}:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model

This command will test if the API proxy is able to properly access the Heapster service

Comment 4 DeShuai Ma 2016-02-22 05:36:59 UTC

1. On origin I build the image from: https://github.com/openshift/origin-metrics/blob/master/hack/build-images.sh
[root@ip-172-18-6-122 sample-app]# curl --insecure -H "Authorization: Bearer XXXXXX" -X GET https://ec2-52-91-177-211.compute-1.amazonaws.com:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model
Unauthorized

2. For stage env, it's deployed by ops, our qe can't change the env.
Image info: 
    registry.access.redhat.com/openshift3/metrics-hawkular-metrics               3.1.1               cd0c2b9424bb        3 weeks ago         1.109 GB
    registry.access.redhat.com/openshift3/metrics-deployer                       3.1.1               f840d6881ae4        3 weeks ago         599.9 MB
    registry.access.redhat.com/openshift3/metrics-cassandra                      3.1.1               227b99b94d05        3 weeks ago         472.7 MB
    registry.access.redhat.com/openshift3/metrics-heapster                       3.1.1               32b5f5bab5e7        3 weeks ago         228.4 MB

Error is similar with this: https://github.com/openshift/origin/issues/6293#issuecomment-183102390

Comment 5 Matt Wringe 2016-02-22 16:26:53 UTC

For origin, are you sure you are pulling in the most recent Heapster base image? Can you also try with the latest from docker hub instead of building locally?

For OSE, what is the result of that curl command?

Comment 9 DeShuai Ma 2016-03-09 05:29:24 UTC

Now on the stage env, hpa can get metrics info. As our qe has no admin permission, we don't know what's happened.
I think this bug can be verified.

[root@dhcp-128-7 kubernetes]# oc get hpa
NAME      REFERENCE                      TARGET    CURRENT   MINPODS   MAXPODS   AGE
mysql     DeploymentConfig/mysql/scale   80%       0%        1         10        14d

Thanks for your help.

Comment 11 DeShuai Ma 2016-03-10 01:30:05 UTC

Verify this bug works well now, if something error again. I will reopen this bug.