Bug 1506363

Summary:	hpa can't get metrics info
Product:	OpenShift Container Platform	Reporter:	Max Whittingham <mwhittin>
Component:	Hawkular	Assignee:	Solly Ross <sross>
Status:	CLOSED NOTABUG	QA Contact:	Junqi Zhao <juzhao>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.5.1	CC:	aos-bugs, mwhittin, mwringe, snegrea
Target Milestone:	---	Keywords:	OpsBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-10-30 20:33:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Max Whittingham 2017-10-25 19:17:34 UTC

Description of problem:
hpa can't get metrics info from heapster, the hpa current always is "<waiting>"

Version-Release number of selected component (if applicable):
atomic-openshift-3.5.5.31-1.git.0.b6f55a2.el7.x86_64
metrics-hawkular-metrics:v3.5
metrics-heapster:v3.5

How reproducible:
Create an hpa, wait some time, pod hpa never comes out of waiting.

Comment 2 Matt Wringe 2017-10-25 19:40:29 UTC

Is the Heapster pod in the ready state?

Can you attach the logs for Heapster, Hawkular Metrics, and Cassandra. As well as the output of 'oc get pods -n openshift-infra -o yaml'

Comment 4 Max Whittingham 2017-10-25 20:06:03 UTC

(In reply to Matt Wringe from comment #2)
> Is the Heapster pod in the ready state?
> 
> Can you attach the logs for Heapster, Hawkular Metrics, and Cassandra. As
> well as the output of 'oc get pods -n openshift-infra -o yaml'

Yes, the heapster pod is in a ready state. I'll upload the requested output & logs shortly.

Comment 6 Matt Wringe 2017-10-25 20:28:29 UTC

It looks like Heapster is having problems writing to Hawkular Metrics, but that shouldn't affect the HPA (the HPA only needs Heapster, it doesn't need Hawkular Metrics).

I am assigning to Solly.

Comment 7 Solly Ross 2017-10-27 17:32:50 UTC

If there are still no events that show up in `oc describe hpa`, can you grab the controller-manager logs from the active controller manager in your cluster?  We're looking for log lines from the file `horizontal.go` or `metrics_client.go`.