Description of problem:

A number of customers are concerned with a HorizontalPodAutoscaler (HPA) issue.

When a customer makes use of HPA, they are being bombarded with Events (e.g.: one customer ~9000 events in last 3 days) which report: 

'unable to get metrics for resource cpu: no metrics returned from heapster'

As far as the customer is concerned, HPA is NOT functioning correctly, but there is no issue with their HPA setup at all.  

The Events they are seeing are for pods that are newly created, pods that are terminating, or dead pods. Once a pod is fully functional, there are no further events for this pod, but as pods are scaled up/down frequently with the nature of HPA - there are many Events and log msgs being created, which is becoming increasingly annoying for customers.

The events they are seeing from the UI don't indicate which pod the error message is referring to also, which make it harder for them to understand what's happening.  


Version-Release number of selected component (if applicable):

oc v3.9.41
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.9.41
kubernetes v1.9.1+a0ce1bc657


How reproducible:
always

Steps to Reproduce:

1. Configure HPA: https://docs.openshift.com/container-platform/3.9/dev_guide/pod_autoscaling.html
2. Scale Pods using HPA
3. View events in the project and view Heapster logs.

Actual results:
There are many events and log entries for new pods, terminating pods, failed pods:
unable to get metrics for resource cpu: no metrics returned from heapster

Expected results:
In HPA setup, we should return metrics and create events for healthy pods and not repeatedly creating events for pods that are newly created or terminating.

Additional info: