Bug 1444600

Created attachment 1273430 [details]
metrics pods logs and rc

Description of problem:

After installing openshift version 3.5 via openshift-ansible (openshift-ansible-3.5.55-1) which downloaded from github. 
I have a problem with hawkular metrics are not showing in dashboard. 

So, I finding the logs of hawkular metrics and I found that heapster logs are showing below:

$ oc log heapster-p7qww -n openshift-infra
W0421 17:21:12.324815   54112 cmd.go:337] log is DEPRECATED and will be removed in a future version. Use logs instead.
Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status
The endpoint check has successfully completed.
Starting Heapster with the following arguments: --source=kubernetes.summary_api:${MASTER_URL}?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=30s --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=$HEAPSTER_PASSWORD&filter=label(container_name:^system.slice.*|^user.slice)
E0421 10:21:05.191928       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.200043       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.213437       1 client.go:243] Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.214178       1 client.go:243] Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.216069       1 client.go:243] Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.216323       1 client.go:243] Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.217593       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.220639       1 client.go:243] Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.227536       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.228527       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.230429       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
E0421 10:21:05.231768       1 client.go:192] Could not update tags: Hawkular returned status code 403, error message: Reply could not be parsed: invalid character '<' looking for beginning of value
. . .


Version-Release number of selected component (if applicable):

OCP 3.5
CentOS Atomic host 7.3.1611

How reproducible:
Customer side

Steps to Reproduce:
1.After installing openshift version 3.5 via openshift-ansible (openshift-ansible-3.5.55-1) which downloaded from github. 
2.
3.

Actual results:


Expected results:


Additional info: