Bug 1634680

Description of problem:

When we try to see the etcd metrics in prometheus, we see all with value "0".
The etcd metrics seems to be fetched by prometheus from the api. If we search for those metrics in the api (/metrics), we see all to 0. If we search for those metrics in the etcd endpoint, we see the correct values.

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.9
OpenShift Container Platform 3.10
Prometheus

How reproducible:
create fresh cluster 3.9.33, 3.9.41 or 3.10.34
run following commands on master-1, in project where prometheus is deployed

export TOKEN=`oc serviceaccounts get-token prometheus`

// incorrect values
curl -k -H "Authorization: Bearer $TOKEN" https://`hostname`:8443/metrics | grep -v "#" | grep "etcd_"

// correct values (on etcd peer)
curl -s --cert /etc/etcd/peer.crt --key /etc/etcd/peer.key --cacert /etc/etcd/ca.crt https://`hostname -i`:2379/metrics | grep -v "#" | grep "etcd_"


Steps to Reproduce:
1.
2.
3.

Actual results:
etcd_network_client_grpc_received_bytes_total 0
etcd_network_client_grpc_sent_bytes_total 0

Expected results:
etcd_network_client_grpc_received_bytes_total 5.929199954e+09
etcd_network_client_grpc_sent_bytes_total 8.0341137205e+10

Additional info:
Also, I see that some of the metrics are not shown if directly from etcd, or from /metrics url.