Created attachment 1556195 [details] prometheus-graph-ui Description of problem: I found when running our teams aws-e2e tests against 4.x, that in some cases prometheus contains no kubelet_volume_stats_capacity_bytes or kubelet_volume_stats_usage_bytes metrics. When I looked at the prometheus.tar.gz containing the Prometheus database, and ran Prometheus locally with the metrics in the e2e run, I found these metrics were not available in Prometheus, but the kubelet is being successfully scraped. Because prometheus is correctly scraping the kubelet, it would seem that kubelet is not exporting these volume metrics correctly. Version-Release number of selected component (if applicable): Whatever aws-e2e uses. How reproducible: Very rare. Currently occurred for us once in https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_release/3515/rehearse-3515-pull-ci-operator-framework-operator-metering-master-metering-e2e-aws/2 but it's likely this is occurring more regularly but is not being tested for. Steps to view issue: 1. Download https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_release/3515/rehearse-3515-pull-ci-operator-framework-operator-metering-master-metering-e2e-aws/2/artifacts/metering-e2e-aws/metrics/prometheus.tar.gz to obtain the metrics for the above job. 2. Follow the instructions for using these metrics by searching the aos-devel mailing list for running Prometheus locally: Use the following search: "[aos-devel] Full prometheus dump is now captured during e2e runs". 3. Use the graph UI to look between 2019-04-18 16:00 and 2019-04-18 18:00 (see attached screenshot) for the kubelet_volume_stats_usage_bytes and up{job="kubelet"} metrics. You should see kubelet target is up and being scraped, but there is no metrics for kubelet_volume_stats_usage_bytes. Actual results: kubelet_volume_stats_usage_bytes and kubelet_volume_stats_capacity_bytes metrics do not exist. Expected results: kubelet_volume_stats_usage_bytes and kubelet_volume_stats_capacity_bytes metrics should exist. Additional info:
FWIW - we do have e2es that test if kubelet is emitting volume stat metrics - https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/volume_metrics.go#L191