Bug 1680517

Created attachment 1538365 [details]
"No data points" for "Disk IO Utilisation" and "Disk IO Saturation"

Description of problem:
Tested with ose-cluster-monitoring-operator/images/v3.11.88-1 on openstack
In grafana UI "K8s / USE Method / Node "page,  "No data points" for "Disk IO Utilisation" and "Disk IO Saturation"

Checked "Disk IO Utilisation" used node:node_disk_utilisation:avg_irate rules and "Disk IO Saturation" used node:node_disk_saturation:avg_irate rules, these rules are OK for AWS/GCE, but not for Openstack, since device name is device="dm-0" and device="vda/c/d"(vd.+)
********************************************************************************
Disk IO Utilisation
https://grafana-openshift-monitoring.apps.0225-4gx.qe.rhcloud.com/api/datasources/proxy/1/api/v1/query_range?query=node:node_disk_utilisation:avg_irate{node="****"}&start=1551069510&end=1551073140&step=30
********************************************************************************
Disk IO Saturation
https://grafana-openshift-monitoring.apps.0225-4gx.qe.rhcloud.com/api/datasources/proxy/1/api/v1/query_range?query=node:node_disk_saturation:avg_irate{node="****"}&start=1551069510&end=1551073140&step=30
********************************************************************************
record: node:node_disk_utilisation:avg_irate
expr: avg
  by(node) (irate(node_disk_io_time_ms{device=~"(sd|xvd|nvme).+",job="node-exporter"}[1m])
  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)

record: node:node_disk_saturation:avg_irate
expr: avg
  by(node) (irate(node_disk_io_time_weighted{device=~"(sd|xvd|nvme).+",job="node-exporter"}[1m])
  / 1000 * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:)
********************************************************************************
node_disk_io_time_ms{job="node-exporter"} in prometheus UI

Element	Value
node_disk_io_time_ms{device="dm-0",endpoint="https",instance="10.0.76.148:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-6jvjv",service="node-exporter"}	954685.0000000001
node_disk_io_time_ms{device="dm-0",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	474357
node_disk_io_time_ms{device="dm-0",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	500224
node_disk_io_time_ms{device="vda",endpoint="https",instance="10.0.76.148:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-6jvjv",service="node-exporter"}	126423
node_disk_io_time_ms{device="vda",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	52844
node_disk_io_time_ms{device="vda",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	59845
node_disk_io_time_ms{device="vdb",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	2988
node_disk_io_time_ms{device="vdb",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	5206
node_disk_io_time_ms{device="vdc",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	257
node_disk_io_time_ms{device="vdc",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	173.00000000000003
node_disk_io_time_ms{device="vdd",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	191
**************************************************************
node_disk_io_time_weighted{job="node-exporter"} in prometheus UI
Element	Value
node_disk_io_time_weighted{device="dm-0",endpoint="https",instance="10.0.76.148:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-6jvjv",service="node-exporter"}	3873.425
node_disk_io_time_weighted{device="dm-0",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	2762.52
node_disk_io_time_weighted{device="dm-0",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	3658.87
node_disk_io_time_weighted{device="vda",endpoint="https",instance="10.0.76.148:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-6jvjv",service="node-exporter"}	1719.608
node_disk_io_time_weighted{device="vda",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	1184.056
node_disk_io_time_weighted{device="vda",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	1273.034
node_disk_io_time_weighted{device="vdb",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	3.911
node_disk_io_time_weighted{device="vdb",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	7.839
node_disk_io_time_weighted{device="vdc",endpoint="https",instance="10.0.76.15:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-4jf4k",service="node-exporter"}	0.293
node_disk_io_time_weighted{device="vdc",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	0.193
node_disk_io_time_weighted{device="vdd",endpoint="https",instance="10.0.76.218:9100",job="node-exporter",namespace="openshift-monitoring",pod="node-exporter-mfg5l",service="node-exporter"}	0.247
Version-Release number of selected component (if applicable):
ose-cluster-monitoring-operator/images/v3.11.88-1
**************************************************************
How reproducible:
IAAS related

Steps to Reproduce:
1. Check grafana UI "K8s / USE Method / Node "page
2.
3.

Actual results:
"No data points" for "Disk IO Utilisation" and "Disk IO Saturation"

Expected results:
Should show metrics diagram

Additional info:
Should backport https://github.com/openshift/cluster-monitoring-operator/blob/e536f4fea32bf02944f583d57eb29951e8c70481/assets/prometheus-k8s/rules.yaml
to 3.11