Description of problem: I see that the heketi_device_free, heketi_device_size, heketi_device_used values are displayed incorrectly in Prometheus. I see that heketi gives the values in kb where as Prometheus excepts it to be in bytes. values from heketi curl : ===================================== curl http://172.31.6.251:8080/metrics # TYPE heketi_device_free gauge heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.134700032e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.041966592e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.041966592e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.14523904e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.14523904e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.136809472e+09 heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.031427584e+09 # HELP heketi_device_size Total size of the device # TYPE heketi_device_size gauge heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.14734848e+09 # HELP heketi_device_used Amount of space used on the device # TYPE heketi_device_used gauge heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 1.2648448e+07 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 1.05381888e+08 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 1.05381888e+08 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.10944e+06 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 0 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.10944e+06 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 1.0539008e+07 heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 1.15920896e+08 Attaching the screenshots for prometheus graph and console. Version-Release number of selected component (if applicable): heketi-client-7.0.0-15.el7rhgs.x86_64 heketi-7.0.0-15.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1. Install OCP + OCS with logging and metrics 2. Add a user by running the command oc adm policy add-cluster-role-to-user cluster-admin test --config=/etc/origin/master/admin.kubeconfig 3. In /etc/origin/master-config.yaml file change kind: DenyAllPasswordIdentityProvider to kind: AllowAllPasswordIdentityProvider 4. Run the command oc login and specify the user as test and password as test, make sure that login passes 5. Now run the playbook to deploy prometheus ansible-playbook -vvv -i basic_inventory /usr/share/ansible/openshift-ansible/playbooks/openshift-prometheus/config.yml 6. Verify that all prometheus pods are up by running the commands below # oc project openshift-metrics # oc get pods 7. Run the command to get HOST/PATH for prometheus # oc get routes, copy the HOST/PATH # oc get pods -o wide (to check where Prometheus pod is running) 8. Add the IP of the node and the HOST/PATH to /etc/hosts file in your laptop as shown below. 10.70.46.15 prometheus-openshift-metrics.router.default.svc.cluster.local 9. Browse the url https://prometheus-openshift-metrics.router.default.svc.cluster.local/ and click Allow selected Permissions. 10. Now run the commands below # oc annotate svc heketi-storage prometheus.io/scheme=http # oc annotate svc heketi-storage prometheus.io/scrape=true 11. Run the command oc edit cm prometheus -o yaml -n openshift-metrics 12. add app-storage as shown below. - job_name: 'kubernetes-service-endpoints' regex: 'default|logging|metrics|kube-.+|openshift|openshift-.+|app-storage 13. Refresh the browser in URL and click on the combo box shown and select the metric heketi_device_size. Actual results: I see that values passed to prometheus are in kb's while prometheus expects it in bytes. I see that the value are 2.1 GB where as the actual disk size is 2000GB Expected results: Values in heketi and prometheus should match. Additional info:
Created attachment 1494761 [details] promethesus console
Created attachment 1494762 [details] prometheus_graph
The difference is only the units, correct? So I agree that what heketi is doing right now is not in line with prometheus best practices: https://prometheus.io/docs/practices/naming/#base-units That said, it is consistent behavior with the existing heketi-metrics-exporter project [1] that was the basis of the metrics that heketi now directly exposes. It also appears that heketi doesn't even advertise these are storage units as there is no "_bytes" suffix on the metric name as recommended by the best practices link. So I'm not sure what is making the decision to display the value with units in the screenshot. Overall, I feel that this should be made an enhancement so that we can add _bytes based metrics along side the old "raw" metrics which we'll keep for backwards compatibility with the old exporter. [1] - https://github.com/CSCfi/heketi-metrics-exporter
Created attachment 1516080 [details] prometheus_UI
There is some issue in converting KiB to bytes, Sent a fix for this issue https://github.com/heketi/heketi/pull/1475
Updated Doc Text field. Kindly verify for technical accuracy.
Verified this BZ with # rpm -qa | grep heketi heketi-client-8.0.0-7.el7rhgs.x86_64 heketi-8.0.0-7.el7rhgs.x86_64 openshift_storage_glusterfs_heketi_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-volmanager-rhel7:3.11.1-7' openshift_storage_glusterfs_block_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-gluster-block-prov-rhel7:3.11.1-2' openshift_storage_glusterfs_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-server-rhel7:3.11.1-8' prometheus GUI now has the new param introduced which reflect the correct value in bytes heketi_device_free_bytes ========= Element Value heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1597740417024 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2026146627584 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1858856812544 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1425052532736 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1680821190656 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1516769378304 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1621492760576 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 1648432775168 ========== heketi_device_used_bytes ========== Element Value heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 601144426496 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 172738215936 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 340028030976 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 773832310784 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 518063652864 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 682115465216 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 577392082944 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 550452068352 =========== heketi_device_size_bytes =========== heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"} 2198884843520 =========== Values from heketi curl curl http://172.31.209.115:8080/metrics ========= # TYPE heketi_device_free gauge heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.560293376e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.978658816e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.815289856e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.391652864e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.641426944e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.481220096e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.583489024e+09 heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.609797632e+09 # HELP heketi_device_free_bytes Amount of Free space available on the device in bytes # TYPE heketi_device_free_bytes gauge heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.597740417024e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.026146627584e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.858856812544e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.425052532736e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.680821190656e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.516769378304e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.621492760576e+12 heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.648432775168e+12 # HELP heketi_device_size Total size of the device # TYPE heketi_device_size gauge heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.14734848e+09 heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.14734848e+09 # HELP heketi_device_size_bytes Total size of the device in bytes # TYPE heketi_device_size_bytes gauge heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.19888484352e+12 heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.19888484352e+12 # HELP heketi_device_used Amount of space used on the device # TYPE heketi_device_used gauge heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 5.87055104e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.68689664e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 3.32058624e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 7.55695616e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 5.05921536e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 6.66128384e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 5.63859456e+08 heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 5.37550848e+08 # HELP heketi_device_used_bytes Amount of space used on the device in bytes # TYPE heketi_device_used_bytes gauge heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 6.01144426496e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.72738215936e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 3.40028030976e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 7.73832310784e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 5.18063652864e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 6.82115465216e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 5.77392082944e+11 heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 5.50452068352e+11 ================== Moving this BZ to verified state
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0286