Bug 1640027

Summary: heketi metrics displayed for disk parameters in prometheus graph and console are incorrect.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: RamaKasturi <knarra>
Component: heketiAssignee: Madhu Rajanna <mrajanna>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: ocs-3.11CC: akrishna, asriram, hchiramm, kramdoss, madam, mrajanna, msaini, rhs-bugs, rtalur, sankarshan, sselvan, storage-qa-internal, vinug
Target Milestone: ---Keywords: ZStream
Target Release: OCS 3.11.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: heketi-8.0.0-7.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Previously, device size metrics did not use the prescribed base unit of bytes and hence, incorrect values were interpreted. With this fix, the device size metric sizes are correctly published in bytes.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-07 03:41:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1641915, 1644154    
Attachments:
Description Flags
promethesus console
none
prometheus_graph
none
prometheus_UI none

Description RamaKasturi 2018-10-17 07:34:22 UTC 
Description of problem:
I see that the heketi_device_free, heketi_device_size, heketi_device_used values are displayed incorrectly  in Prometheus. I see that heketi gives the values in kb where as Prometheus excepts it to be in  bytes.

values from heketi curl :
=====================================
curl http://172.31.6.251:8080/metrics

# TYPE heketi_device_free gauge
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.134700032e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.041966592e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.041966592e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.14523904e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.14523904e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.136809472e+09
heketi_device_free{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.031427584e+09
# HELP heketi_device_size Total size of the device
# TYPE heketi_device_size gauge
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.14734848e+09
# HELP heketi_device_used Amount of space used on the device
# TYPE heketi_device_used gauge
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 1.2648448e+07
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 1.05381888e+08
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 1.05381888e+08
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sdd",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 2.10944e+06
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-15.lab.eng.blr.redhat.com"} 0
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-22.lab.eng.blr.redhat.com"} 2.10944e+06
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp46-39.lab.eng.blr.redhat.com"} 1.0539008e+07
heketi_device_used{cluster="842e0591bf9951a40b7e0e5a44aaf6eb",device="/dev/sde",hostname="dhcp47-114.lab.eng.blr.redhat.com"} 1.15920896e+08


Attaching the screenshots for prometheus graph and console.

Version-Release number of selected component (if applicable):
heketi-client-7.0.0-15.el7rhgs.x86_64
heketi-7.0.0-15.el7rhgs.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Install OCP + OCS with logging and metrics
2. Add a user by running the command oc adm policy add-cluster-role-to-user cluster-admin test --config=/etc/origin/master/admin.kubeconfig
3. In /etc/origin/master-config.yaml file change kind: DenyAllPasswordIdentityProvider to kind: AllowAllPasswordIdentityProvider
4. Run the command oc login and specify the user as test and password as test, make sure that login passes
5. Now run the playbook to deploy prometheus ansible-playbook -vvv -i basic_inventory /usr/share/ansible/openshift-ansible/playbooks/openshift-prometheus/config.yml
6. Verify that all prometheus pods are up by running the commands below
# oc project openshift-metrics
# oc get pods
7. Run the command to get HOST/PATH for prometheus 
# oc get routes, copy the HOST/PATH
# oc get pods -o wide (to check where Prometheus pod is running)
8. Add the IP of the node and the HOST/PATH to /etc/hosts file in your laptop as shown below.
10.70.46.15 prometheus-openshift-metrics.router.default.svc.cluster.local
9. Browse the url https://prometheus-openshift-metrics.router.default.svc.cluster.local/ and click Allow selected Permissions.
10. Now run the commands below
# oc annotate svc heketi-storage prometheus.io/scheme=http
# oc annotate svc heketi-storage prometheus.io/scrape=true
11. Run the command oc edit cm prometheus -o yaml -n openshift-metrics
12. add app-storage as shown below.
 - job_name: 'kubernetes-service-endpoints'

      	         regex: 'default|logging|metrics|kube-.+|openshift|openshift-.+|app-storage

13. Refresh the browser in URL and click on the combo box shown and select the metric heketi_device_size.

Actual results:
I see that values passed to prometheus are in kb's while prometheus expects it in bytes. I see that the value are 2.1 GB where as the actual disk size is 2000GB

Expected results:
Values in heketi and prometheus should match.

Additional info:
Comment 2 RamaKasturi 2018-10-17 07:36:17 UTC 
Created attachment 1494761 [details]
promethesus console
Comment 3 RamaKasturi 2018-10-17 07:36:55 UTC 
Created attachment 1494762 [details]
prometheus_graph
Comment 4 John Mulligan 2018-10-17 15:04:40 UTC 
The difference is only the units, correct?

So I agree that what heketi is doing right now is not in line with prometheus best practices: https://prometheus.io/docs/practices/naming/#base-units

That said, it is consistent behavior with the existing heketi-metrics-exporter project [1] that was the basis of the metrics that heketi now directly exposes.
It also appears that heketi doesn't even advertise these are storage units as there is no "_bytes" suffix on the metric name as recommended by the best practices link. So I'm not sure what is making the decision to display the value with units in the screenshot.

Overall, I feel that this should be made an enhancement so that we can add _bytes based metrics along side the old "raw" metrics which we'll keep for backwards compatibility with the old exporter.


[1] - https://github.com/CSCfi/heketi-metrics-exporter
Comment 13 Manisha Saini 2018-12-21 10:41:35 UTC 
Created attachment 1516080 [details]
prometheus_UI
Comment 14 Madhu Rajanna 2018-12-24 09:56:09 UTC 
There is some issue in converting KiB to bytes, Sent a fix for this issue https://github.com/heketi/heketi/pull/1475
Comment 17 Anjana KD 2019-01-07 07:53:04 UTC 
Updated Doc Text field. Kindly verify for technical accuracy.
Comment 21 Manisha Saini 2019-01-08 15:23:45 UTC 
Verified this BZ with

# rpm -qa | grep heketi
heketi-client-8.0.0-7.el7rhgs.x86_64
heketi-8.0.0-7.el7rhgs.x86_64


openshift_storage_glusterfs_heketi_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-volmanager-rhel7:3.11.1-7' 
openshift_storage_glusterfs_block_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-gluster-block-prov-rhel7:3.11.1-2'  
openshift_storage_glusterfs_image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ocs/rhgs-server-rhel7:3.11.1-8'


prometheus GUI now has the new param introduced which reflect the correct value in bytes

heketi_device_free_bytes
=========

Element	Value
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1597740417024
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2026146627584
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1858856812544
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1425052532736
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1680821190656
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1516769378304
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1621492760576
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	1648432775168

==========



heketi_device_used_bytes
==========
Element	Value
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	601144426496
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	172738215936
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	340028030976
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	773832310784
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	518063652864
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	682115465216
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	577392082944
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	550452068352

===========



heketi_device_size_bytes
===========

heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-137.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp46-141.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-123.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",endpoint="heketi",hostname="dhcp47-130.lab.eng.blr.redhat.com",instance="10.130.0.2:8080",job="heketi-storage",namespace="app-storage",pod="heketi-storage-1-4mxbl",service="heketi-storage"}	2198884843520
===========



Values from heketi curl
 
curl http://172.31.209.115:8080/metrics

=========

# TYPE heketi_device_free gauge
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.560293376e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.978658816e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.815289856e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.391652864e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.641426944e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.481220096e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.583489024e+09
heketi_device_free{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.609797632e+09
# HELP heketi_device_free_bytes Amount of Free space available on the device in bytes
# TYPE heketi_device_free_bytes gauge
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.597740417024e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.026146627584e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.858856812544e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.425052532736e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 1.680821190656e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.516769378304e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 1.621492760576e+12
heketi_device_free_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 1.648432775168e+12
# HELP heketi_device_size Total size of the device
# TYPE heketi_device_size gauge
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.14734848e+09
heketi_device_size{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.14734848e+09
# HELP heketi_device_size_bytes Total size of the device in bytes
# TYPE heketi_device_size_bytes gauge
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 2.19888484352e+12
heketi_device_size_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 2.19888484352e+12
# HELP heketi_device_used Amount of space used on the device
# TYPE heketi_device_used gauge
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 5.87055104e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.68689664e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 3.32058624e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 7.55695616e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 5.05921536e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 6.66128384e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 5.63859456e+08
heketi_device_used{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 5.37550848e+08
# HELP heketi_device_used_bytes Amount of space used on the device in bytes
# TYPE heketi_device_used_bytes gauge
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 6.01144426496e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 1.72738215936e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 3.40028030976e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sdd",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 7.73832310784e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-137.lab.eng.blr.redhat.com"} 5.18063652864e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp46-141.lab.eng.blr.redhat.com"} 6.82115465216e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-123.lab.eng.blr.redhat.com"} 5.77392082944e+11
heketi_device_used_bytes{cluster="d52e8dbd82f1ff4fd9a3970dfca97622",device="/dev/sde",hostname="dhcp47-130.lab.eng.blr.redhat.com"} 5.50452068352e+11

==================

Moving this BZ to verified state
Comment 23 errata-xmlrpc 2019-02-07 03:41:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0286