Bug 1841883

Summary: namespace-persistentvolumeclaim-usage query returns unexpected values
Product: OpenShift Container Platform Reporter: Brett Tofel <btofel>
Component: Metering OperatorAssignee: tflannag
Status: CLOSED ERRATA QA Contact: Peter Ruan <pruan>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.7CC: asolanas, emoss, erich, pamoedom, pruan, pweil, sd-operator-metering, tflannag
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1822472 Environment:
Last Closed: 2021-02-24 15:01:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1921390    

Description Brett Tofel 2020-05-29 17:43:49 UTC
+++ This bug was initially created as a clone of Bug #1822472 +++

Description of problem:

 Reports using the query namespace-persistentvolumeclaim-usage show values that do not match the actual space used.


Version-Release number of selected component (if applicable):

  # oc version
  Client Version: 4.3.9
  Server Version: 4.3.9
  Kubernetes Version: v1.16.2


How reproducible:

 Always

Steps to Reproduce:

1.  Create the following two reports:

apiVersion: metering.openshift.io/v1
kind: Report
metadata:
  name: ns-pvc-usage
  namespace: openshift-metering
spec:
  query: namespace-persistentvolumeclaim-usage
  reportingEnd: "2020-04-08T19:30:00Z"
  reportingStart: "2020-04-08T18:40:00Z"


apiVersion: metering.openshift.io/v1
kind: Report
metadata:
  name: ns-pvc-usage-hourly
  namespace: openshift-metering
spec:
  query: namespace-persistentvolumeclaim-usage
  reportingStart: "2020-04-08T18:50:00Z"
  schedule:
    hourly: {}
    period: hourly

 
2. Create a new project, add a pod, and a PVC, and fill up some space in the PV.
   In this case I created a 3GB PVC, and wrote 512MB: 

  sh-4.2$ df /data 
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sdd         2819104 532884   2269836  20% /data


3. Wait for the reports to execute.


Actual results:

 The report that only runs for a defined period of time shows the following data: 

  period_start,period_end,namespace,persistentvolumeclaim_usage_bytes            
  2020-04-08 18:40:00 +0000 UTC,2020-04-08 19:30:00 +0000 UTC,test-metering,27283660800.000000

 Those 27283660800 bytes do not match with the 512MB actually written in the PVC, or with the block size, that would be  2886762496 bytes.
 The value shown is one order of magnitude larger than the block device.


 The report that runs hourly shows completely different values from the other report, even though they use the same reportquery:

 The first execution shows a completely different value from the other report, even though no more data was written in the volume,
 and the second and subsequent executions show a new value.  

 None of those three values represent the actual space usage on the PVC:

  2020-04-08 18:50:00 +0000 UTC,2020-04-08 19:00:00 +0000 UTC,test-metering,5456732160.000000
  2020-04-08 19:00:00 +0000 UTC,2020-04-08 20:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 20:00:00 +0000 UTC,2020-04-08 21:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 21:00:00 +0000 UTC,2020-04-08 22:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 22:00:00 +0000 UTC,2020-04-08 23:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 23:00:00 +0000 UTC,2020-04-09 00:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 00:00:00 +0000 UTC,2020-04-09 01:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 01:00:00 +0000 UTC,2020-04-09 02:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 02:00:00 +0000 UTC,2020-04-09 03:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 03:00:00 +0000 UTC,2020-04-09 04:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 04:00:00 +0000 UTC,2020-04-09 05:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 05:00:00 +0000 UTC,2020-04-09 06:00:00 +0000 UTC,test-metering,32740392960.000000


Expected results:

 The reports should show accurate and consistent values of the space used. 


Additional info:

Comment 5 Paul Weil 2020-08-06 18:46:13 UTC
*** Bug 1841882 has been marked as a duplicate of this bug. ***

Comment 10 tflannag 2020-08-21 16:28:51 UTC
Adding the UpcomingSprint keyword now as we prioritized other bugs and work this last sprint. We plan on tackling this bug this next sprint.

Comment 12 Brett Tofel 2020-10-02 15:28:33 UTC
Marking UpcomingSprint as we worked more critical bugs and didn't get a chance to do the above analysis of the proposed change.

Comment 15 Peter Ruan 2021-02-09 06:25:28 UTC
verified with 4.7.0-0.nightly-2021-02-08-191932
PVC metrics looks reasonable.
$ display_report_using_exposed_route persistentvolumeclaim-usage-now
period_start			period_end			namespace		persistentvolumeclaim				persistentvolumeclaim_usage_bytes
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-logging	elasticsearch-elasticsearch-cdm-f50bqm1n-1	7993864192.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-logging	elasticsearch-elasticsearch-cdm-f50bqm1n-2	8061263872.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-logging	elasticsearch-elasticsearch-cdm-f50bqm1n-3	8042872832.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-metering	hive-metastore-db-data				20963328.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-metering	hive-warehouse-data				27122466816.000000

Comment 19 errata-xmlrpc 2021-02-24 15:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 extras and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5635