Bug 1841883 - namespace-persistentvolumeclaim-usage query returns unexpected values
Summary: namespace-persistentvolumeclaim-usage query returns unexpected values
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metering Operator
Version: 4.7
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.7.0
Assignee: tflannag
QA Contact: Peter Ruan
URL:
Whiteboard:
: 1841882 (view as bug list)
Depends On:
Blocks: 1921390
TreeView+ depends on / blocked
 
Reported: 2020-05-29 17:43 UTC by Brett Tofel
Modified: 2021-02-24 15:02 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1822472
Environment:
Last Closed: 2021-02-24 15:01:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kube-reporting metering-operator pull 1471 0 None closed Bug 1841883: Fix the namespace-persistentvolumeclaim-usage ReportQuery 2021-02-09 06:24:24 UTC
Red Hat Product Errata RHSA-2020:5635 0 None None None 2021-02-24 15:02:11 UTC

Description Brett Tofel 2020-05-29 17:43:49 UTC
+++ This bug was initially created as a clone of Bug #1822472 +++

Description of problem:

 Reports using the query namespace-persistentvolumeclaim-usage show values that do not match the actual space used.


Version-Release number of selected component (if applicable):

  # oc version
  Client Version: 4.3.9
  Server Version: 4.3.9
  Kubernetes Version: v1.16.2


How reproducible:

 Always

Steps to Reproduce:

1.  Create the following two reports:

apiVersion: metering.openshift.io/v1
kind: Report
metadata:
  name: ns-pvc-usage
  namespace: openshift-metering
spec:
  query: namespace-persistentvolumeclaim-usage
  reportingEnd: "2020-04-08T19:30:00Z"
  reportingStart: "2020-04-08T18:40:00Z"


apiVersion: metering.openshift.io/v1
kind: Report
metadata:
  name: ns-pvc-usage-hourly
  namespace: openshift-metering
spec:
  query: namespace-persistentvolumeclaim-usage
  reportingStart: "2020-04-08T18:50:00Z"
  schedule:
    hourly: {}
    period: hourly

 
2. Create a new project, add a pod, and a PVC, and fill up some space in the PV.
   In this case I created a 3GB PVC, and wrote 512MB: 

  sh-4.2$ df /data 
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sdd         2819104 532884   2269836  20% /data


3. Wait for the reports to execute.


Actual results:

 The report that only runs for a defined period of time shows the following data: 

  period_start,period_end,namespace,persistentvolumeclaim_usage_bytes            
  2020-04-08 18:40:00 +0000 UTC,2020-04-08 19:30:00 +0000 UTC,test-metering,27283660800.000000

 Those 27283660800 bytes do not match with the 512MB actually written in the PVC, or with the block size, that would be  2886762496 bytes.
 The value shown is one order of magnitude larger than the block device.


 The report that runs hourly shows completely different values from the other report, even though they use the same reportquery:

 The first execution shows a completely different value from the other report, even though no more data was written in the volume,
 and the second and subsequent executions show a new value.  

 None of those three values represent the actual space usage on the PVC:

  2020-04-08 18:50:00 +0000 UTC,2020-04-08 19:00:00 +0000 UTC,test-metering,5456732160.000000
  2020-04-08 19:00:00 +0000 UTC,2020-04-08 20:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 20:00:00 +0000 UTC,2020-04-08 21:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 21:00:00 +0000 UTC,2020-04-08 22:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 22:00:00 +0000 UTC,2020-04-08 23:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-08 23:00:00 +0000 UTC,2020-04-09 00:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 00:00:00 +0000 UTC,2020-04-09 01:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 01:00:00 +0000 UTC,2020-04-09 02:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 02:00:00 +0000 UTC,2020-04-09 03:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 03:00:00 +0000 UTC,2020-04-09 04:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 04:00:00 +0000 UTC,2020-04-09 05:00:00 +0000 UTC,test-metering,32740392960.000000
  2020-04-09 05:00:00 +0000 UTC,2020-04-09 06:00:00 +0000 UTC,test-metering,32740392960.000000


Expected results:

 The reports should show accurate and consistent values of the space used. 


Additional info:

Comment 5 Paul Weil 2020-08-06 18:46:13 UTC
*** Bug 1841882 has been marked as a duplicate of this bug. ***

Comment 10 tflannag 2020-08-21 16:28:51 UTC
Adding the UpcomingSprint keyword now as we prioritized other bugs and work this last sprint. We plan on tackling this bug this next sprint.

Comment 12 Brett Tofel 2020-10-02 15:28:33 UTC
Marking UpcomingSprint as we worked more critical bugs and didn't get a chance to do the above analysis of the proposed change.

Comment 15 Peter Ruan 2021-02-09 06:25:28 UTC
verified with 4.7.0-0.nightly-2021-02-08-191932
PVC metrics looks reasonable.
$ display_report_using_exposed_route persistentvolumeclaim-usage-now
period_start			period_end			namespace		persistentvolumeclaim				persistentvolumeclaim_usage_bytes
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-logging	elasticsearch-elasticsearch-cdm-f50bqm1n-1	7993864192.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-logging	elasticsearch-elasticsearch-cdm-f50bqm1n-2	8061263872.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-logging	elasticsearch-elasticsearch-cdm-f50bqm1n-3	8042872832.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-metering	hive-metastore-db-data				20963328.000000
2021-01-01 00:00:00 +0000 UTC	2021-12-30 23:59:59 +0000 UTC	openshift-metering	hive-warehouse-data				27122466816.000000

Comment 19 errata-xmlrpc 2021-02-24 15:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 extras and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5635


Note You need to log in before you can comment on or make changes to this bug.