Bug 1899587

Description of problem:
On an OCS external mode cluster, where noobaa is configured to use pv-pool for the backingstore, the RGW usage metrics shown in the Capacity Breakdown card on the Object Service dashboard is incorrect. The used space for RGW shown in the dashboard is less than the used space for that pool on the RHCS cluster.

On increasing the IO on noobaa OBCs and keeping the IO on the RGW OBCs almost constant, it was observed that the used space of RGW kept decreasing. It appears to be an error in how these values are calculated.

# ceph df |grep rgw
    .rgw.root                      3     2.1 KiB           5     960 KiB         0       1.5 TiB 
    default.rgw.control            4         0 B           8         0 B         0       1.5 TiB 
    default.rgw.meta               5      15 KiB          63     9.4 MiB         0       1.5 TiB 
    default.rgw.log                6     3.4 KiB         206       6 MiB         0       1.5 TiB 
    default.rgw.buckets.index      8       976 B         165       976 B         0       1.5 TiB 
    default.rgw.buckets.data       9     4.1 GiB       4.16k      12 GiB      0.26       1.5 TiB 


$ oc describe backingstore noobaa-default-backing-store
Name:         noobaa-default-backing-store
Namespace:    openshift-storage
Labels:       app=noobaa
Annotations:  <none>
API Version:  noobaa.io/v1alpha1
Kind:         BackingStore
Metadata:
  Creation Timestamp:  2020-11-19T09:23:24Z
  Finalizers:
    noobaa.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  noobaa.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
        f:labels:
          .:
          f:app:
        f:ownerReferences:
      f:spec:
        .:
        f:pvPool:
          .:
          f:numVolumes:
          f:resources:
            .:
            f:requests:
              .:
              f:storage:
          f:secret:
        f:type:
      f:status:
        .:
        f:conditions:
        f:mode:
          .:
          f:modeCode:
          f:timeStamp:
        f:phase:
    Manager:    noobaa-operator
    Operation:  Update
    Time:       2020-11-19T12:51:10Z
  Owner References:
    API Version:           noobaa.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  NooBaa
    Name:                  noobaa
    UID:                   e832c40c-4c61-45a2-800d-e7d93261c76c
  Resource Version:        161472
  Self Link:               /apis/noobaa.io/v1alpha1/namespaces/openshift-storage/backingstores/noobaa-default-backing-store
  UID:                     3bdcfbad-74e3-4844-8640-ae5d59543685
Spec:
  Pv Pool:
    Num Volumes:  1
    Resources:
      Requests:
        Storage:  50Gi
    Secret:
  Type:  pv-pool


Screenshots of the metrics and the Object Service dashboards can be found here: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/rgw_usage/


Version-Release number of selected component (if applicable):
OCP: 4.6.0-0.nightly-2020-11-18-154058
OCS: ocs-operator.v4.6.0-160.ci

How reproducible: 2/2


Steps to Reproduce:
1. Deploy an external mode OCS cluster without RGW, this will configure noobaa to use pv-pool for its backingstore
2. After the deployment, update the rook-ceph-external-cluster-details secret with RGW details, to allow OCS to use the RGW on the RHCS cluster (https://bugzilla.redhat.com/show_bug.cgi?id=1865825)
3. Create Noobaa and RGW OBCs and run IO on them
4. Compare the used space/usage of RGW on the Object Service Dashboard and RHCS cluster

Actual results:
The usage shown on the dashboard is less than the usage on the RHCS cluster for that rgw pool

Expected results:
The used space should be equal