Description of problem (please be as detailed as possible and provide log
snippests):
In the DR dashboard, snapshot-related graphs are not visualizing the metrics. When the ACM MulticlusterObservability CR has a sync interval of 5 min, the DR dashboard is not getting any metrics, because by default DR UI is hardcoded 5 mins as the range value in the query(idelta(ceph_rbd_mirror_snapshot_snapshots[5m])).

When the metrics scraping value is 5 min in ACM observability, For a 5m range value ACM Thanos does not have any values present for the idelta computation.

As per the ACM observability team, It is recommended to keep the range value 3X or 4X of the scrap interval. 

Version of all relevant components (if applicable):
2.9.0-DOWNSTREAM-2023-10-08-08-16-57
quay.io/rhceph-dev/ocs-registry:4.14.0-151

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?
Keep the scraping interval(sync interval) on MulticlusterObservability CR to less than 5 minutes.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
100%

Can this issue reproduce from the UI?
100%

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Create MulticlusterObservability CR with a sync interval of 300 secs.
2. Apply the policy to enable replication
3. Snapshot realted graphs will be always empty.


Actual results:
Both Snapshot realted graphs are empty.

Expected results:

Additional info: