Created attachment 2015332 [details]
PrometheusRuleFailures alert details

Description of problem (please be detailed as possible and provide log
snippests):

Prometheus is continuously raising an alert "PrometheusRuleFailures" for "rule_group = /etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-storage-prometheus-ceph-rules-57d3a51f-f3ed-4c66-ab52-8f74a5c6b2b9.yaml;pool-quota.rules"

Alert details:
```
Name: PrometheusRuleFailures

Severity: Warning

Description: Prometheus Namespace openshift-monitoring/Pod prometheus-k8s-1 has failed to evaluate 20 rules in the last 5m.

Summary: Prometheus is failing rule evaluations.

Runbook: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusRuleFailures.md
```

These rules come from ocs-operator: https://github.com/red-hat-storage/ocs-operator/blob/73ef95adeacf878338ffa1452d3857b410ee1172/controllers/storagecluster/prometheus/localcephrules.yaml#L340-L365

Version of all relevant components (if applicable):
ODF 4.15.0
OCP 4.15.0-rc.4

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
Probably, this alert was not seen before.

Steps to Reproduce:
1. Deploy ODF
2. Create StorageCluster
3. Check Observe > Alerting > Alerts or Status card in Home > Overview


Actual results:
PrometheusRuleFailures alert is raised.

Expected results:
PrometheusRule evaluations should not fail.

Additional info:
Attached screenshots