Bug 2172588

Description of problem:

We are seeing the following errors in the prometheus log:

ts=2023-02-22T15:18:22.189Z caller=manager.go:715 level=warn component="rule manager" file=/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml group=./openstack.rules name=job:disk:time:read:rate_5m index=18 msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=5

ts=2023-02-22T15:18:22.199Z caller=manager.go:715 level=warn component="rule manager" file=/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml group=./openstack.rules name=job:disk:time:write:rate_5m index=23 msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=37


Version-Release number of selected component (if applicable):
stable-1.5

How reproducible:
Always

Steps to Reproduce:
Deploy alerts.yaml from the repo and check prometheus logs or "Rules" screen in the UI

Actual results:
ERR	
duplicate sample for timestamp

Expected results:
OK


Additional info:
The cause is because of a series of typos in the alert rules leading to duplicate recording rules:

record: 'job:disk:time:read:rate_5m'
* Correct: https://github.com/infrawatch/service-telemetry-operator/blob/37dceed7e55856820c10fb812da0ed9cd6551a3b/deploy/alerts/alerts.yaml#L45
* Should be 'job:disk:ops:read:rate_5m' https://github.com/infrawatch/service-telemetry-operator/blob/37dceed7e55856820c10fb812da0ed9cd6551a3b/deploy/alerts/alerts.yaml#L91

record: 'job:disk:time:write:rate_5m'
* Correct: https://github.com/infrawatch/service-telemetry-operator/blob/37dceed7e55856820c10fb812da0ed9cd6551a3b/deploy/alerts/alerts.yaml#L68
* Should be 'job:disk:ops:write:rate_5m' https://github.com/infrawatch/service-telemetry-operator/blob/37dceed7e55856820c10fb812da0ed9cd6551a3b/deploy/alerts/alerts.yaml#L114

Associated lines also need to be adjusted.

This bug likely causes the "disk ops" alarms to be non-functional; I think intstead they would function as duplicates of the "IO" alarms.