Bug 2010371 - OpenShift Alerting Rules Style-Guide Compliance
Summary: OpenShift Alerting Rules Style-Guide Compliance
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: MCO Team
QA Contact: Rio Liu
Depends On:
TreeView+ depends on / blocked
Reported: 2021-10-04 14:03 UTC by Brad Ison
Modified: 2022-03-15 09:28 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-12-01 19:20:53 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker MCO-1 0 None None None 2021-12-01 19:20:53 UTC

Description Brad Ison 2021-10-04 14:03:24 UTC

The OpenShift Monitoring Team has published a set guidelines for
writing alerting rules in OpenShift, including a basic style guide.
You can find these here:


A subset of these are now being enforced in OpenShift End-to-End
tests [1], with temporary exceptions for existing non-compliant rules.

This component was found to have the following issues:

* Alerts without summary and/or description annotations:

  - KubeletHealthState
  - MCDDrainError
  - MCDPivotError
  - MCDRebootError
  - MasterNodesHighMemoryUsage
  - SystemMemoryExceedsReservation

Alerts MUST include summary and description annotations.

Think of summary as the first line of a commit message, or an email
subject line. It should be brief but informative. The description is
the longer, more detailed explanation of the alert.

The enhancement document linked above has examples of alerts with
these annotations.

* Alerts found to not include a namespace label:

  - SystemMemoryExceedsReservation

Alerts SHOULD include a namespace label indicating the alert's source.

This requirement originally comes from our SRE team, as they use the
namespace label as the first means of routing alerts. Many alerts
already include a namespace label as a result of the PromQL
expressions used, others may require a static label.

Example of a change to PromQL to include a namespace label:


Example of adding a static namespace label:


If you have questions about how to best to modify your alerting rules
to include a namespace label, please reach out to the OpenShift
Monitoring Team in the #forum-monitoring channel on Slack, or on our
mailing list: team-monitoring

Thank you!

Repo: openshift/machine-config-operator

[1]: https://github.com/openshift/origin/commit/097e7a6

Comment 1 John Kyros 2021-12-01 19:20:53 UTC
Thanks Brad! 

We are making sure any new alerts we add comply with these guidelines, and we plan to address these existing ones as part of our upcoming "Enhance metrics and alerting" epic [1]. I have linked that epic to this bug so we don't lose it. 

[1] https://issues.redhat.com/browse/MCO-1

Note You need to log in before you can comment on or make changes to this bug.