Bug 2010364

Summary:	OpenShift Alerting Rules Style-Guide Compliance
Product:	OpenShift Container Platform	Reporter:	Brad Ison <brad.ison>
Component:	Samples	Assignee:	Feny Mehta <fmehta>
Status:	CLOSED ERRATA	QA Contact:	Sushanta Das <susdas>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	4.10	CC:	aos-bugs, susdas
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:38:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brad Ison 2021-10-04 13:58:48 UTC

Hello,

The OpenShift Monitoring Team has published a set guidelines for
writing alerting rules in OpenShift, including a basic style guide.
You can find these here:

  https://github.com/openshift/enhancements/blob/master/enhancements/monitoring/alerting-consistency.md
  https://github.com/openshift/enhancements/blob/master/enhancements/monitoring/alerting-consistency.md#style-guide

A subset of these are now being enforced in OpenShift End-to-End
tests [1], with temporary exceptions for existing non-compliant rules.

This component was found to have the following issues:

* Alerts without summary and/or description annotations:

  - SamplesDegraded
  - SamplesImagestreamImportFailing
  - SamplesInvalidConfig
  - SamplesMissingSecret
  - SamplesMissingTBRCredential
  - SamplesRetriesMissingOnImagestreamImportFailing
  - SamplesTBRInaccessibleOnBoot

Alerts MUST include summary and description annotations.

Think of summary as the first line of a commit message, or an email
subject line. It should be brief but informative. The description is
the longer, more detailed explanation of the alert.

The enhancement document linked above has examples of alerts with
these annotations.


* Alerts found to not include a namespace label:

  - SamplesImagestreamImportFailing
  - SamplesRetriesMissingOnImagestreamImportFailing

Alerts SHOULD include a namespace label indicating the alert's source.

This requirement originally comes from our SRE team, as they use the
namespace label as the first means of routing alerts. Many alerts
already include a namespace label as a result of the PromQL
expressions used, others may require a static label.

Example of a change to PromQL to include a namespace label:

  https://github.com/openshift/cluster-monitoring-operator/commit/52d1f05#diff-9024dcef0fd244c0267c46858da24fbd1f45633515fafae0f98781b20805ff1dL22-R22

Example of adding a static namespace label:

  https://github.com/openshift/cluster-monitoring-operator/commit/52d1f05#diff-352702e71122d34a1be04c0588356cd8cb8a10df547f1c3c39fec18fa75b1593R304

If you have questions about how to best to modify your alerting rules
to include a namespace label, please reach out to the OpenShift
Monitoring Team in the #forum-monitoring channel on Slack, or on our
mailing list: team-monitoring

Thank you!

Repo: openshift/cluster-samples-operator

[1]: https://github.com/openshift/origin/commit/097e7a6

Comment 3 Sushanta Das 2022-05-04 09:27:51 UTC

Verified to make sure samples operator event firing still working with below steps.

1. launch a cluster via clusterbot with this PR
2. set samplesRegistry in the config to an invalid host so the imports fail
3. confirmed the alerts "SamplesImagestreamImportFailing" is visible in the prometheus console after 30mins

Comment 5 errata-xmlrpc 2022-08-10 10:38:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069