Bug 2084504 - can not silent platform alert from developer console
Summary: can not silent platform alert from developer console
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.11
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.12.0
Assignee: Gabriel Bernal
QA Contact: Junqi Zhao
Brian Burt
Depends On:
TreeView+ depends on / blocked
Reported: 2022-05-12 10:04 UTC by Junqi Zhao
Modified: 2023-01-17 19:49 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
* Before this update, if you created an alert silence from the *Developer* perspective in the {product-title} web console, external labels were included that did not match the alert. Therefore, the alert would not be silenced. With this update, external labels are now excluded when you create a silence in the *Developer* perspective so that newly created silences function as expected. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2084504[*BZ#2084504*])
Clone Of:
Last Closed: 2023-01-17 19:48:48 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift console pull 11895 0 None open Bug 2084504: can not silence platform alert from developer console 2022-08-02 04:31:42 UTC
Red Hat Product Errata RHSA-2022:7399 0 None None None 2023-01-17 19:49:04 UTC

Description Junqi Zhao 2022-05-12 10:04:56 UTC
Description of problem:
admin user, login developer console, click Observe, and namespace select openshift-monitoring, click Alerts tab, Watchdog alert, turn off the toggle button under Notifications column, choose a silent time and silent the alert.
check in admin console, "Observe -> Alerting -> Alerts" tab, the Watchdog alert is still firing, "Observe -> Alerting -> Silences" tab, the silence is created:


admin console, since the "Observe -> Alerting -> Alerts" tab uses /api/v1/rules API, and the labels are get from alerts.labels, there is not prometheus=openshift-monitoring/k8s label for Watchdog alert, so the alert is not silenced.
if we can not fix in monitoring side, we could let developer console to change. also not fix this issue is fine, usually developer console users don't have the permission to view other projects' alerts
same bug we filed before: bug 1984345
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules' | jq
        "file": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-eca87b8b-cb68-4b82-8446-522ee4cac906.yaml",
        "rules": [
            "state": "firing",
            "name": "Watchdog",
            "query": "vector(1)",
            "duration": 0,
            "labels": {
              "namespace": "openshift-monitoring",
              "prometheus": "openshift-monitoring/k8s",
              "severity": "none"
            "annotations": {
              "description": "This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n",
              "summary": "An alert that should always be firing to certify that Alertmanager is working properly."
            "alerts": [
                "labels": {
                  "alertname": "Watchdog",
                  "namespace": "openshift-monitoring",
                  "severity": "none"
                "annotations": {

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. developer console, silent platform alert
2. check in admin console to see if the alert is silenced

Actual results:
can not silent platform alert from developer console

Expected results:
be able to do so

Additional info:
can silent user project alert since the alert is wothout prometheus=openshift-monitoring/k8s label

Comment 1 Joao Marcal 2022-05-31 15:53:02 UTC
@juzhao I've tested this behaviour and it worked for me... I've tested it on a #forum-ui-clusters cluster 4.11.0-0.nightly-2022-05-25-193227 please let me know if it still doesn't work for you. If it does feel free to close the bug

Comment 2 Junqi Zhao 2022-06-06 03:44:22 UTC
checked with 4.11.0-0.nightly-2022-06-04-014713, still reproduced, this issue should happen with 4.11.0-0.nightly-2022-05-25-193227
note: there are administrator/developer consoles, see the attached picture, you could open administrator in one tab, and open developer console in another tab, the issue should be exist in every 4.11 cluster due to the Watchdog alert has following labels in developer console

but in administrator console, the labels are:

the difference is prometheus=openshift-monitoring/k8s, this caused the Watchdog alert can't be silenced

Comment 6 Junqi Zhao 2022-08-02 11:59:17 UTC
tested with PR, alert from developer console could be silenced, see the attached picture

Comment 12 Junqi Zhao 2022-08-18 03:58:28 UTC
checked with 4.12.0-0.nightly-2022-08-15-150248, the fix is in the payload and same behaviours with Comment 7 and Comment 8

Comment 15 errata-xmlrpc 2023-01-17 19:48:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.