Description of problem: admin user, login developer console, click Observe, and namespace select openshift-monitoring, click Alerts tab, Watchdog alert, turn off the toggle button under Notifications column, choose a silent time and silent the alert. check in admin console, "Observe -> Alerting -> Alerts" tab, the Watchdog alert is still firing, "Observe -> Alerting -> Silences" tab, the silence is created: Watchdog namespace=openshift-monitoring alertname=Watchdog prometheus=openshift-monitoring/k8s severity=none admin console, since the "Observe -> Alerting -> Alerts" tab uses /api/v1/rules API, and the labels are get from alerts.labels, there is not prometheus=openshift-monitoring/k8s label for Watchdog alert, so the alert is not silenced. if we can not fix in monitoring side, we could let developer console to change. also not fix this issue is fine, usually developer console users don't have the permission to view other projects' alerts same bug we filed before: bug 1984345 # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules' | jq "file": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-eca87b8b-cb68-4b82-8446-522ee4cac906.yaml", "rules": [ { "state": "firing", "name": "Watchdog", "query": "vector(1)", "duration": 0, "labels": { "namespace": "openshift-monitoring", "prometheus": "openshift-monitoring/k8s", "severity": "none" }, "annotations": { "description": "This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n", "summary": "An alert that should always be firing to certify that Alertmanager is working properly." }, "alerts": [ { "labels": { "alertname": "Watchdog", "namespace": "openshift-monitoring", "severity": "none" }, "annotations": { Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-05-11-054135 How reproducible: always Steps to Reproduce: 1. developer console, silent platform alert 2. check in admin console to see if the alert is silenced 3. Actual results: can not silent platform alert from developer console Expected results: be able to do so Additional info: can silent user project alert since the alert is wothout prometheus=openshift-monitoring/k8s label
@juzhao I've tested this behaviour and it worked for me... I've tested it on a #forum-ui-clusters cluster 4.11.0-0.nightly-2022-05-25-193227 please let me know if it still doesn't work for you. If it does feel free to close the bug
checked with 4.11.0-0.nightly-2022-06-04-014713, still reproduced, this issue should happen with 4.11.0-0.nightly-2022-05-25-193227 note: there are administrator/developer consoles, see the attached picture, you could open administrator in one tab, and open developer console in another tab, the issue should be exist in every 4.11 cluster due to the Watchdog alert has following labels in developer console namespace=openshift-monitoring alertname=Watchdog prometheus=openshift-monitoring/k8s severity=none but in administrator console, the labels are: namespace=openshift-monitoring alertname=Watchdog severity=none the difference is prometheus=openshift-monitoring/k8s, this caused the Watchdog alert can't be silenced
tested with PR, alert from developer console could be silenced, see the attached picture
checked with 4.12.0-0.nightly-2022-08-15-150248, the fix is in the payload and same behaviours with Comment 7 and Comment 8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399