Bug 2084504

Summary: can not silent platform alert from developer console
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Gabriel Bernal <gbernal>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact: Brian Burt <bburt>
Priority: low    
Version: 4.11CC: anpicker, bburt, gbernal, sthaha
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
* Before this update, if you created an alert silence from the *Developer* perspective in the {product-title} web console, external labels were included that did not match the alert. Therefore, the alert would not be silenced. With this update, external labels are now excluded when you create a silence in the *Developer* perspective so that newly created silences function as expected. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2084504[*BZ#2084504*])
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-17 19:48:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2022-05-12 10:04:56 UTC 
Description of problem:
admin user, login developer console, click Observe, and namespace select openshift-monitoring, click Alerts tab, Watchdog alert, turn off the toggle button under Notifications column, choose a silent time and silent the alert.
check in admin console, "Observe -> Alerting -> Alerts" tab, the Watchdog alert is still firing, "Observe -> Alerting -> Silences" tab, the silence is created:

Watchdog
namespace=openshift-monitoring
alertname=Watchdog
prometheus=openshift-monitoring/k8s
severity=none

admin console, since the "Observe -> Alerting -> Alerts" tab uses /api/v1/rules API, and the labels are get from alerts.labels, there is not prometheus=openshift-monitoring/k8s label for Watchdog alert, so the alert is not silenced.
if we can not fix in monitoring side, we could let developer console to change. also not fix this issue is fine, usually developer console users don't have the permission to view other projects' alerts
same bug we filed before: bug 1984345
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules' | jq
        "file": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-eca87b8b-cb68-4b82-8446-522ee4cac906.yaml",
        "rules": [
          {
            "state": "firing",
            "name": "Watchdog",
            "query": "vector(1)",
            "duration": 0,
            "labels": {
              "namespace": "openshift-monitoring",
              "prometheus": "openshift-monitoring/k8s",
              "severity": "none"
            },
            "annotations": {
              "description": "This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n",
              "summary": "An alert that should always be firing to certify that Alertmanager is working properly."
            },
            "alerts": [
              {
                "labels": {
                  "alertname": "Watchdog",
                  "namespace": "openshift-monitoring",
                  "severity": "none"
                },
                "annotations": {

Version-Release number of selected component (if applicable):
4.11.0-0.nightly-2022-05-11-054135

How reproducible:
always

Steps to Reproduce:
1. developer console, silent platform alert
2. check in admin console to see if the alert is silenced
3.

Actual results:
can not silent platform alert from developer console

Expected results:
be able to do so

Additional info:
can silent user project alert since the alert is wothout prometheus=openshift-monitoring/k8s label
Comment 1 Joao Marcal 2022-05-31 15:53:02 UTC 
@juzhao I've tested this behaviour and it worked for me... I've tested it on a #forum-ui-clusters cluster 4.11.0-0.nightly-2022-05-25-193227 please let me know if it still doesn't work for you. If it does feel free to close the bug
Comment 2 Junqi Zhao 2022-06-06 03:44:22 UTC 
checked with 4.11.0-0.nightly-2022-06-04-014713, still reproduced, this issue should happen with 4.11.0-0.nightly-2022-05-25-193227
note: there are administrator/developer consoles, see the attached picture, you could open administrator in one tab, and open developer console in another tab, the issue should be exist in every 4.11 cluster due to the Watchdog alert has following labels in developer console
namespace=openshift-monitoring
alertname=Watchdog
prometheus=openshift-monitoring/k8s
severity=none

but in administrator console, the labels are:
namespace=openshift-monitoring
alertname=Watchdog
severity=none

the difference is prometheus=openshift-monitoring/k8s, this caused the Watchdog alert can't be silenced
Comment 6 Junqi Zhao 2022-08-02 11:59:17 UTC 
tested with PR, alert from developer console could be silenced, see the attached picture
Comment 12 Junqi Zhao 2022-08-18 03:58:28 UTC 
checked with 4.12.0-0.nightly-2022-08-15-150248, the fix is in the payload and same behaviours with Comment 7 and Comment 8
Comment 15 errata-xmlrpc 2023-01-17 19:48:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399