Bug 1976765 - AlertmanagerMembersInconsistent fires too quickly, causing serial-test noise
Summary: AlertmanagerMembersInconsistent fires too quickly, causing serial-test noise
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.8.z
Assignee: Jayapriya Pai
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1936919
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-28 07:58 UTC by Filip Petkovski
Modified: 2021-08-16 18:32 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1936919
Environment:
[Late] Alerts shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Suite:openshift/conformance/parallel]
Last Closed: 2021-08-16 18:32:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1244 0 None closed jsonnet: pull latest deps 2021-06-28 08:52:08 UTC
Github openshift cluster-monitoring-operator pull 1283 0 None open Bug 1976765:Update AlertmanagerMembersInconsistent rule 2021-07-14 10:31:57 UTC
Red Hat Product Errata RHBA-2021:3121 0 None None None 2021-08-16 18:32:25 UTC

Comment 1 W. Trevor King 2021-07-14 16:37:07 UTC
I'm updating the bug title (and moving the test-case name into the environment field for Sippy) to make it more clear what is getting fixed here.

Comment 5 Junqi Zhao 2021-08-10 05:46:45 UTC
checked with CI jobs, no firing AlertmanagerMembersInconsistent alert
https://search.ci.openshift.org/?search=AlertmanagerMembersInconsistent&maxAge=48h&context=1&type=bug%2Bjunit&name=periodic-ci-openshift-release-master-ci-4.8.*&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

    - alert: AlertmanagerMembersInconsistent
      annotations:
        description: Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only
          found {{ $value }} members of the {{$labels.job}} cluster.
        summary: A member of an Alertmanager cluster has not found all other cluster
          members.
      expr: |
        # Without max_over_time, failed scrapes could create false negatives, see
        # https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.
          max_over_time(alertmanager_cluster_members{job="alertmanager-main",namespace="openshift-monitoring"}[5m])
        < on (namespace,service) group_left
          count by (namespace,service) (max_over_time(alertmanager_cluster_members{job="alertmanager-main",namespace="openshift-monitoring"}[5m]))
      for: 15m
      labels:
        severity: critical

Comment 7 errata-xmlrpc 2021-08-16 18:32:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.5 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3121


Note You need to log in before you can comment on or make changes to this bug.