Bug 1845444 - KubeApiLatency alert firing even though not all conditions are matched
Summary: KubeApiLatency alert firing even though not all conditions are matched
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.4.z
Assignee: Rick Rackow
QA Contact: Junqi Zhao
Whiteboard: wip
Depends On: 1845445
Blocks: 1845443
TreeView+ depends on / blocked
Reported: 2020-06-09 09:15 UTC by Rick Rackow
Modified: 2020-07-28 12:37 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1845443
: 1845445 (view as bug list)
Last Closed: 2020-07-28 12:37:03 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 809 0 None closed Bug 1845444: KubeApiLatency alert firing even though not all conditions are matched 2021-02-09 03:15:57 UTC
Red Hat Product Errata RHBA-2020:3075 0 None None None 2020-07-28 12:37:28 UTC

Description Rick Rackow 2020-06-09 09:15:53 UTC
+++ This bug was initially created as a clone of Bug #1845443 +++

Description of problem:

KubeApiLatencyHigh warning should only be firing if all conditions are met AND the latency is over 1s.
However we have seen this fire with

The API server has an abnormal latency of 0.05685404799999992 seconds for PUT namespace

Version-Release number of selected component (if applicable):
OpenShift Dedicated 4.3.18

How reproducible:

Steps to Reproduce:
1. Execute alerting rule in Prometheus to graph
2. Scroll out until you find an occurence

Actual results:

Expected results:

Additional info:
This can as well be fixed by adjusting the message to be something more meaningful

Comment 1 Rick Rackow 2020-07-09 11:58:58 UTC
Blocked because we're waiting for https://bugzilla.redhat.com/show_bug.cgi?id=1845445

Comment 6 Junqi Zhao 2020-07-20 03:15:19 UTC
tested with 4.4.0-0.nightly-2020-07-18-033102,KubeAPILatencyHigh warning alert details see below, and there is not such alert in the cluster
  - alert: KubeAPILatencyHigh
      message: The API server has an abnormal latency of {{ $value }} seconds for
        {{ $labels.verb }} {{ $labels.resource }}.
    expr: |
      and on (verb,resource)
        on (verb) group_left()
          avg by (verb) (cluster:apiserver_request_duration_seconds:mean5m{job="apiserver"} >= 0)
          2*stddev by (verb) (cluster:apiserver_request_duration_seconds:mean5m{job="apiserver"} >= 0)
      ) > on (verb) group_left()
      1.2 * avg by (verb) (cluster:apiserver_request_duration_seconds:mean5m{job="apiserver"} >= 0)
    for: 5m
      severity: warning

Comment 8 errata-xmlrpc 2020-07-28 12:37:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.