Bug 1784144 - No etcd alert fires when high number of leader changes occur
Summary: No etcd alert fires when high number of leader changes occur
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Sam Batschelet
QA Contact: ge liu
URL:
Whiteboard:
Depends On:
Blocks: 1784166
TreeView+ depends on / blocked
 
Reported: 2019-12-16 19:11 UTC by Clayton Coleman
Modified: 2020-05-04 11:20 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1784166 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:20:19 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 591 None closed Bug 1784144: Bump jsonnet dependencies 2020-09-09 10:36:35 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:20:43 UTC

Description Clayton Coleman 2019-12-16 19:11:14 UTC
During investigation of a bug we realized etcdHighNumberOfLeaderChanges was not firing in the fleet due to a copy-paste error when the query was refactored upstream.  It was firing if rate was above 3/s, when it should be 3/15m.

The alert should be fixed and backported since leader changes are symptomatic of insufficient resources or other disruption.

Comment 1 Greg Blomquist 2019-12-19 19:26:01 UTC
https://github.com/openshift/cluster-monitoring-operator/pull/591 is merged, moving to modified

Comment 4 ge liu 2020-01-02 10:11:33 UTC
Hello @paulfantom@redhat.com, could u help to take a look for this issue, thanks in advance!

Comment 8 ge liu 2020-03-08 05:08:24 UTC
Verified on 4.4 0303
etcdHighNumberOfLeaderChanges
etcd cluster "etcd": 21.428571428571427 leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.

Comment 10 errata-xmlrpc 2020-05-04 11:20:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.