Bug 1784144

Summary: No etcd alert fires when high number of leader changes occur
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: EtcdAssignee: Sam Batschelet <sbatsche>
Status: CLOSED ERRATA QA Contact: ge liu <geliu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: gblomqui, sttts
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1784166 (view as bug list) Environment:
Last Closed: 2020-05-04 11:20:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1784166    

Description Clayton Coleman 2019-12-16 19:11:14 UTC
During investigation of a bug we realized etcdHighNumberOfLeaderChanges was not firing in the fleet due to a copy-paste error when the query was refactored upstream.  It was firing if rate was above 3/s, when it should be 3/15m.

The alert should be fixed and backported since leader changes are symptomatic of insufficient resources or other disruption.

Comment 1 Greg Blomquist 2019-12-19 19:26:01 UTC
https://github.com/openshift/cluster-monitoring-operator/pull/591 is merged, moving to modified

Comment 4 ge liu 2020-01-02 10:11:33 UTC
Hello @paulfantom, could u help to take a look for this issue, thanks in advance!

Comment 8 ge liu 2020-03-08 05:08:24 UTC
Verified on 4.4 0303
etcdHighNumberOfLeaderChanges
etcd cluster "etcd": 21.428571428571427 leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.

Comment 10 errata-xmlrpc 2020-05-04 11:20:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581