Bug 2008175

Summary:	fsync controller will show false positive if gaps in metrics are observed.
Product:	OpenShift Container Platform	Reporter:	Sam Batschelet <sbatsche>
Component:	Etcd	Assignee:	Sam Batschelet <sbatsche>
Status:	CLOSED ERRATA	QA Contact:	Sandeep <skundu>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.6	CC:	geliu, skundu
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2013646 (view as bug list)		Environment:
Last Closed:	2021-11-22 21:47:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2013646
Bug Blocks:

Description Sam Batschelet 2021-09-27 13:48:24 UTC

Description of problem: If gaps in metrics are observed by fsync controller they can provide false positives for leader election growth over time series. This can result in sending admin on an invalid path of etcd performance triage.

> Detected leader change increase of 1.25 over 5 minutes on "BareMetal"; disk metrics are: etcd-nfvpe-12.oot.lab.eng.bos.redhat.com=0.0025688888888888875,etcd-nfvpe-13.oot.lab.eng.bos.redhat.com=0.0018824444444444331,etcd-nfvpe-02.oot.lab.eng.bos.redhat.com=0.001912424242424239


The image below shows leader elections(green) with spikes in leader elections(blue) as a result of the query.

https://user-images.githubusercontent.com/1249749/134918692-43ca6cf0-4fb0-4ee3-8735-f5d035d56170.png

This metric query clearly shows the gaps in the collection of metrics from possible networking issues.

https://user-images.githubusercontent.com/1249749/134919312-f62591ee-57df-41de-a313-545e207d89ad.png

Version-Release number of selected component (if applicable):


How reproducible: given condition 100%


Steps to Reproduce:
1. generate gaps in metrics collection by disrupting networking
2. review events from cluster-etcd-operator namespace for Detected leader change increase events.

3. verify etcd is not actually experiencing leader elections `etcd_server_leader_changes_seen_total`

Actual results: leader elections are fasley reported by fsync controller 


Expected results: fsync controller should event only with an actual observation of the issue.


Additional info:

Comment 4 ge liu 2021-11-09 09:59:51 UTC

This bug verification be blocked by new bug: https://bugzilla.redhat.com/show_bug.cgi?id=2021453

Comment 13 errata-xmlrpc 2021-11-22 21:47:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.8 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4712