Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2024311

Summary:	cluster-etcd-operator: defrag controller needs to provide proper observability
Product:	OpenShift Container Platform	Reporter:	Sam Batschelet <sbatsche>
Component:	Etcd	Assignee:	Dean West <dwest>
Status:	CLOSED NOTABUG	QA Contact:	ge liu <geliu>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.9	CC:	alray, geliu, tjungblu
Target Milestone:	---
Target Release:	4.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2024309	Environment:
Last Closed:	2022-07-20 11:12:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2024309
Bug Blocks:

Description Sam Batschelet 2021-11-17 19:30:47 UTC

+++ This bug was initially created as a clone of Bug #2024309 +++

Description of problem: Today cluster-etcd-operator's defrag controller has a few flaws which this bug intends to cover.

- first, cluster-etcd-oeprator does not collect etcd client metrics and expose those to telemetry. By doing this we can consider alerts on failed client attempts to Defrag.

- the defrag controller currently performs defragmentation then checks cluster health if success defrag the next. In some circumstances such as smaller state files the duration in between defrag attempts can be very short. This can cause the etcd client balancer to perhaps choose poorly and result in a timeout. Resolution would be to ensure a reasonable time exits between defrag attempts.

- the current metrics scrape rate for etcd is 30s so when we are trying to identify defragmentation attempts they appear in some circumstances to happen at the same time. to improve this lets ensure we wait +30s in between defrag attempts.

- if defrag fails we only log the error and do not event.



Version-Release number of selected component (if applicable):


How reproducible: high


Steps to Reproduce:
1. install 4.9 cluster and observe metrics/logs/events
2.
3.

Actual results: defragmentation is not clearly observeable 


Expected results: clean signal and understanding of defrag success/failure.


Additional info:

Comment 3 Thomas Jungblut 2022-07-20 11:12:45 UTC

create this as a backlog item in https://issues.redhat.com/browse/ETCD-303