Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2068084

Summary:	cluster-etcd-operator - disable defrag-controller as there is unpredictable impact on large OpenShift Container Platform 4 - Cluster
Product:	OpenShift Container Platform	Reporter:	Simon Reber <sreber>
Component:	Etcd	Assignee:	Thomas Jungblut <tjungblu>
Status:	CLOSED ERRATA	QA Contact:	ge liu <geliu>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.9	CC:	alray, cpassare, dwest, melbeher, moddi, tjungblu
Target Milestone:	---
Target Release:	4.9.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-25 04:30:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2079725
Bug Blocks:

Description Simon Reber 2022-03-24 12:30:18 UTC

Description of problem:

It seems that the defrag-controller was missing proper testing for large scale OpenShift Container Platform 4 - Clusters, as based on https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501 there is a number of issues discovered when etcd has a size bigger than 3 GB.

Given the current issues reported and not knowing the overall impact for 3rd party workloads it's required to disable the defrag controller with immediate effect or provide a switch to disable it in case it's causing issues.

Considering that in large scale OpenShift Container Platform 4 - Clusters the defrag is likely to happen every 10 minute and taking up to 60 second this will introduce massive disruption on the OpenShift Container Platform 4 - API with unknown effects on 3rd party application and components.

Seeing the issues easily found with some regular testing (check https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501) it's time to act now before certain critical issues arise.

Version-Release number of selected component (if applicable):

 - OpenShift Container Platform 4.9.24 (but actually starting with OpenShift Container Platform 4.9)

How reproducible:

 - Always

Steps to Reproduce:
1. Setup OpenShift Container Platform on AWS with Master and Worker of type m5.4xlarge
2. Install Elasticsearch, Logging, Jaeger, NFD, GitOps, Pipelines, Kiali and Service Mesh Operator
3. Run `for i in {5000..7125}; do oc new-project project-$i; oc create configmap project-$i --from-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt; done`
   Mind `/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt` is about 264K in size and therefore something in that area should be used

Actual results:

Many component to restart as reported in https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501 and unpredictable effects on 3rd party application and components

Also every 10 minute an outage of the OpenShift Container Platform 4 - API for about 60 seconds which will quickly eat up error budget for the OpenShift Container Platform 4 - API

Expected results:

Either make the defrag controller more robust for large scale OpenShift Container Platform 4 - environment or allow disable the defrag-controller to prevent major issues from happening (as there is currently no way to prevent failures in case some are found)

Additional info:

Comment 13 errata-xmlrpc 2022-05-25 04:30:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.35 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:2283