Bug 2068084 - cluster-etcd-operator - disable defrag-controller as there is unpredictable impact on large OpenShift Container Platform 4 - Cluster
Summary: cluster-etcd-operator - disable defrag-controller as there is unpredictable i...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.9
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 4.9.z
Assignee: Thomas Jungblut
QA Contact: ge liu
Depends On: 2079725
TreeView+ depends on / blocked
Reported: 2022-03-24 12:30 UTC by Simon Reber
Modified: 2022-06-14 08:55 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-05-25 04:30:33 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-etcd-operator pull 810 0 None open [release-4.9] Bug 2068084: manually disable defrag 2022-05-11 06:52:18 UTC
Red Hat Issue Tracker INSIGHTOCP-767 0 None None None 2022-06-14 08:55:10 UTC
Red Hat Knowledge Base (Solution) 6840041 0 None None None 2022-03-24 12:33:03 UTC
Red Hat Product Errata RHSA-2022:2283 0 None None None 2022-05-25 04:30:54 UTC

Description Simon Reber 2022-03-24 12:30:18 UTC
Description of problem:

It seems that the defrag-controller was missing proper testing for large scale OpenShift Container Platform 4 - Clusters, as based on https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501 there is a number of issues discovered when etcd has a size bigger than 3 GB.

Given the current issues reported and not knowing the overall impact for 3rd party workloads it's required to disable the defrag controller with immediate effect or provide a switch to disable it in case it's causing issues.

Considering that in large scale OpenShift Container Platform 4 - Clusters the defrag is likely to happen every 10 minute and taking up to 60 second this will introduce massive disruption on the OpenShift Container Platform 4 - API with unknown effects on 3rd party application and components.

Seeing the issues easily found with some regular testing (check https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501) it's time to act now before certain critical issues arise.

Version-Release number of selected component (if applicable):

 - OpenShift Container Platform 4.9.24 (but actually starting with OpenShift Container Platform 4.9)

How reproducible:

 - Always

Steps to Reproduce:
1. Setup OpenShift Container Platform on AWS with Master and Worker of type m5.4xlarge
2. Install Elasticsearch, Logging, Jaeger, NFD, GitOps, Pipelines, Kiali and Service Mesh Operator
3. Run `for i in {5000..7125}; do oc new-project project-$i; oc create configmap project-$i --from-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt; done`
   Mind `/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt` is about 264K in size and therefore something in that area should be used

Actual results:

Many component to restart as reported in https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501 and unpredictable effects on 3rd party application and components

Also every 10 minute an outage of the OpenShift Container Platform 4 - API for about 60 seconds which will quickly eat up error budget for the OpenShift Container Platform 4 - API

Expected results:

Either make the defrag controller more robust for large scale OpenShift Container Platform 4 - environment or allow disable the defrag-controller to prevent major issues from happening (as there is currently no way to prevent failures in case some are found)

Additional info:

Comment 13 errata-xmlrpc 2022-05-25 04:30:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.35 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.