Description of problem: It seems that the defrag-controller was missing proper testing for large scale OpenShift Container Platform 4 - Clusters, as based on https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501 there is a number of issues discovered when etcd has a size bigger than 3 GB. Given the current issues reported and not knowing the overall impact for 3rd party workloads it's required to disable the defrag controller with immediate effect or provide a switch to disable it in case it's causing issues. Considering that in large scale OpenShift Container Platform 4 - Clusters the defrag is likely to happen every 10 minute and taking up to 60 second this will introduce massive disruption on the OpenShift Container Platform 4 - API with unknown effects on 3rd party application and components. Seeing the issues easily found with some regular testing (check https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501) it's time to act now before certain critical issues arise. Version-Release number of selected component (if applicable): - OpenShift Container Platform 4.9.24 (but actually starting with OpenShift Container Platform 4.9) How reproducible: - Always Steps to Reproduce: 1. Setup OpenShift Container Platform on AWS with Master and Worker of type m5.4xlarge 2. Install Elasticsearch, Logging, Jaeger, NFD, GitOps, Pipelines, Kiali and Service Mesh Operator 3. Run `for i in {5000..7125}; do oc new-project project-$i; oc create configmap project-$i --from-file=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt; done` Mind `/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt` is about 264K in size and therefore something in that area should be used Actual results: Many component to restart as reported in https://issues.redhat.com/browse/ETCD-58?focusedCommentId=19976501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19976501 and unpredictable effects on 3rd party application and components Also every 10 minute an outage of the OpenShift Container Platform 4 - API for about 60 seconds which will quickly eat up error budget for the OpenShift Container Platform 4 - API Expected results: Either make the defrag controller more robust for large scale OpenShift Container Platform 4 - environment or allow disable the defrag-controller to prevent major issues from happening (as there is currently no way to prevent failures in case some are found) Additional info:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.35 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:2283