Bug 1984736 - [master] ClusterDeployment controller watches all Secrets from all namespaces
Summary: [master] ClusterDeployment controller watches all Secrets from all namespaces
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.9
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.9.0
Assignee: Fred Rolland
QA Contact: Yuri Obshansky
URL:
Whiteboard:
Depends On:
Blocks: 1986081
TreeView+ depends on / blocked
 
Reported: 2021-07-22 05:18 UTC by Flavio Percoco
Modified: 2021-10-18 17:40 UTC (History)
3 users (show)

Fixed In Version: OCP-Metal-v1.0.24.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1986081 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:40:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift assisted-service pull 2286 0 None closed OCPBUGSM-32293: Cache only required secrets 2021-07-29 06:54:53 UTC
Red Hat Bugzilla 1956551 1 low CLOSED assisted-service pod consuming high memory utilization 2021-09-27 11:58:40 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:40:57 UTC

Internal Links: 1986081

Description Flavio Percoco 2021-07-22 05:18:43 UTC
Description of problem:

Most of the memory for assited-service is being used by the controllers and, most of that memory seems to be allocated by the kube client's cache. From a recent investigation, after a 1k deployemnt and when assisted-service was idle, there were ~350Mb allocated only by Secret objects.

It seems like we are watching the Secrets in the clusterdeployment controller I'm looking to see if there's a way for us to not do this, or to optimize it so that we only watch for the secrets we care about (by using some predicates and custom cache instance).

Few ideas here:

* It looks like not watching the secret is not an option. Is this assumption correct?

* Would it be possible to require the secret needed by ClusterDeployment (I believe it's just the pull-secret) to be annotated/labeled?

* If we don't want the user to annotate/label the secret, could we annotate the secret automatically in the reconcile loop?

An early discussion on the above resulted in folks aligning on the idea of using labels and assigning them automatically to the Secrets so that a more restricted cache can be implemented.

Comment 5 errata-xmlrpc 2021-10-18 17:40:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.