Bug 2126626

Issue 2

Description of the problem:

Currently, a bunch of ocs-consumers are failing. The trigger is the ocs-operator pod getting OOMKilled because of its demand of higher memory usage than the memory limits allocated to them.
The root cause behind this is that ocs-operator is relying on the default "AllNamespaces" cache in controller-runtime, which works by syncing all the Kubernetes resources in it when the operator starts running for the first time.

The problem here is that the initial informer cache sync is so huge that it causes a sudden massive spike in the memory usage of the operator. And even worse, this spike is directly proportional to the amount of resources present in the underlying Kubernetes/Openshift cluster.

The kind of memory limits configured for ocs-operator mostly compensates for the memory spike by a close margin, yet there are a few situations where the memory spike is not compensated by the set memory limits which causes OOMKilled failures for the ocs-operator pods.

Reproducing the issue:

Currently, there isn't any sure short way of reproducing this issue except somehow making the ocs-operator pod consume higher memory. You can start off with having a lot of resources on the cluster, and then installing ocs-operator on it.

Component controlling this:

ocs-operator 4.10.4

Proposed Mitigation:

This is not a solution but just a mitigation for the existing problems (solution in the next section):

Bump the memory limits of ocs-operator from 200Mi to 800Mi. We came to 800Mi as per our experience of manually fixing the clusters facing the same issue.

Specifically, this line (https://github.com/red-hat-storage/ocs-osd-deployer/blob/02ebe3916210326d00fae53bf55cbfef53ac1edb/utils/resources.go#L90) has to be modified to resource.MustParse("200Mi"),

Proposed solution:

The ideal solution would be to have a dedicated informer caching mechanism set, rather than the default "AllNamespaces" one, which would only cache-sync the resources / custom-resources which ocs-operator cares about.

ocs-operator's code would have to use the controller-runtime's cache package to build a customer cache builder for itself.

For example: https://github.com/openshift/addon-operator/blob/main/cmd/addon-operator-manager/main.go#L179-L188

Expected Results:
ocs-operator pods running without facing any OOMKill failures and without needing any restarts.

 Actual Results:
❯ oc get pods -n openshift-storage ocs-operator-5bf7c58cc9-pbjtj
NAME                            READY   STATUS             RESTARTS         AGE
ocs-operator-5bf7c58cc9-pbjtj   0/1     CrashLoopBackOff   9520 (74s ago)   37d
ocs-operator pod stuck in a CrashLoopBackOff because of getting OOMKilled and restarting itself in the hopes of working properly.

--- Additional comment from Leela Venkaiah Gangavarapu on 2022-09-12 10:49:47 UTC ---

- Interim fix will be delivered via https://bugzilla.redhat.com/show_bug.cgi?id=2125254

--- Additional comment from Yashvardhan Kukreja on 2022-09-13 07:26:38 UTC ---