Back to bug 2126626

Who When What Removed Added
Malay Kumar parida 2022-09-14 07:52:06 UTC Assignee nigoyal mparida
Component odf-operator ocs-operator
CC madam, mparida, sostapov
Malay Kumar parida 2022-09-16 17:40:28 UTC Link ID Github red-hat-storage/ocs-operator/pull/1817
Status NEW POST
Leela Venkaiah Gangavarapu 2022-09-26 10:57:44 UTC Blocks 2121329
Depends On 2121329
Dhruv Bindra 2022-10-03 07:57:52 UTC Blocks 2131662
Nitin Goyal 2022-10-11 14:57:02 UTC CC dbindra
Nitin Goyal 2022-10-11 14:57:59 UTC Status POST MODIFIED
CC nigoyal
Malay Kumar parida 2022-10-12 07:24:14 UTC Status MODIFIED ON_QA
krishnaram Karthick 2022-10-14 04:28:54 UTC QA Contact mbukatov nberry
CC kramdoss
Neha Berry 2022-10-17 07:19:36 UTC QA Contact nberry ebondare
RHEL Program Management 2022-10-27 07:11:16 UTC Target Release --- ODF 4.12.0
Sunil Kumar Acharya 2022-12-08 12:52:38 UTC Flags needinfo?(mparida)
Malay Kumar parida 2022-12-08 13:25:20 UTC Doc Type If docs needed, set a value Bug Fix
Flags needinfo?(mparida)
Doc Text Cause:
OCS-operator is relying on the default "AllNamespaces" cache in the controller-runtime, which works by syncing all the Kubernetes resources in it when the operator starts running for the first time.

Consequence:
The initial informer cache sync is so huge that it causes a sudden massive spike in the memory usage of the operator. And this spike is directly proportional to the amount of resources present in the underlying Kubernetes/Openshift cluster.
The kind of memory limits configured for ocs-operator mostly compensates for the memory spike by a close margin, yet there are a few situations where the memory spike is not compensated by the set memory limits which causes OOMKilled failures for the ocs-operator pods.

Fix:
Rather than the default "AllNamespaces" cache, We specify a cache which would only cache-sync the resources / custom-resources in the same namespace.

Result:
This massively reduces operator memory usage spike & would help to avoid OOMKIlled situation for OCS operator pod.
Red Hat Bugzilla 2022-12-31 19:29:35 UTC CC dbindra
Red Hat Bugzilla 2022-12-31 19:35:15 UTC CC mparida
Assignee mparida muagarwa
Red Hat Bugzilla 2022-12-31 19:54:48 UTC CC nberry
Red Hat Bugzilla 2022-12-31 22:31:45 UTC CC aeyal
Red Hat Bugzilla 2022-12-31 23:27:13 UTC QA Contact ebondare mbukatov
Red Hat Bugzilla 2023-01-01 08:32:22 UTC CC kramdoss
Red Hat Bugzilla 2023-01-01 08:33:13 UTC CC lgangava
Red Hat Bugzilla 2023-01-01 08:44:10 UTC CC sostapov
Harish NV Rao 2023-01-03 06:36:21 UTC QA Contact mbukatov nberry
CC hnallurv
Alasdair Kergon 2023-01-04 04:38:46 UTC QA Contact nberry ebondare
Alasdair Kergon 2023-01-04 04:46:40 UTC Assignee muagarwa mparida
Alasdair Kergon 2023-01-04 04:51:45 UTC CC dbindra
Alasdair Kergon 2023-01-04 05:07:00 UTC CC kramdoss
Alasdair Kergon 2023-01-04 05:12:30 UTC CC lgangava
Alasdair Kergon 2023-01-04 05:17:49 UTC CC mparida
Alasdair Kergon 2023-01-04 05:18:56 UTC CC nberry
Alasdair Kergon 2023-01-04 05:43:50 UTC CC sostapov
Alasdair Kergon 2023-01-04 06:06:57 UTC CC aeyal
Red Hat Bugzilla 2023-01-16 08:27:46 UTC CC jrivera
Neha Berry 2023-01-17 13:17:20 UTC Status ON_QA VERIFIED
Neha Berry 2023-01-17 13:19:33 UTC Blocks 2161650
Red Hat Bugzilla 2023-01-31 22:26:50 UTC CC omitrani
Red Hat Bugzilla 2023-01-31 23:38:57 UTC CC madam
Rejy M Cyriac 2023-02-08 14:06:28 UTC Resolution --- CURRENTRELEASE
Status VERIFIED CLOSED
Last Closed 2023-02-08 14:06:28 UTC
Elad 2023-08-09 17:00:43 UTC CC odf-bz-bot

Back to bug 2126626