Bug 2010599

Summary: Assign realistic resource requests to pods in openshift-marketplace namespace
Product: OpenShift Container Platform Reporter: Hradayesh Shukla <hshukla>
Component: OLMAssignee: Kevin Rizza <krizza>
OLM sub component: OperatorHub QA Contact: Jian Zhang <jiazha>
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: medium CC: davegord, krizza, oarribas, sdharma
Version: 4.7   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-01 17:19:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Hradayesh Shukla 2021-10-05 06:48:08 UTC
Description of problem:
- The requested memory limit for most of the pods under openshift-marketplace is ~50Mi. 

- However, the actual consumption reaches 800 to 1200 MB which risks other workloads on the node as well as jeopardizing the stability of our environment. 

- It can be noticed that these pods utilize significantly higher memory.  
$ oc adm top pods -n openshift-marketplace
NAME                                    CPU(cores)   MEMORY(bytes)   
certified-operators-jx2hg               2m           858Mi           
community-operators-mfm8b               3m           21Mi            
marketplace-operator-798dd66c59-2bjpl   3m           84Mi            
redhat-marketplace-8f7sv                3m           652Mi           
redhat-operators-m2j5g                  3m           1058Mi    

- Due to this there was a node almost about to go down due to OOM. 

- Since these requests are hard-coded in the operator, there's no way to manipulate these values.  

- As per the following BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2000144, there are some great improvements expected in newer releases, but there is a chance they may/may not be backported into 4.7/4.8 and it might take a while too. 

- The customer is looking for some urgent help on this since it's impacting the business as well. Additional info in the internal notes. 

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Login to a running cluster and run:
$ oc adm top pods -n openshift-marketplace

Comment 1 Hradayesh Shukla 2021-10-05 06:48:34 UTC
Actual results:
- Notice the high memory utilization by these pods. 

Expected results:
- It is expected to apply realistic memory requests to these pods.