Bug 2010599

Summary:	Assign realistic resource requests to pods in openshift-marketplace namespace
Product:	OpenShift Container Platform	Reporter:	Hradayesh Shukla <hshukla>
Component:	OLM	Assignee:	Kevin Rizza <krizza>
OLM sub component:	OperatorHub	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED WONTFIX	Docs Contact:
Severity:	medium
Priority:	medium	CC:	davegord, krizza, oarribas, sdharma
Version:	4.7
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-01 17:19:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hradayesh Shukla 2021-10-05 06:48:08 UTC

Description of problem:
- The requested memory limit for most of the pods under openshift-marketplace is ~50Mi.

- However, the actual consumption reaches 800 to 1200 MB which risks other workloads on the node as well as jeopardizing the stability of our environment.

- It can be noticed that these pods utilize significantly higher memory.
~~~
$ oc adm top pods -n openshift-marketplace
NAME CPU(cores) MEMORY(bytes)
certified-operators-jx2hg 2m 858Mi
community-operators-mfm8b 3m 21Mi
marketplace-operator-798dd66c59-2bjpl 3m 84Mi
redhat-marketplace-8f7sv 3m 652Mi
redhat-operators-m2j5g 3m 1058Mi
~~~

- Due to this there was a node almost about to go down due to OOM.

- Since these requests are hard-coded in the operator, there's no way to manipulate these values.

- As per the following BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2000144, there are some great improvements expected in newer releases, but there is a chance they may/may not be backported into 4.7/4.8 and it might take a while too.

- The customer is looking for some urgent help on this since it's impacting the business as well. Additional info in the internal notes.

Version-Release number of selected component (if applicable):

How reproducible:
100%

Steps to Reproduce:
1. Login to a running cluster and run:
$ oc adm top pods -n openshift-marketplace

Comment 1 Hradayesh Shukla 2021-10-05 06:48:34 UTC

Actual results:
- Notice the high memory utilization by these pods. 


Expected results:
- It is expected to apply realistic memory requests to these pods.