Bug 2010599 - Assign realistic resource requests to pods in openshift-marketplace namespace
Summary: Assign realistic resource requests to pods in openshift-marketplace namespace
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.7
Hardware: All
OS: All
Target Milestone: ---
: ---
Assignee: Kevin Rizza
QA Contact: Jian Zhang
Depends On:
TreeView+ depends on / blocked
Reported: 2021-10-05 06:48 UTC by Hradayesh Shukla
Modified: 2022-01-19 08:47 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-11-01 17:19:46 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Hradayesh Shukla 2021-10-05 06:48:08 UTC
Description of problem:
- The requested memory limit for most of the pods under openshift-marketplace is ~50Mi. 

- However, the actual consumption reaches 800 to 1200 MB which risks other workloads on the node as well as jeopardizing the stability of our environment. 

- It can be noticed that these pods utilize significantly higher memory.  
$ oc adm top pods -n openshift-marketplace
NAME                                    CPU(cores)   MEMORY(bytes)   
certified-operators-jx2hg               2m           858Mi           
community-operators-mfm8b               3m           21Mi            
marketplace-operator-798dd66c59-2bjpl   3m           84Mi            
redhat-marketplace-8f7sv                3m           652Mi           
redhat-operators-m2j5g                  3m           1058Mi    

- Due to this there was a node almost about to go down due to OOM. 

- Since these requests are hard-coded in the operator, there's no way to manipulate these values.  

- As per the following BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2000144, there are some great improvements expected in newer releases, but there is a chance they may/may not be backported into 4.7/4.8 and it might take a while too. 

- The customer is looking for some urgent help on this since it's impacting the business as well. Additional info in the internal notes. 

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Login to a running cluster and run:
$ oc adm top pods -n openshift-marketplace

Comment 1 Hradayesh Shukla 2021-10-05 06:48:34 UTC
Actual results:
- Notice the high memory utilization by these pods. 

Expected results:
- It is expected to apply realistic memory requests to these pods.

Note You need to log in before you can comment on or make changes to this bug.