Bug 1783284 - OLM process CPU grows as number of projects increase. Spikes to 2+ cores every 5 minutes with 2K projects
Summary: OLM process CPU grows as number of projects increase. Spikes to 2+ cores ev...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Evan Cordell
QA Contact: Mike Fiedler
URL:
Whiteboard:
Depends On: 1810698
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-13 14:00 UTC by Mike Fiedler
Modified: 2020-05-13 21:55 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1810698 (view as bug list)
Environment:
Last Closed: 2020-05-13 21:55:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
OLM cpu on idle cluster after fresh install (59.13 KB, image/png)
2019-12-13 14:00 UTC, Mike Fiedler
no flags Details
OLM cpu on idle cluster with 2K projects (48.45 KB, image/png)
2019-12-13 14:04 UTC, Mike Fiedler
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1356 0 None closed [release-4.4] Bug 1783284: feat(jitter): add jitter to controllers to smooth out our spiky resource 2020-12-03 12:19:35 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-13 21:55:16 UTC

Description Mike Fiedler 2019-12-13 14:00:02 UTC
Created attachment 1644869 [details]
OLM cpu on idle cluster after fresh install

Description of problem:

In a newly installed cluster, the olm process spikes to 15% of 1 core every 5 minutes at steady state - no activity.   As the number of projects grows (project spec below), the 5 minute olm process spikes grow.   At 2000 projects the spikes use 2 full cores every 5 minutes at steady state.

Other control plane processes on the master do not exhibit this behavior when the cluster is idle.

Each project:

6 builds
10 image streams
2 deployments
40 secrets
5 routes
20 configmaps 

Version-Release number of selected component (if applicable):

4.3.0-0.nightly-2019-12-12-021332


How reproducible: always


Steps to Reproduce:
1. Install a 4.3 cluster with 3 masters and 3 computes
2. Measure olm process CPU usage over time,  observe spikes every 5 minutes to ~15%
3. Create more projects with the spec above
4. Observe the growth in the olm process usage

CPU usage graphs for OLM process attached.

Comment 1 Mike Fiedler 2019-12-13 14:04:50 UTC
Created attachment 1644870 [details]
OLM cpu on idle cluster with 2K projects

Comment 7 Mike Fiedler 2020-03-12 20:15:38 UTC
Verified on 4.4.0-0.nightly-2020-03-12-082023

Same results as for https://bugzilla.redhat.com/show_bug.cgi?id=1810698

Comment 9 errata-xmlrpc 2020-05-13 21:55:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.