Bug 1576543 - prometheus-operator pods getting OoM killed @ 750 nodes
Summary: prometheus-operator pods getting OoM killed @ 750 nodes
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.2.0
Assignee: Frederic Branczyk
QA Contact: Mike Fiedler
Whiteboard: aos-scalability-310
Depends On:
TreeView+ depends on / blocked
Reported: 2018-05-09 17:19 UTC by Jiří Mencák
Modified: 2019-10-16 06:27 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2019-10-16 06:27:40 UTC
Target Upstream Version:

Attachments (Terms of Use)
OoM kills (dmesg), oc get pods, oc describe pod prometheus-operator* (41.76 KB, application/x-gzip)
2018-05-09 17:19 UTC, Jiří Mencák
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:27:56 UTC

Description Jiří Mencák 2018-05-09 17:19:11 UTC
Created attachment 1433959 [details]
OoM kills (dmesg), oc get pods, oc describe pod prometheus-operator*

Description of problem:

$ oc version
oc v3.10.0-0.32.0
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://lb-0.scale-ci.example.com:8443
openshift v3.10.0-0.32.0
kubernetes v1.10.0+b81c8f8

Steps to Reproduce:
1. Install a larger OCP cluster and watch prometheus-operator getting OoM killed

Actual results:
Please see attachment.O

Expected results:
No OoM kills.

Comment 2 Frederic Branczyk 2019-02-06 17:21:32 UTC
We have done various scalability changes for 4.0, this needs to be re-assessed in the 4.0 scope.

Comment 4 Junqi Zhao 2019-02-19 01:27:45 UTC

Could you also help to test this aos-scalability bug?

Comment 6 Junqi Zhao 2019-04-11 00:58:23 UTC
Did not find this issue in one smaller cluster, not sure if it would be happen in a larger cluster

Comment 9 Mike Fiedler 2019-09-13 16:39:59 UTC
Marking verified on 4.2.   There won't be another 750+ node cluster run until post-4.2 and a new bz can be opened then if there is an issue.   In a 250 node cluster on GCP, prometheus-operator is using 80Mb VSZ and 2MB RSS

Comment 11 errata-xmlrpc 2019-10-16 06:27:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.