With a high number of servicemonitor objects (+2000), the container prometheus-config-reloader from the Prometheus user workload monitoring stack is OOM killed by the kernel due to it exceeds the configured memory CGroup. I realized that the memory usage of this container is limited to 25 MiB. sh-4.4# systemctl status 25868 Warning: The unit file, source configuration file or drop-ins of crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope changed on disk. Run 'systemctl daemon-reload' to reload units. ● crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope - libcontainer container b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952 Loaded: loaded (/run/systemd/transient/crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope; transient) Transient: yes Drop-In: /run/systemd/transient/crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope.d └─50-DevicePolicy.conf, 50-DeviceAllow.conf, 50-MemoryLimit.conf, 50-CPUShares.conf, 50-CPUQuota.conf, 50-TasksAccounting.conf, 50-TasksMax.conf Active: active (running) since Tue 2020-10-13 10:42:16 UTC; 10min ago Tasks: 11 (limit: 1024) Memory: 23.8M (limit: 25.0M) CPU: 89ms CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140bf972_2678_4117_b21e_b3b40c3aba75.slice/crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope └─25868 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://localhost:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.y> Still have to confirm if the number of targets pointed by a ServiceMonitor object also affects memory usage. Note: The same container for the monitoring cluster stack doesn't have this resource limitation.
Good catch! Somehow we never set "--config-reloader-memory=0" for the Prometheus operator running in openshift-user-workload-monitoring namespace (unlike what is done in openshift-monitoring).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633