With a high number of servicemonitor objects (+2000), the container prometheus-config-reloader from the Prometheus user workload monitoring stack is OOM killed by the kernel due to it exceeds the configured memory CGroup. I realized that the memory usage of this container is limited to 25 MiB.
sh-4.4# systemctl status 25868
Warning: The unit file, source configuration file or drop-ins of crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope changed on disk. Run 'systemctl daemon-reload' to reload units.
● crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope - libcontainer container b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952
Loaded: loaded (/run/systemd/transient/crio-b15f21dd2a580453516a5979d17b744e6ecce0a7c81f309f78e532d78ced4952.scope; transient)
└─50-DevicePolicy.conf, 50-DeviceAllow.conf, 50-MemoryLimit.conf, 50-CPUShares.conf, 50-CPUQuota.conf, 50-TasksAccounting.conf, 50-TasksMax.conf
Active: active (running) since Tue 2020-10-13 10:42:16 UTC; 10min ago
Tasks: 11 (limit: 1024)
Memory: 23.8M (limit: 25.0M)
└─25868 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://localhost:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.y>
Still have to confirm if the number of targets pointed by a ServiceMonitor object also affects memory usage.
Note: The same container for the monitoring cluster stack doesn't have this resource limitation.
Good catch! Somehow we never set "--config-reloader-memory=0" for the Prometheus operator running in openshift-user-workload-monitoring namespace (unlike what is done in openshift-monitoring).
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.