Created attachment 1700896 [details] OOM messages Description of problem: We have a 250 node cluster with very large infra nodes for elasticsearch. Elasticsearch pods are continuously getting oom-killed due to cgroup limits causing issues with large scale logging testing and missing messages. Version-Release number of selected component (if applicable): 4.5 How reproducible: 100% Steps to Reproduce: 1. Deploy CLO with Elasticsearch requests/limits set to 16cpu/96Gi 2. Generate logs at 100/s across 250 projects Actual results: Dropped messages Elasticsearch pods restarting ~16 times during a 30 minute test run Expected results: No dropped messages Stable elasticsearch pods Additional info: OOM messages attached
Created attachment 1700897 [details] Elasticsearch pod logs
Created attachment 1700898 [details] Describe elasticsearch pod Looks like it's actually the proxy container being restarted and not the elasticsearch container.
Elasticsearch proxy containers are now properly adjustable when specified during clusterlogging instance creation. However, changing this spec after creation triggers elasticsearch pods to be restarted but the new proxy container resources do not take effect. I see that the change is made in the elasticsearch deployments, but does not propagate to the containers. [ematysek@jump ~]$ oc get deployments NAME READY UP-TO-DATE AVAILABLE AGE cluster-logging-operator 1/1 1 1 18m elasticsearch-cdm-jtnifyr9-1 1/1 0 1 17m elasticsearch-cdm-jtnifyr9-2 1/1 0 1 17m elasticsearch-cdm-jtnifyr9-3 1/1 0 1 17m [ematysek@jump ~]$ oc get deployment elasticsearch-cdm-jtnifyr9-1 -o yaml | grep -C 1 -i limit -- resources: limits: memory: 256Mi [ematysek@jump ~]$ oc get pod elasticsearch-cdm-jtnifyr9-1-74cfdf7cb4-cznkm -o yaml | grep -C 1 -i limit -- resources: limits: memory: 64Mi
Created attachment 1710691 [details] deployment and elasticsearch yaml
Verified on 4.6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196