Created attachment 1700896 [details]
Description of problem:
We have a 250 node cluster with very large infra nodes for elasticsearch. Elasticsearch pods are continuously getting oom-killed due to cgroup limits causing issues with large scale logging testing and missing messages.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Deploy CLO with Elasticsearch requests/limits set to 16cpu/96Gi
2. Generate logs at 100/s across 250 projects
Elasticsearch pods restarting ~16 times during a 30 minute test run
No dropped messages
Stable elasticsearch pods
OOM messages attached
Created attachment 1700897 [details]
Elasticsearch pod logs
Created attachment 1700898 [details]
Describe elasticsearch pod
Looks like it's actually the proxy container being restarted and not the elasticsearch container.
Elasticsearch proxy containers are now properly adjustable when specified during clusterlogging instance creation.
However, changing this spec after creation triggers elasticsearch pods to be restarted but the new proxy container resources do not take effect.
I see that the change is made in the elasticsearch deployments, but does not propagate to the containers.
[ematysek@jump ~]$ oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
cluster-logging-operator 1/1 1 1 18m
elasticsearch-cdm-jtnifyr9-1 1/1 0 1 17m
elasticsearch-cdm-jtnifyr9-2 1/1 0 1 17m
elasticsearch-cdm-jtnifyr9-3 1/1 0 1 17m
[ematysek@jump ~]$ oc get deployment elasticsearch-cdm-jtnifyr9-1 -o yaml | grep -C 1 -i limit
[ematysek@jump ~]$ oc get pod elasticsearch-cdm-jtnifyr9-1-74cfdf7cb4-cznkm -o yaml | grep -C 1 -i limit
Created attachment 1710691 [details]
deployment and elasticsearch yaml
Verified on 4.6
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.