Bug 1856502
| Summary: | Proxy container in elasticsearch pods restarting due to oom-kill during test of scale cluster | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Eric Matysek <ematysek> | ||||||||||
| Component: | Logging | Assignee: | Vimal Kumar <vimalkum> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Eric Matysek <ematysek> | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | urgent | ||||||||||||
| Version: | 4.5 | CC: | anli, aos-bugs, ewolinet, jcantril, sreber, vimalkum, ychoukse, yhe | ||||||||||
| Target Milestone: | --- | Keywords: | Regression | ||||||||||
| Target Release: | 4.6.0 | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | |||||||||||||
| : | 1860084 1894632 (view as bug list) | Environment: | |||||||||||
| Last Closed: | 2020-10-27 16:14:10 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 1860084, 1894632 | ||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 1700897 [details]
Elasticsearch pod logs
Created attachment 1700898 [details]
Describe elasticsearch pod
Looks like it's actually the proxy container being restarted and not the elasticsearch container.
Elasticsearch proxy containers are now properly adjustable when specified during clusterlogging instance creation.
However, changing this spec after creation triggers elasticsearch pods to be restarted but the new proxy container resources do not take effect.
I see that the change is made in the elasticsearch deployments, but does not propagate to the containers.
[ematysek@jump ~]$ oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
cluster-logging-operator 1/1 1 1 18m
elasticsearch-cdm-jtnifyr9-1 1/1 0 1 17m
elasticsearch-cdm-jtnifyr9-2 1/1 0 1 17m
elasticsearch-cdm-jtnifyr9-3 1/1 0 1 17m
[ematysek@jump ~]$ oc get deployment elasticsearch-cdm-jtnifyr9-1 -o yaml | grep -C 1 -i limit
--
resources:
limits:
memory: 256Mi
[ematysek@jump ~]$ oc get pod elasticsearch-cdm-jtnifyr9-1-74cfdf7cb4-cznkm -o yaml | grep -C 1 -i limit
--
resources:
limits:
memory: 64Mi
Created attachment 1710691 [details]
deployment and elasticsearch yaml
Verified on 4.6 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Created attachment 1700896 [details] OOM messages Description of problem: We have a 250 node cluster with very large infra nodes for elasticsearch. Elasticsearch pods are continuously getting oom-killed due to cgroup limits causing issues with large scale logging testing and missing messages. Version-Release number of selected component (if applicable): 4.5 How reproducible: 100% Steps to Reproduce: 1. Deploy CLO with Elasticsearch requests/limits set to 16cpu/96Gi 2. Generate logs at 100/s across 250 projects Actual results: Dropped messages Elasticsearch pods restarting ~16 times during a 30 minute test run Expected results: No dropped messages Stable elasticsearch pods Additional info: OOM messages attached