Bug 1856502 - Proxy container in elasticsearch pods restarting due to oom-kill during test of scale cluster
Summary: Proxy container in elasticsearch pods restarting due to oom-kill during test ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.6.0
Assignee: Vimal Kumar
QA Contact: Eric Matysek
URL:
Whiteboard:
Depends On:
Blocks: 1860084 1894632
TreeView+ depends on / blocked
 
Reported: 2020-07-13 19:18 UTC by Eric Matysek
Modified: 2020-11-23 15:31 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1860084 1894632 (view as bug list)
Environment:
Last Closed: 2020-10-27 16:14:10 UTC
Target Upstream Version:


Attachments (Terms of Use)
OOM messages (1.21 KB, text/plain)
2020-07-13 19:18 UTC, Eric Matysek
no flags Details
Elasticsearch pod logs (114.15 KB, application/gzip)
2020-07-13 19:26 UTC, Eric Matysek
no flags Details
Describe elasticsearch pod (6.42 KB, text/plain)
2020-07-13 19:38 UTC, Eric Matysek
no flags Details
deployment and elasticsearch yaml (3.92 KB, application/gzip)
2020-08-06 18:28 UTC, Eric Matysek
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 610 0 None closed Bug 1856502: Added Elasticsearch proxy resources 2021-02-03 16:35:34 UTC
Github openshift elasticsearch-operator pull 426 0 None closed Bug 1856502: Add ES-Proxy Resources to Elasticsearch CR 2021-02-03 16:35:35 UTC
Github openshift elasticsearch-operator pull 448 0 None closed Bug 1856502: Fixing rollout of nodes when proxy resources are updated 2021-02-03 16:35:35 UTC
Red Hat Knowledge Base (Solution) 5541601 0 None None None 2020-11-04 17:03:08 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:14:24 UTC

Description Eric Matysek 2020-07-13 19:18:20 UTC
Created attachment 1700896 [details]
OOM messages

Description of problem:
We have a 250 node cluster with very large infra nodes for elasticsearch. Elasticsearch pods are continuously getting oom-killed due to cgroup limits causing issues with large scale logging testing and missing messages.


Version-Release number of selected component (if applicable):
4.5

How reproducible:
100%

Steps to Reproduce:
1. Deploy CLO with Elasticsearch requests/limits set to 16cpu/96Gi
2. Generate logs at 100/s across 250 projects


Actual results:
Dropped messages
Elasticsearch pods restarting ~16 times during a 30 minute test run

Expected results:
No dropped messages
Stable elasticsearch pods

Additional info:
OOM messages attached

Comment 1 Eric Matysek 2020-07-13 19:26:46 UTC
Created attachment 1700897 [details]
Elasticsearch pod logs

Comment 2 Eric Matysek 2020-07-13 19:38:49 UTC
Created attachment 1700898 [details]
Describe elasticsearch pod

Looks like it's actually the proxy container being restarted and not the elasticsearch container.

Comment 6 Eric Matysek 2020-07-30 16:00:55 UTC
Elasticsearch proxy containers are now properly adjustable when specified during clusterlogging instance creation.

However, changing this spec after creation triggers elasticsearch pods to be restarted but the new proxy container resources do not take effect.
I see that the change is made in the elasticsearch deployments, but does not propagate to the containers.

[ematysek@jump ~]$ oc get deployments
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
cluster-logging-operator       1/1     1            1           18m
elasticsearch-cdm-jtnifyr9-1   1/1     0            1           17m
elasticsearch-cdm-jtnifyr9-2   1/1     0            1           17m
elasticsearch-cdm-jtnifyr9-3   1/1     0            1           17m

[ematysek@jump ~]$ oc get deployment elasticsearch-cdm-jtnifyr9-1 -o yaml | grep -C 1 -i limit
--
        resources:
          limits:
            memory: 256Mi

[ematysek@jump ~]$ oc get pod elasticsearch-cdm-jtnifyr9-1-74cfdf7cb4-cznkm -o yaml | grep -C 1 -i limit
--
    resources:
      limits:
        memory: 64Mi

Comment 9 Eric Matysek 2020-08-06 18:28:10 UTC
Created attachment 1710691 [details]
deployment and elasticsearch yaml

Comment 12 Eric Matysek 2020-08-11 20:05:58 UTC
Verified on 4.6

Comment 14 errata-xmlrpc 2020-10-27 16:14:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.