Red Hat Bugzilla – Bug 1430651
Not able to set elasticsearch minimum heap size
Last modified: 2017-11-28 16:53:01 EST
Description of problem: While debugging an issue with ES instance memory, it has been noticed that in OCP documentation[0] there is a link to Elastic docs regarding heap size[1]. In the Elastic docs[1], it's mentioned ES_HEAP_SIZE and also ES_JAVA_OPTS and there's an example about using the latter that sets heap min and max to same value. None of these env. vars is being used. Then there's ES_INSTANCE_RAM which is used to set the max heap size to half of it, but does nothing to the min value, based on the run.sh[2] script. Shouldn't there be a way to influence also the minimum size as shown in the Elastic docs[1] ? [0] https://docs.openshift.com/container-platform/3.4/install_config/aggregate_logging_sizing.html#install-config-aggregate-logging-sizing-guidelines-installation [1] https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_give_half_your_memory_to_lucene [2] https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/run.sh Version-Release number of selected component (if applicable): 3.4.0 How reproducible: Always Steps to Reproduce: 1. Start an ES instance, min heap size will always be set to 128M. Actual results: Min heap size set to 128M. Expected results: Allow the user to customize it, e.g. using ES_JAVA_OPTS. Additional info:
This appears to be fixed by Lukas Vlcek in commit 5ae268a6a67dc999d7ebe64bc31b4c9ab76695cc. Both initial and max heap size are derived from 'openshift_logging_es_memory_limit' and calculated as half of what the variable contains by setting 'ES_JAVA_OPTS'. Investigating the container, however, it appears that elastic search passes to the java command both original and 'ES_JAVA_OPTS' heap sizes. This shouldn't be a problem with current java implementation, but behavior is undefined and could cause issues some other day. I will try to track the source of this problem. ps aux /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start --path.conf=/usr/share/elasticsearch/config --security.manager.enabled false
This will be resolved in OCP once the changes from origin are merged downstream
According to the fix, -Xms and -Xmx size is half of available space, The available space is 8G, so -Xms and -Xmx is 4096m in my testing. In es log # oc logs ${ES_POD} Comparing the specified RAM to the maximum recommended for Elasticsearch... Inspecting the maximum RAM available... ES_JAVA_OPTS: '-Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m' # oc rsh ${ES_POD} sh-4.2$ ps -aux | grep java 1000080+ 1 5.6 6.5 6864932 493792 ? Ssl 03:42 4:30 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m -Des.path.home=/usr/share/java/elasticsearch -cp /usr/share/java/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/java/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start --path.conf=/usr/share/java/elasticsearch/config --security.manager.enabled false Testing environment: # openshift version openshift v3.6.95 kubernetes v1.6.1+5115d708d7 etcd 3.1.0 Image id from ops registry: logging-elasticsearch v3.6 b0c2867edcbc 7 hours ago 404.5 MB logging-kibana v3.6 5c3c0c5b7c4a 7 hours ago 342.4 MB logging-fluentd v3.6 7cc7b5e98300 7 hours ago 232.5 MB logging-auth-proxy v3.6 b9354650082f 7 hours ago 229.6 MB logging-curator v3.6 028e689a3276 4 weeks ago 211.1 MB t
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188