Bug 1430651

Summary:	Not able to set elasticsearch minimum heap size
Product:	OpenShift Container Platform	Reporter:	Sergi Jimenez Romero <sjr>
Component:	Logging	Assignee:	Jeff Cantrill <jcantril>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	3.4.0	CC:	aos-bugs, fcami, jcantril, jwozniak, pportant, rmeggins, saime, wsun
Target Milestone:	---
Target Release:	3.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-11-28 21:53:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sergi Jimenez Romero 2017-03-09 08:41:28 UTC

Description of problem:

While debugging an issue with ES instance memory, it has been noticed that in OCP documentation[0] there is a link to Elastic docs regarding heap size[1].

In the Elastic docs[1], it's mentioned ES_HEAP_SIZE and also ES_JAVA_OPTS and there's an example about using the latter that sets heap min and max to same value. None of these env. vars is being used.

Then there's ES_INSTANCE_RAM which is used to set the max heap size to half of it, but does nothing to the min value, based on the run.sh[2] script.

Shouldn't there be a way to influence also the minimum size as shown in the Elastic docs[1] ?

[0] https://docs.openshift.com/container-platform/3.4/install_config/aggregate_logging_sizing.html#install-config-aggregate-logging-sizing-guidelines-installation

[1] https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_give_half_your_memory_to_lucene

[2] https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/run.sh

Version-Release number of selected component (if applicable):
3.4.0

How reproducible:

Always

Steps to Reproduce:

1. Start an ES instance, min heap size will always be set to 128M.


Actual results:

Min heap size set to 128M.

Expected results:

Allow the user to customize it, e.g. using ES_JAVA_OPTS.

Additional info:

Comment 5 Jan Wozniak 2017-05-31 15:05:23 UTC

This appears to be fixed by Lukas Vlcek in commit 5ae268a6a67dc999d7ebe64bc31b4c9ab76695cc. Both initial and max heap size are derived from 'openshift_logging_es_memory_limit' and calculated as half of what the variable contains by setting 'ES_JAVA_OPTS'.

Investigating the container, however, it appears that elastic search passes to the java command both original and 'ES_JAVA_OPTS' heap sizes. This shouldn't be a problem with current java implementation, but behavior is undefined and could cause issues some other day. I will try to track the source of this problem.



ps aux
/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start --path.conf=/usr/share/elasticsearch/config --security.manager.enabled false

Comment 6 Jeff Cantrill 2017-06-05 15:53:29 UTC

This will be resolved in OCP once the changes from origin are merged downstream

Comment 7 Junqi Zhao 2017-06-06 05:35:53 UTC

According to the fix, -Xms and -Xmx size is half of available space, 

The available space is 8G, so -Xms and -Xmx is 4096m in my testing.

In es log
# oc logs ${ES_POD}
Comparing the specified RAM to the maximum recommended for Elasticsearch...
Inspecting the maximum RAM available...
ES_JAVA_OPTS: '-Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m'

# oc rsh ${ES_POD}
sh-4.2$ ps -aux | grep java
1000080+      1  5.6  6.5 6864932 493792 ?      Ssl  03:42   4:30 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m -Des.path.home=/usr/share/java/elasticsearch -cp /usr/share/java/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/java/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start --path.conf=/usr/share/java/elasticsearch/config --security.manager.enabled false

Testing environment:
# openshift version
openshift v3.6.95
kubernetes v1.6.1+5115d708d7
etcd 3.1.0

Image id from ops registry:
logging-elasticsearch                                v3.6                b0c2867edcbc        7 hours ago         404.5 MB
logging-kibana                                       v3.6                5c3c0c5b7c4a        7 hours ago         342.4 MB
logging-fluentd                                      v3.6                7cc7b5e98300        7 hours ago         232.5 MB
logging-auth-proxy                                   v3.6                b9354650082f        7 hours ago         229.6 MB
logging-curator                                      v3.6                028e689a3276        4 weeks ago         211.1 MB
t

Comment 12 errata-xmlrpc 2017-11-28 21:53:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188