Bug 1369914

Summary: Elasticsearch heap dump should not be written to the root partition
Product: OpenShift Container Platform Reporter: Luke Meyer <lmeyer>
Component: LoggingAssignee: Jan Wozniak <jwozniak>
Status: CLOSED ERRATA QA Contact: Xia Zhao <xiazhao>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: aos-bugs, jcantril, jwozniak, pportant, rmeggins, smunilla
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Consequence: Fix: Specify a location to write a heap dump other then the root partition Result:
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:15:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Luke Meyer 2016-08-24 17:26:20 UTC 
Description of problem:
If ES heap dumps, that is written to / immediately filling the container volume.

Steps to Reproduce:
1. Install logging
2. Induce a heap dump 
  * in our case, we hit an OOME by creating a whole lot of indices; you could probably do this by lowering RAM, creating a lot of projects, and waiting
  * there may be some java tool to induce heap dump, but I couldn't get jmap to work from outside the container
3. rsh into the ES container and df -h /

Actual results:
/ is full


Expected results:
Heap dump either goes to persistent storage volume, or doesn't happen. / does not get filled.

Additional info:
We could hopefully provide a deployer+ES option to either disable or redirect the heap dump.
Comment 2 Jan Wozniak 2017-06-21 06:45:53 UTC 
This is implemented and waiting for a merge:

https://github.com/openshift/openshift-ansible/pull/4451
https://github.com/openshift/origin-aggregated-logging/pull/486
Comment 3 openshift-github-bot 2017-06-22 15:51:23 UTC 
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/cc1ede6f9f296ed4725d32bb580196f9b69b7383
bug 1369914 - write ES heap dump to HEAP_DUMP_LOCATION

Writing ES heap dump to default location filled up the container volume. Heap
dump location is now configurable with HEAP_DUMP_LOCATION env variable and
defaults to /elasticsearch/persistent/hdump.prof
Comment 5 Xia Zhao 2017-07-12 02:51:55 UTC 
It's fixed, checked the recent es image containing bug fix, env variable HEAP_DUMP_LOCATION is effective and the heap dump data no longer goes into / anymore, set to verified:

Image tested with:
logging-elasticsearch      v3.6                bcacb3cf0505        9 hours ago         404.2 MB


# oc rsh logging-es-data-master-kmimip8i-1-5s5qg
sh-4.2$ env | grep HEAP_DUMP_LOCATION
HEAP_DUMP_LOCATION=/elasticsearch/persistent/heapdump.hprof


sh-4.2$ mount | grep /elasticsearch/persistent
/dev/mapper/rhel-root on /elasticsearch/persistent type xfs (rw,relatime,seclabel,attr2,inode64,noquota)


sh-4.2$ df -h  
Filesystem                                                                                        Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:0-394235-7255b42e8f2039064761cc6383db4f650d2fe1be772d386f19ff1254dac2f5aa   10G  442M  9.6G   5% /
tmpfs                                                                                             3.9G     0  3.9G   0% /dev
tmpfs                                                                                             3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/rhel-root                                                                              10G  4.7G  5.4G  47% /etc/hosts
shm                                                                                                64M     0   64M   0% /dev/shm
tmpfs                                                                                             3.9G   32K  3.9G   1% /etc/elasticsearch/secret
tmpfs                                                                                             3.9G   16K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
Comment 7 errata-xmlrpc 2017-08-10 05:15:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716