Red Hat Bugzilla – Bug 1369914
Elasticsearch heap dump should not be written to the root partition
Last modified: 2017-08-16 15:51 EDT
Description of problem: If ES heap dumps, that is written to / immediately filling the container volume. Steps to Reproduce: 1. Install logging 2. Induce a heap dump * in our case, we hit an OOME by creating a whole lot of indices; you could probably do this by lowering RAM, creating a lot of projects, and waiting * there may be some java tool to induce heap dump, but I couldn't get jmap to work from outside the container 3. rsh into the ES container and df -h / Actual results: / is full Expected results: Heap dump either goes to persistent storage volume, or doesn't happen. / does not get filled. Additional info: We could hopefully provide a deployer+ES option to either disable or redirect the heap dump.
This is implemented and waiting for a merge: https://github.com/openshift/openshift-ansible/pull/4451 https://github.com/openshift/origin-aggregated-logging/pull/486
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/cc1ede6f9f296ed4725d32bb580196f9b69b7383 bug 1369914 - write ES heap dump to HEAP_DUMP_LOCATION Writing ES heap dump to default location filled up the container volume. Heap dump location is now configurable with HEAP_DUMP_LOCATION env variable and defaults to /elasticsearch/persistent/hdump.prof
It's fixed, checked the recent es image containing bug fix, env variable HEAP_DUMP_LOCATION is effective and the heap dump data no longer goes into / anymore, set to verified: Image tested with: logging-elasticsearch v3.6 bcacb3cf0505 9 hours ago 404.2 MB # oc rsh logging-es-data-master-kmimip8i-1-5s5qg sh-4.2$ env | grep HEAP_DUMP_LOCATION HEAP_DUMP_LOCATION=/elasticsearch/persistent/heapdump.hprof sh-4.2$ mount | grep /elasticsearch/persistent /dev/mapper/rhel-root on /elasticsearch/persistent type xfs (rw,relatime,seclabel,attr2,inode64,noquota) sh-4.2$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker-253:0-394235-7255b42e8f2039064761cc6383db4f650d2fe1be772d386f19ff1254dac2f5aa 10G 442M 9.6G 5% / tmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/mapper/rhel-root 10G 4.7G 5.4G 47% /etc/hosts shm 64M 0 64M 0% /dev/shm tmpfs 3.9G 32K 3.9G 1% /etc/elasticsearch/secret tmpfs 3.9G 16K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716