Bug 816299

+++ This bug was initially created as a clone of Bug #807875 +++

Description of problem:
<customer>
Recently, the root volume of our satellite server filled to 100%.  As it turns out, it was filled by numerous JVM heap dumps.  I found the following KB article that seems to match our symptoms:

https://access.redhat.com/knowledge/solutions/43122

Also, and perhaps more importantly, the heap dumps are ending up in /usr/sbin, which seems like an inappropriate place to put them -- somewhere in /var seems like it would be a more standard place.  Is there a way we can alter the location where these are created so that we don't fill the root filesystem?
</customer>

Version-Release number of selected component (if applicable):
Satellite 5.4.1

How reproducible:
Always when generating a heapdump with taskomatic (or rhn-search).

Steps to Reproduce:
1. service taskomatic dump  or service rhn-search dump
2. dump files are written to /usr/sbin
  
Actual results:
Heapdumps are written to /usr/sbin, which isn't really an appropriate place and can cause the root filesystem to fill up.

Expected results:
Have the dump files written to a more appropriate place, eg somewhere in /var, like /var/crash.

Additional info:
Tomcat also tries to write heapdumps to /usr/share/tomcatX (permission denied) and then /tmp.  Perhaps it could be changed too to write heapdumps to somewhere in /var.

--- Additional comment from mhuth on 2012-03-28 22:01:53 EDT ---

There are 3 apps that use JVMs in satellite - taskomatic, rhn-search and tomcat.

taskomatic and rhn-search can easily be made to write heapdumps to somewhere like /var/crash like so:

For taskomatic, make the following changes to /etc/rhn/default/rhn_taskomatic_daemon.conf (in the Additional Parameters section):
wrapper.java.additional.2=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
wrapper.java.additional.3=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt

For rhn-search, make the following changes to /etc/rhn/search/rhn_search_daemon.conf (in the Additional Parameters section):
wrapper.java.additional.1=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
wrapper.java.additional.2=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt


For tomcat, it runs as the tomcat user, so can't write to /var/crash by default because its owned by root:root.  And by default, tomcat will try to write to /usr/share/tomcatX by default, get permission denied, and then try to write to /tmp ... and /tmp isn't so bad a location for the dump files.  However, perhaps the following commented out lines could be added to the /etc/tomcatX/tomcatX.conf file to make it easier for a user to change the location where tomcat writes dump files:

# Set heapdump location.  The specified directory will need to write privileges for the tomcat user and/or group
# JAVA_OPTS="$JAVA_OPTS -Xdump:heap:file=/var/heapdumps/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd -Xdump:java:file=/var/heapdumps/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt"

Cheers,
Mark