Bug 816299 - Change default java heapdump directory for taskomatic (and other JVMs) from /usr/sbin to somewhere in /var
Summary: Change default java heapdump directory for taskomatic (and other JVMs) from /...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Spacewalk
Classification: Community
Component: Server
Version: 1.8
Hardware: All
OS: All
high
high
Target Milestone: ---
Assignee: Stephen Herr
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks: space18
TreeView+ depends on / blocked
 
Reported: 2012-04-25 18:37 UTC by Stephen Herr
Modified: 2012-11-01 16:18 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 807875
Environment:
Last Closed: 2012-11-01 16:18:35 UTC


Attachments (Terms of Use)

Description Stephen Herr 2012-04-25 18:37:24 UTC
+++ This bug was initially created as a clone of Bug #807875 +++

Description of problem:
<customer>
Recently, the root volume of our satellite server filled to 100%.  As it turns out, it was filled by numerous JVM heap dumps.  I found the following KB article that seems to match our symptoms:

https://access.redhat.com/knowledge/solutions/43122

Also, and perhaps more importantly, the heap dumps are ending up in /usr/sbin, which seems like an inappropriate place to put them -- somewhere in /var seems like it would be a more standard place.  Is there a way we can alter the location where these are created so that we don't fill the root filesystem?
</customer>

Version-Release number of selected component (if applicable):
Satellite 5.4.1

How reproducible:
Always when generating a heapdump with taskomatic (or rhn-search).

Steps to Reproduce:
1. service taskomatic dump  or service rhn-search dump
2. dump files are written to /usr/sbin
  
Actual results:
Heapdumps are written to /usr/sbin, which isn't really an appropriate place and can cause the root filesystem to fill up.

Expected results:
Have the dump files written to a more appropriate place, eg somewhere in /var, like /var/crash.

Additional info:
Tomcat also tries to write heapdumps to /usr/share/tomcatX (permission denied) and then /tmp.  Perhaps it could be changed too to write heapdumps to somewhere in /var.

--- Additional comment from mhuth@redhat.com on 2012-03-28 22:01:53 EDT ---

There are 3 apps that use JVMs in satellite - taskomatic, rhn-search and tomcat.

taskomatic and rhn-search can easily be made to write heapdumps to somewhere like /var/crash like so:

For taskomatic, make the following changes to /etc/rhn/default/rhn_taskomatic_daemon.conf (in the Additional Parameters section):
wrapper.java.additional.2=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
wrapper.java.additional.3=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt

For rhn-search, make the following changes to /etc/rhn/search/rhn_search_daemon.conf (in the Additional Parameters section):
wrapper.java.additional.1=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
wrapper.java.additional.2=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt


For tomcat, it runs as the tomcat user, so can't write to /var/crash by default because its owned by root:root.  And by default, tomcat will try to write to /usr/share/tomcatX by default, get permission denied, and then try to write to /tmp ... and /tmp isn't so bad a location for the dump files.  However, perhaps the following commented out lines could be added to the /etc/tomcatX/tomcatX.conf file to make it easier for a user to change the location where tomcat writes dump files:

# Set heapdump location.  The specified directory will need to write privileges for the tomcat user and/or group
# JAVA_OPTS="$JAVA_OPTS -Xdump:heap:file=/var/heapdumps/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd -Xdump:java:file=/var/heapdumps/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt"

Cheers,
Mark

Comment 1 Stephen Herr 2012-04-25 18:40:16 UTC
Note that this is only possible (and the patch only works for) IBM JVMs. If spacewalk users who are not running on an IBM JVM attempt to use these config options their JVMs will likely fail to start. 

I will commit the changes to the default config file, but will leave them commented out with a note.

Comment 2 Stephen Herr 2012-04-25 19:18:18 UTC
Committed to Spacewalk master as 7853915c3960caa81366ea2e43b79212cfafe362

Comment 4 Jan Pazdziora 2012-10-30 19:23:22 UTC
Moving ON_QA. Packages that address this bugzilla should now be available in yum repos at http://yum.spacewalkproject.org/nightly/

Comment 5 Jan Pazdziora 2012-11-01 16:18:35 UTC
Spacewalk 1.8 has been released: https://fedorahosted.org/spacewalk/wiki/ReleaseNotes18


Note You need to log in before you can comment on or make changes to this bug.