Bug 816299 - Change default java heapdump directory for taskomatic (and other JVMs) from /usr/sbin to somewhere in /var
Change default java heapdump directory for taskomatic (and other JVMs) from /...
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Server (Show other bugs)
1.8
All All
high Severity high
: ---
: ---
Assigned To: Stephen Herr
Red Hat Satellite QA List
:
Depends On:
Blocks: space18
  Show dependency treegraph
 
Reported: 2012-04-25 14:37 EDT by Stephen Herr
Modified: 2012-11-01 12:18 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 807875
Environment:
Last Closed: 2012-11-01 12:18:35 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Stephen Herr 2012-04-25 14:37:24 EDT
+++ This bug was initially created as a clone of Bug #807875 +++

Description of problem:
<customer>
Recently, the root volume of our satellite server filled to 100%.  As it turns out, it was filled by numerous JVM heap dumps.  I found the following KB article that seems to match our symptoms:

https://access.redhat.com/knowledge/solutions/43122

Also, and perhaps more importantly, the heap dumps are ending up in /usr/sbin, which seems like an inappropriate place to put them -- somewhere in /var seems like it would be a more standard place.  Is there a way we can alter the location where these are created so that we don't fill the root filesystem?
</customer>

Version-Release number of selected component (if applicable):
Satellite 5.4.1

How reproducible:
Always when generating a heapdump with taskomatic (or rhn-search).

Steps to Reproduce:
1. service taskomatic dump  or service rhn-search dump
2. dump files are written to /usr/sbin
  
Actual results:
Heapdumps are written to /usr/sbin, which isn't really an appropriate place and can cause the root filesystem to fill up.

Expected results:
Have the dump files written to a more appropriate place, eg somewhere in /var, like /var/crash.

Additional info:
Tomcat also tries to write heapdumps to /usr/share/tomcatX (permission denied) and then /tmp.  Perhaps it could be changed too to write heapdumps to somewhere in /var.

--- Additional comment from mhuth@redhat.com on 2012-03-28 22:01:53 EDT ---

There are 3 apps that use JVMs in satellite - taskomatic, rhn-search and tomcat.

taskomatic and rhn-search can easily be made to write heapdumps to somewhere like /var/crash like so:

For taskomatic, make the following changes to /etc/rhn/default/rhn_taskomatic_daemon.conf (in the Additional Parameters section):
wrapper.java.additional.2=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
wrapper.java.additional.3=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt

For rhn-search, make the following changes to /etc/rhn/search/rhn_search_daemon.conf (in the Additional Parameters section):
wrapper.java.additional.1=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
wrapper.java.additional.2=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt


For tomcat, it runs as the tomcat user, so can't write to /var/crash by default because its owned by root:root.  And by default, tomcat will try to write to /usr/share/tomcatX by default, get permission denied, and then try to write to /tmp ... and /tmp isn't so bad a location for the dump files.  However, perhaps the following commented out lines could be added to the /etc/tomcatX/tomcatX.conf file to make it easier for a user to change the location where tomcat writes dump files:

# Set heapdump location.  The specified directory will need to write privileges for the tomcat user and/or group
# JAVA_OPTS="$JAVA_OPTS -Xdump:heap:file=/var/heapdumps/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd -Xdump:java:file=/var/heapdumps/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt"

Cheers,
Mark
Comment 1 Stephen Herr 2012-04-25 14:40:16 EDT
Note that this is only possible (and the patch only works for) IBM JVMs. If spacewalk users who are not running on an IBM JVM attempt to use these config options their JVMs will likely fail to start. 

I will commit the changes to the default config file, but will leave them commented out with a note.
Comment 2 Stephen Herr 2012-04-25 15:18:18 EDT
Committed to Spacewalk master as 7853915c3960caa81366ea2e43b79212cfafe362
Comment 4 Jan Pazdziora 2012-10-30 15:23:22 EDT
Moving ON_QA. Packages that address this bugzilla should now be available in yum repos at http://yum.spacewalkproject.org/nightly/
Comment 5 Jan Pazdziora 2012-11-01 12:18:35 EDT
Spacewalk 1.8 has been released: https://fedorahosted.org/spacewalk/wiki/ReleaseNotes18

Note You need to log in before you can comment on or make changes to this bug.