+++ This bug was initially created as a clone of Bug #807875 +++ Description of problem: <customer> Recently, the root volume of our satellite server filled to 100%. As it turns out, it was filled by numerous JVM heap dumps. I found the following KB article that seems to match our symptoms: https://access.redhat.com/knowledge/solutions/43122 Also, and perhaps more importantly, the heap dumps are ending up in /usr/sbin, which seems like an inappropriate place to put them -- somewhere in /var seems like it would be a more standard place. Is there a way we can alter the location where these are created so that we don't fill the root filesystem? </customer> Version-Release number of selected component (if applicable): Satellite 5.4.1 How reproducible: Always when generating a heapdump with taskomatic (or rhn-search). Steps to Reproduce: 1. service taskomatic dump or service rhn-search dump 2. dump files are written to /usr/sbin Actual results: Heapdumps are written to /usr/sbin, which isn't really an appropriate place and can cause the root filesystem to fill up. Expected results: Have the dump files written to a more appropriate place, eg somewhere in /var, like /var/crash. Additional info: Tomcat also tries to write heapdumps to /usr/share/tomcatX (permission denied) and then /tmp. Perhaps it could be changed too to write heapdumps to somewhere in /var. --- Additional comment from mhuth on 2012-03-28 22:01:53 EDT --- There are 3 apps that use JVMs in satellite - taskomatic, rhn-search and tomcat. taskomatic and rhn-search can easily be made to write heapdumps to somewhere like /var/crash like so: For taskomatic, make the following changes to /etc/rhn/default/rhn_taskomatic_daemon.conf (in the Additional Parameters section): wrapper.java.additional.2=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd wrapper.java.additional.3=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt For rhn-search, make the following changes to /etc/rhn/search/rhn_search_daemon.conf (in the Additional Parameters section): wrapper.java.additional.1=-Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd wrapper.java.additional.2=-Xdump:java:file=/var/crash/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt For tomcat, it runs as the tomcat user, so can't write to /var/crash by default because its owned by root:root. And by default, tomcat will try to write to /usr/share/tomcatX by default, get permission denied, and then try to write to /tmp ... and /tmp isn't so bad a location for the dump files. However, perhaps the following commented out lines could be added to the /etc/tomcatX/tomcatX.conf file to make it easier for a user to change the location where tomcat writes dump files: # Set heapdump location. The specified directory will need to write privileges for the tomcat user and/or group # JAVA_OPTS="$JAVA_OPTS -Xdump:heap:file=/var/heapdumps/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd -Xdump:java:file=/var/heapdumps/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt" Cheers, Mark
Note that this is only possible (and the patch only works for) IBM JVMs. If spacewalk users who are not running on an IBM JVM attempt to use these config options their JVMs will likely fail to start. I will commit the changes to the default config file, but will leave them commented out with a note.
Committed to Spacewalk master as 7853915c3960caa81366ea2e43b79212cfafe362
Moving ON_QA. Packages that address this bugzilla should now be available in yum repos at http://yum.spacewalkproject.org/nightly/
Spacewalk 1.8 has been released: https://fedorahosted.org/spacewalk/wiki/ReleaseNotes18