Description of problem: tmpwatch is deleting /tmp/hsperfdata_${user} of started java-processes that are older than 10 days. jps and jstat are completely useless afterwards since all performance data is stored in a PID-named file in this folder. Version-Release number of selected component (if applicable): tmpwatch-2.9.7-1.1.el5.2 How reproducible: Start a java process, wait 11 days and use jps or jstat. Alternatively one could change the day to 11 days later and wait for the daily cronjob-run. Steps to Reproduce: 1. start a java process 2. wait 11 days 3. jps for the process or try a jstat on the started process with e.g. 3.1 ps -ef|grep java|grep -v grep jboss 29945 29936 0 Sep18 ? 01:56:36 java -Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxNewSize=768M -XX:NewSize=768M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:ThreadStackSize=320 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jar org.jboss.Main -b 192.168.1.13 -c ocsp3 jboss 30940 30931 0 Sep18 ? 01:53:09 java -Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxNewSize=768M -XX:NewSize=768M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:ThreadStackSize=320 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jar org.jboss.Main -b 192.168.1.14 -c ocsp4 3.2 jstat -gc -t 29945 1"s" 1 29945 not found 3.3 jps 17882 Jps Actual results: The java process can't be found via jps or jstat Expected results: The java process is found, statusinformation for this process are available Additional info: /etc/cron.daily/tmpwatch musst be altered to include /tmp/hsperfdata* when ignoring files to delete since the PID-named file in /tmp/hsperfdata_${user}/${PID} has atime of the last time the process was started. # ps -ef|grep java|grep -v grep tomcat 29221 1 0 Oct05 ? 00:09:41 /usr/lib/jvm/java/bin/java -server -Xmx512M -Xms512M -XX:+UseConcMarkSweepGC -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar:/usr/share/java/mx4j/mx4j-impl.jar:/usr/share/java/mx4j/mx4j-jmx.jar -Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start # ls -l /tmp/hsperfdata_tomcat/29221 -rw------- 1 tomcat tomcat 32768 Oct 5 11:57 /tmp/hsperfdata_tomcat/29221 # ls -l --time=atime /tmp/hsperfdata_tomcat/29221 -rw------- 1 tomcat tomcat 32768 Oct 6 12:47 /tmp/hsperfdata_tomcat/29221 Since this file does not grow over the runtime of a java process the ignoring of this directory is justified. Here a jboss on a Solaris 9-server: # ps -ef|grep java|grep -v grep jboss 18265 18254 1 Jun 30 ? 27240:06 /opt/java/bin/java -Dprogram.name=run.sh -Xms256m -Xmx512m -Xss256k -server -XX # du -hs /tmp/hsperfdata_jboss/* 32K /tmp/hsperfdata_jboss/18265 # ls -l /tmp/hsperfdata_jboss/ total 64 -rw------- 1 jboss jboss 32768 Jun 30 14:18 18265 # jstat -gc -h20 -t 18265 1"s" 1 Timestamp S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT 8463812.7 10944.0 10688.0 0.0 0.0 125824.0 7448.1 352256.0 145523.1 49152.0 46602.8 260070 16229.205 134603 377413.259 393642.464
Thanks for your report. If you are a RHEL customer and have an active support entitlement, please contact official Red Hat Support at https://www.redhat.com/apps/support/ to allow correct prioritization of this issue.
duly executed. I filed the ticket with my private account since at work there is no routine present for handling ticket creation with Red Hat Support. But the problem is more severe on the servers at work. But not live threatening overall, i helped myself with the attached patch. /tmp/hsperfdata_* is not working though, tmpwatch does not accept this as it seems.
Created attachment 363875 [details] patch for tmpwatch to ignore hsperfdata_jboss only works for the jboss-user and the tomcat-user since tmpwatch does not accept wildcards as it seems to me.
Hm on the official Red Hat Support they are passing the buck over to Java. This is a philosophical problem: Is tmpwatch to blame deleting content in /tmp older than ten days or is java to blame creating content there that could look like it is older than ten days since last usage in the first place. Since tmpwatch has exceptions in /etc/cron.daily/tmpwatch for some applications i think it could be updated to ignore the content java created too...
Well, Java should probably create the files in /var/run or use POSIX named shared memory in /dev/shm instead of using /tmp - but that can't be helped now. In any case, tmpwatch depends on atime/mtime of a file, which can't detect mmap(), so tmpwatch needs to compensate. Fix built in Fedora tmpwatch-2.9.16-1.fc13.
Created attachment 364939 [details] Upstream patch
Quality Engineering Management has reviewed and declined this request. You may appeal this decision by reopening this request.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: tmpwatch automatically deletes temporary directories, which used to break some Java management functionality. This issue has been resolved and the newly added option --exclude-pattern is now used to preserve these directories.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0619.html