Bug 527425 - /tmp/hsperfdata_${user}/${PID} is deleted by tmpwatch
Summary: /tmp/hsperfdata_${user}/${PID} is deleted by tmpwatch
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: tmpwatch
Version: 5.4
Hardware: All
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Miloslav Trmač
QA Contact: Zbysek MRAZ
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-10-06 11:22 UTC by Gerrit Slomma
Modified: 2018-06-22 13:27 UTC (History)
4 users (show)

Fixed In Version: tmpwatch-2.9.7-1.1.el5.5
Doc Type: Bug Fix
Doc Text:
tmpwatch automatically deletes temporary directories, which used to break some Java management functionality. This issue has been resolved and the newly added option --exclude-pattern is now used to preserve these directories.
Clone Of:
Environment:
Last Closed: 2010-08-11 09:13:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
patch for tmpwatch to ignore hsperfdata_jboss (534 bytes, patch)
2009-10-06 18:30 UTC, Gerrit Slomma
no flags Details | Diff
Upstream patch (3.73 KB, patch)
2009-10-15 15:21 UTC, Miloslav Trmač
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0619 0 normal SHIPPED_LIVE tmpwatch bug fix update 2010-08-11 09:13:16 UTC

Description Gerrit Slomma 2009-10-06 11:22:47 UTC
Description of problem:

tmpwatch is deleting /tmp/hsperfdata_${user} of started java-processes that are older than 10 days. jps and jstat are completely useless afterwards since all performance data is stored in a PID-named file in this folder.

Version-Release number of selected component (if applicable):

tmpwatch-2.9.7-1.1.el5.2

How reproducible:

Start a java process, wait 11 days and use jps or jstat.
Alternatively one could change the day to 11 days later and wait for the daily cronjob-run.

Steps to Reproduce:
1. start a java process
2. wait 11 days
3. jps for the process or try a jstat on the started process with e.g.
3.1 ps -ef|grep java|grep -v grep
jboss    29945 29936  0 Sep18 ?        01:56:36 java -Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxNewSize=768M -XX:NewSize=768M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:ThreadStackSize=320 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jar org.jboss.Main -b 192.168.1.13 -c ocsp3
jboss    30940 30931  0 Sep18 ?        01:53:09 java -Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxNewSize=768M -XX:NewSize=768M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:ThreadStackSize=320 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath /opt/jboss/bin/run.jar org.jboss.Main -b 192.168.1.14 -c ocsp4
3.2 jstat -gc -t 29945 1"s" 1
29945 not found
3.3 jps
17882 Jps

Actual results:

The java process can't be found via jps or jstat

Expected results:

The java process is found, statusinformation for this process are available

Additional info:

/etc/cron.daily/tmpwatch musst be altered to include /tmp/hsperfdata* when ignoring files to delete since the PID-named file in /tmp/hsperfdata_${user}/${PID} has atime of the last time the process was started.

# ps -ef|grep java|grep -v grep
tomcat   29221     1  0 Oct05 ?        00:09:41 /usr/lib/jvm/java/bin/java -server -Xmx512M -Xms512M -XX:+UseConcMarkSweepGC -Dcatalina.ext.dirs=/usr/share/tomcat5/shared/lib:/usr/share/tomcat5/common/lib -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/jvm/java/lib/tools.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar:/usr/share/java/mx4j/mx4j-impl.jar:/usr/share/java/mx4j/mx4j-jmx.jar -Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start
# ls -l /tmp/hsperfdata_tomcat/29221
-rw------- 1 tomcat tomcat 32768 Oct  5 11:57 /tmp/hsperfdata_tomcat/29221
# ls -l --time=atime /tmp/hsperfdata_tomcat/29221
-rw------- 1 tomcat tomcat 32768 Oct  6 12:47 /tmp/hsperfdata_tomcat/29221

Since this file does not grow over the runtime of a java process the ignoring of this directory is justified. Here a jboss on a Solaris 9-server:

# ps -ef|grep java|grep -v grep
   jboss 18265 18254  1   Jun 30 ?       27240:06 /opt/java/bin/java -Dprogram.name=run.sh -Xms256m -Xmx512m -Xss256k -server -XX
# du -hs /tmp/hsperfdata_jboss/*
  32K   /tmp/hsperfdata_jboss/18265
# ls -l /tmp/hsperfdata_jboss/
total 64
-rw-------   1 jboss    jboss      32768 Jun 30 14:18 18265 
# jstat -gc -h20 -t 18265 1"s" 1
Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
      8463812.7 10944.0 10688.0  0.0    0.0   125824.0  7448.1   352256.0   145523.1  49152.0 46602.8 260070 16229.205 134603 377413.259 393642.464

Comment 1 Miloslav Trmač 2009-10-06 17:41:26 UTC
Thanks for your report.

If you are a RHEL customer and have an active support entitlement, please
contact official Red Hat Support at https://www.redhat.com/apps/support/ to
allow correct prioritization of this issue.

Comment 2 Gerrit Slomma 2009-10-06 18:26:53 UTC
duly executed.
I filed the ticket with my private account since at work there is no routine present for handling ticket creation with Red Hat Support. But the problem is more severe on the servers at work. But not live threatening overall, i helped myself with the attached patch. /tmp/hsperfdata_* is not working though, tmpwatch does not accept this as it seems.

Comment 3 Gerrit Slomma 2009-10-06 18:30:49 UTC
Created attachment 363875 [details]
patch for tmpwatch to ignore hsperfdata_jboss

only works for the jboss-user and the tomcat-user since tmpwatch does not accept wildcards as it seems to me.

Comment 6 Gerrit Slomma 2009-10-13 09:43:06 UTC
Hm on the official Red Hat Support they are passing the buck over to Java.
This is a philosophical problem: Is tmpwatch to blame deleting content in /tmp older than ten days or is java to blame creating content there that could look like it is older than ten days since last usage in the first place. Since tmpwatch has exceptions in /etc/cron.daily/tmpwatch for some applications i think it could be updated to ignore the content java created too...

Comment 7 Miloslav Trmač 2009-10-15 15:00:00 UTC
Well, Java should probably create the files in /var/run or use POSIX named shared memory in /dev/shm instead of using /tmp - but that can't be helped now.

In any case, tmpwatch depends on atime/mtime of a file, which can't detect mmap(), so tmpwatch needs to compensate.  Fix built in Fedora tmpwatch-2.9.16-1.fc13.

Comment 8 Miloslav Trmač 2009-10-15 15:21:55 UTC
Created attachment 364939 [details]
Upstream patch

Comment 10 RHEL Program Management 2009-10-26 15:39:53 UTC
Quality Engineering Management has reviewed and declined this request.  You may
appeal this decision by reopening this request.

Comment 19 Florian Nadge 2010-07-14 10:21:02 UTC
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
tmpwatch automatically deletes temporary directories, which used to break some Java management functionality. This issue has been resolved and the newly added option --exclude-pattern is now used to preserve these directories.

Comment 24 errata-xmlrpc 2010-08-11 09:13:24 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0619.html


Note You need to log in before you can comment on or make changes to this bug.