Bug 1483503
Summary: | JAVA core dumps occupy disk space in /usr/sbin | ||
---|---|---|---|
Product: | Red Hat Satellite 5 | Reporter: | Pavel Studeník <pstudeni> |
Component: | Server | Assignee: | Michael Mráka <mmraka> |
Status: | CLOSED ERRATA | QA Contact: | Radovan Drazny <rdrazny> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 580 | CC: | cdonnell, dyordano, mmraka, rdrazny, shughes, tkasparek, tlestach |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | nutch-1.0-0.17.20081201040121nightly-sat spacewalk-java-2.5.14-106-sat spacewalk-setup-2.5.1-23-sat spacewalk-search-2.5.1-5-sat | Doc Type: | If docs needed, set a value |
Doc Text: |
Solution:
Tomcat configuration have to be updated by running
spacewalk-setup-tomcat
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2017-12-13 07:59:26 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1450111 |
Description
Pavel Studeník
2017-08-21 09:52:08 UTC
It looks that I can see reproducer in taskomatic log.
>> tail /var/log/rhn/rhn_taskomatic_daemon.log
INFO | jvm 1 | 2017/09/15 11:21:20 | 2017-09-15 11:21:20,106 [Thread-59] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating new repository metadata for channel 'rhel-x86_64-server-6'(sh
a256) 19705 packages, 4079 errata
INFO | jvm 1 | 2017/09/15 11:22:00 | 2017-09-15 11:22:00,103 [DefaultQuartzScheduler_Worker-4] WARN com.redhat.rhn.taskomatic.task.ChannelRepodata - Maximum number of workers already put ... skipping.
INFO | jvm 1 | 2017/09/15 11:22:00 | 2017-09-15 11:22:00,158 [DefaultQuartzScheduler_Worker-1] INFO com.redhat.rhn.taskomatic.task.ErrataQueue - In the queue: 1756
INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/09/15 11:22:29 - please wait.
INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/09/15 11:22:29 - please wait.
INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/09/15 11:22:29 - please wait.
INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP032I JVM requested System dump using '/usr/sbin/core.20170915.112229.30907.0001.dmp' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:00 | JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" specifies that the core dump is to be piped to an external program. A
ttempting to rename either core or core.31428.
INFO | jvm 1 | 2017/09/15 11:23:00 |
INFO | jvm 1 | 2017/09/15 11:23:00 | JVMDUMP010I System dump written to /usr/sbin/core.20170915.112229.30907.0001.dmp
INFO | jvm 1 | 2017/09/15 11:23:00 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20170915.112229.30907.0002.phd' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:01 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20170915.112229.30907.0002.phd
INFO | jvm 1 | 2017/09/15 11:23:01 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20170915.112229.30907.0003.phd' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:02 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20170915.112229.30907.0003.phd
INFO | jvm 1 | 2017/09/15 11:23:02 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20170915.112229.30907.0004.phd' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20170915.112229.30907.0004.phd
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20170915.112229.30907.0005.txt' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20170915.112229.30907.0005.txt
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20170915.112229.30907.0008.trc' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20170915.112229.30907.0008.trc
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20170915.112229.30907.0006.txt' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20170915.112229.30907.0006.txt
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20170915.112229.30907.0009.trc' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Snap dump written to {nothing to snap}
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20170915.112229.30907.0007.txt' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20170915.112229.30907.0007.txt
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20170915.112229.30907.0010.trc' in response to an event
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20170915.112229.30907.0010.trc
INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
INFO | jvm 1 | 2017/09/15 11:23:03 | Exception in thread "Thread-92" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2017/09/15 11:23:03 | at java.lang.Class.getConstructorImpl(Native Method)
INFO | jvm 1 | 2017/09/15 11:23:03 | at java.lang.Class.getConstructor(Class.java:541)
Reproducer: # check running java processes by ps -fC java # output should be similar to this (for running tomcat, taskomaticd and rhnsearchd) #UID PID PPID C STIME TTY TIME CMD #tomcat 5354 1 0 Sep20 ? 00:02:00 /usr/lib/jvm/java/bin/java -ea -Xms256m -Xmx256m -Djava.aw #root 5730 5722 0 Sep20 ? 00:03:13 /usr/bin/java -Dibm.dst.compatibility=true -Dcom.ibm.tools #root 26787 26785 1 04:48 ? 00:00:16 /usr/bin/java -Dcom.ibm.tools.attach.enable=no -Djava.libr # send SIGQUIT to processes pkill -SIGQUIT java # locate javacore files updatedb locate javacore # javacore has been created in /usr/bin for taskomatic and rhnsearchd and in /usr/share/tomcat6 for tomcat #/usr/sbin/javacore.20170921.050341.26787.0003.txt #/usr/sbin/javacore.20170921.050341.5730.0001.txt #/usr/share/tomcat6/javacore.20170921.050341.5354.0001.txt Fixed in spacewalk upstream git by commit efa585e7f9fdd450c16eaf8e8d800a28270c308c 1483503 - disable ibm java coredumps for tomcat commit 74a784e40f75f10c493b2f17412681e71ca60b7a 1483503 - disable ibm java coredumps in tanukiwrapper On Spacewalk Nightly I can see still log in sbin directory spacewalk-java-2.8.23-1.el6.noarch spacewalk-search-2.8.2-1.el6.noarch # tail /usr/sbin/hadoop.log.2017-10-04 ... 2017-10-04 23:55:43,550 INFO tasks.GenericIndexTask - Deleted 0 records from index <serverCustomInfo> 2017-10-04 23:55:43,553 INFO tasks.GenericIndexTask - Deleted 0 records from index <xccdfIdent> 2017-10-04 23:55:43,584 INFO tasks.GenericIndexTask - Deleted 0 records from index <hwdevice> 2017-10-04 23:55:43,648 INFO tasks.GenericIndexTask - GenericIndexTask<class com.redhat.satellite.search.scheduler.tasks.IndexSystemsTask number of results returned = 0 2017-10-04 23:55:43,652 INFO tasks.GenericIndexTask - class com.redhat.satellite.search.scheduler.tasks.IndexSystemsTaskfound [0] items to index 2017-10-04 23:55:43,654 INFO index.IndexManager - IndexManager::getIndexReader(server, en_US) path = /var/lib/rhn/search/indexes/server 2017-10-04 23:55:43,659 INFO tasks.GenericIndexTask - Deleted 0 records from index <server> hadoop.log is a bit different issue but definitely it should not end in /usr/sbin as well. Fixed in spacewalk master by commit d4ad44b2190c97f1a461f5cb0a71652d3648843d 1483503 - move hadoop logs to /var/log Reproducer: # service rhn-search restart Reproduced on: spacewalk-java-2.5.14-101 spacewalk-setup-2.5.1-22 spacewalk-search-2.5.1-4 nutch-1.0-0.16.20081201040121nightly using the reproducer from comments #8 and #12. Updated to: spacewalk-java-2.5.14-106 spacewalk-setup-2.5.1-23 spacewalk-search-2.5.1-5 nutch-1.0-0.17.20081201040121nightly In one terminal: $ inotifywatch -v -e create -r /lib/ /sbin/ /bin/ /usr/ In another terminal: $ pkill -SIGQUIT java $ service rhn-search restart Stopping rhn-search... Stopped rhn-search. Starting rhn-search... Ctrl+C in the first terminal: total create filename 2 2 /usr/share/tomcat6/ $ ls /usr/share/tomcat6/ bin conf javacore.20171121.122010.48904.0006.txt lib logs temp webapps work A javacore file is still created in the /usr/share/tomcat6 directory FailedQA As a part of fix tomcat configuration has to be updated by running spacewalk-setup-tomcat After running the spacewalk-setup-tomcat script as described in the comment #17, javacore files are no longer dumped into the /usr/share/tomcat6/ directory. Tested using the procedure from the comment #16. VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3445 |