Description of problem: I look at my long term running stable satellite and I can see that /usr/sbin occupies 60G # du -sh /usr/sbin/ 60G /usr/sbin/ I found many big files in the directory: ... 2,5G /usr/sbin/core.20170623.013422.2945.0002.dmp 1,7G /usr/sbin/core.20170629.150723.19613.0002.dmp 2,1G /usr/sbin/core.20170712.013229.30598.0001.dmp 1,6G /usr/sbin/core.20170713.150636.16320.0001.dmp 3,6G /usr/sbin/core.20170810.013525.13414.0001.dmp It looks that the files are ibm java core dump. I am sure that this directory "sbin" is not good path for java dumps. Version-Release number of selected component (if applicable): java-1.8.0-ibm-1.8.0.4.1-1jpp.1.el6_8.x86_64 spacewalk-java-2.5.14-91.el6sat.noarch How reproducible: sometimes for long time Actual results: java core dumps in /usr/sbin Expected results: Better path for storage dumps - for example: /var/crash/java ... Additional info: I found some java dumps on Satellite 5.7, but not so many.
It looks that I can see reproducer in taskomatic log. >> tail /var/log/rhn/rhn_taskomatic_daemon.log INFO | jvm 1 | 2017/09/15 11:21:20 | 2017-09-15 11:21:20,106 [Thread-59] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating new repository metadata for channel 'rhel-x86_64-server-6'(sh a256) 19705 packages, 4079 errata INFO | jvm 1 | 2017/09/15 11:22:00 | 2017-09-15 11:22:00,103 [DefaultQuartzScheduler_Worker-4] WARN com.redhat.rhn.taskomatic.task.ChannelRepodata - Maximum number of workers already put ... skipping. INFO | jvm 1 | 2017/09/15 11:22:00 | 2017-09-15 11:22:00,158 [DefaultQuartzScheduler_Worker-1] INFO com.redhat.rhn.taskomatic.task.ErrataQueue - In the queue: 1756 INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/09/15 11:22:29 - please wait. INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/09/15 11:22:29 - please wait. INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/09/15 11:22:29 - please wait. INFO | jvm 1 | 2017/09/15 11:22:29 | JVMDUMP032I JVM requested System dump using '/usr/sbin/core.20170915.112229.30907.0001.dmp' in response to an event INFO | jvm 1 | 2017/09/15 11:23:00 | JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" specifies that the core dump is to be piped to an external program. A ttempting to rename either core or core.31428. INFO | jvm 1 | 2017/09/15 11:23:00 | INFO | jvm 1 | 2017/09/15 11:23:00 | JVMDUMP010I System dump written to /usr/sbin/core.20170915.112229.30907.0001.dmp INFO | jvm 1 | 2017/09/15 11:23:00 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20170915.112229.30907.0002.phd' in response to an event INFO | jvm 1 | 2017/09/15 11:23:01 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20170915.112229.30907.0002.phd INFO | jvm 1 | 2017/09/15 11:23:01 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20170915.112229.30907.0003.phd' in response to an event INFO | jvm 1 | 2017/09/15 11:23:02 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20170915.112229.30907.0003.phd INFO | jvm 1 | 2017/09/15 11:23:02 | JVMDUMP032I JVM requested Heap dump using '/usr/sbin/heapdump.20170915.112229.30907.0004.phd' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Heap dump written to /usr/sbin/heapdump.20170915.112229.30907.0004.phd INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20170915.112229.30907.0005.txt' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20170915.112229.30907.0005.txt INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20170915.112229.30907.0008.trc' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20170915.112229.30907.0008.trc INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20170915.112229.30907.0006.txt' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20170915.112229.30907.0006.txt INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20170915.112229.30907.0009.trc' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Snap dump written to {nothing to snap} INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Java dump using '/usr/sbin/javacore.20170915.112229.30907.0007.txt' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Java dump written to /usr/sbin/javacore.20170915.112229.30907.0007.txt INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP032I JVM requested Snap dump using '/usr/sbin/Snap.20170915.112229.30907.0010.trc' in response to an event INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP010I Snap dump written to /usr/sbin/Snap.20170915.112229.30907.0010.trc INFO | jvm 1 | 2017/09/15 11:23:03 | JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". INFO | jvm 1 | 2017/09/15 11:23:03 | Exception in thread "Thread-92" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2017/09/15 11:23:03 | at java.lang.Class.getConstructorImpl(Native Method) INFO | jvm 1 | 2017/09/15 11:23:03 | at java.lang.Class.getConstructor(Class.java:541)
Reproducer: # check running java processes by ps -fC java # output should be similar to this (for running tomcat, taskomaticd and rhnsearchd) #UID PID PPID C STIME TTY TIME CMD #tomcat 5354 1 0 Sep20 ? 00:02:00 /usr/lib/jvm/java/bin/java -ea -Xms256m -Xmx256m -Djava.aw #root 5730 5722 0 Sep20 ? 00:03:13 /usr/bin/java -Dibm.dst.compatibility=true -Dcom.ibm.tools #root 26787 26785 1 04:48 ? 00:00:16 /usr/bin/java -Dcom.ibm.tools.attach.enable=no -Djava.libr # send SIGQUIT to processes pkill -SIGQUIT java # locate javacore files updatedb locate javacore # javacore has been created in /usr/bin for taskomatic and rhnsearchd and in /usr/share/tomcat6 for tomcat #/usr/sbin/javacore.20170921.050341.26787.0003.txt #/usr/sbin/javacore.20170921.050341.5730.0001.txt #/usr/share/tomcat6/javacore.20170921.050341.5354.0001.txt
Fixed in spacewalk upstream git by commit efa585e7f9fdd450c16eaf8e8d800a28270c308c 1483503 - disable ibm java coredumps for tomcat commit 74a784e40f75f10c493b2f17412681e71ca60b7a 1483503 - disable ibm java coredumps in tanukiwrapper
On Spacewalk Nightly I can see still log in sbin directory spacewalk-java-2.8.23-1.el6.noarch spacewalk-search-2.8.2-1.el6.noarch # tail /usr/sbin/hadoop.log.2017-10-04 ... 2017-10-04 23:55:43,550 INFO tasks.GenericIndexTask - Deleted 0 records from index <serverCustomInfo> 2017-10-04 23:55:43,553 INFO tasks.GenericIndexTask - Deleted 0 records from index <xccdfIdent> 2017-10-04 23:55:43,584 INFO tasks.GenericIndexTask - Deleted 0 records from index <hwdevice> 2017-10-04 23:55:43,648 INFO tasks.GenericIndexTask - GenericIndexTask<class com.redhat.satellite.search.scheduler.tasks.IndexSystemsTask number of results returned = 0 2017-10-04 23:55:43,652 INFO tasks.GenericIndexTask - class com.redhat.satellite.search.scheduler.tasks.IndexSystemsTaskfound [0] items to index 2017-10-04 23:55:43,654 INFO index.IndexManager - IndexManager::getIndexReader(server, en_US) path = /var/lib/rhn/search/indexes/server 2017-10-04 23:55:43,659 INFO tasks.GenericIndexTask - Deleted 0 records from index <server>
hadoop.log is a bit different issue but definitely it should not end in /usr/sbin as well. Fixed in spacewalk master by commit d4ad44b2190c97f1a461f5cb0a71652d3648843d 1483503 - move hadoop logs to /var/log Reproducer: # service rhn-search restart
Reproduced on: spacewalk-java-2.5.14-101 spacewalk-setup-2.5.1-22 spacewalk-search-2.5.1-4 nutch-1.0-0.16.20081201040121nightly using the reproducer from comments #8 and #12. Updated to: spacewalk-java-2.5.14-106 spacewalk-setup-2.5.1-23 spacewalk-search-2.5.1-5 nutch-1.0-0.17.20081201040121nightly In one terminal: $ inotifywatch -v -e create -r /lib/ /sbin/ /bin/ /usr/ In another terminal: $ pkill -SIGQUIT java $ service rhn-search restart Stopping rhn-search... Stopped rhn-search. Starting rhn-search... Ctrl+C in the first terminal: total create filename 2 2 /usr/share/tomcat6/ $ ls /usr/share/tomcat6/ bin conf javacore.20171121.122010.48904.0006.txt lib logs temp webapps work A javacore file is still created in the /usr/share/tomcat6 directory FailedQA
As a part of fix tomcat configuration has to be updated by running spacewalk-setup-tomcat
After running the spacewalk-setup-tomcat script as described in the comment #17, javacore files are no longer dumped into the /usr/share/tomcat6/ directory. Tested using the procedure from the comment #16. VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3445