Bug 1051194 - [abrt] java-1.7.0-openjdk-headless: jni_FatalError(): java killed by SIGABRT
Summary: [abrt] java-1.7.0-openjdk-headless: jni_FatalError(): java killed by SIGABRT
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: tomcat
Version: 20
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ivan Afonichev
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:6104d80f0abda16e5f88d7e814d...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-09 20:11 UTC by Alex Wood
Modified: 2014-10-09 20:57 UTC (History)
10 users (show)

Fixed In Version: tomcat-7.0.52-1.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-09-26 09:02:23 UTC
Type: ---


Attachments (Terms of Use)
File: backtrace (41.46 KB, text/plain)
2014-01-09 20:11 UTC, Alex Wood
no flags Details
File: cgroup (165 bytes, text/plain)
2014-01-09 20:11 UTC, Alex Wood
no flags Details
File: core_backtrace (33.56 KB, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: dso_list (4.71 KB, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: environ (510 bytes, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: limits (1.29 KB, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: maps (22.93 KB, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: open_fds (784 bytes, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: proc_pid_status (921 bytes, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details
File: var_log_messages (17.98 KB, text/plain)
2014-01-09 20:12 UTC, Alex Wood
no flags Details

Description Alex Wood 2014-01-09 20:11:41 UTC
Description of problem:
tomcat-7.0.47-1.fc20.noarch running with CATALINA_OPTS set to "-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n"

Stopping the service with 'service tomcat stop' yields the reported error.

Version-Release number of selected component:
java-1.7.0-openjdk-headless-1.7.0.60-2.4.3.0.fc20

Additional info:
reporter:       libreport-2.1.10
backtrace_rating: 4
cmdline:        java -agentpath:/usr/lib64/libabrt-java-connector.so=abrt=on -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap stop
crash_function: jni_FatalError
executable:     /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/jre-abrt/bin/java
kernel:         3.12.6-300.fc20.x86_64
runlevel:       N 5
type:           CCpp
uid:            91

Truncated backtrace:
Thread no. 1 (10 frames)
 #3 jni_FatalError at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/prims/jni.cpp:870
 #4 jniFatalError at ../../../src/share/back/debugInit.c:671
 #5 debugInit_exit at ../../../src/share/back/debugInit.c:1334
 #6 initialize at ../../../src/share/back/debugInit.c:750
 #7 cbEarlyVMInit at ../../../src/share/back/debugInit.c:458
 #8 JvmtiExport::post_vm_initialized at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiExport.cpp:466
 #9 Threads::create_vm at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3635
 #10 JNI_CreateJavaVM at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/prims/jni.cpp:5128
 #11 InitializeJVM at ../../../src/share/bin/java.c:1120
 #12 JavaMain at ../../../src/share/bin/java.c:358

Comment 1 Alex Wood 2014-01-09 20:11:55 UTC
Created attachment 847799 [details]
File: backtrace

Comment 2 Alex Wood 2014-01-09 20:11:57 UTC
Created attachment 847800 [details]
File: cgroup

Comment 3 Alex Wood 2014-01-09 20:12:01 UTC
Created attachment 847801 [details]
File: core_backtrace

Comment 4 Alex Wood 2014-01-09 20:12:04 UTC
Created attachment 847802 [details]
File: dso_list

Comment 5 Alex Wood 2014-01-09 20:12:06 UTC
Created attachment 847803 [details]
File: environ

Comment 6 Alex Wood 2014-01-09 20:12:09 UTC
Created attachment 847804 [details]
File: limits

Comment 7 Alex Wood 2014-01-09 20:12:13 UTC
Created attachment 847805 [details]
File: maps

Comment 8 Alex Wood 2014-01-09 20:12:15 UTC
Created attachment 847806 [details]
File: open_fds

Comment 9 Alex Wood 2014-01-09 20:12:16 UTC
Created attachment 847807 [details]
File: proc_pid_status

Comment 10 Alex Wood 2014-01-09 20:12:18 UTC
Created attachment 847808 [details]
File: var_log_messages

Comment 11 Deepak Bhole 2014-01-20 22:23:57 UTC
Looks like there was an error during JDWP initialization. Is this reproducible? If so, can you kill all Java processes (or reboot) and try again?

Comment 12 Emmanuel Touzery 2014-01-21 07:25:01 UTC
Sadly I have no idea. I was using Netbeans and Glassfish 3.1.2.2 for development. Actually I didn't notice anything special (I think the computer load was a bit high but I don't recall the details), and at some point the fedora bug report tool notified me that Java had crashed and prompted me to report it. Honestly, had it not popped up, I would not have noticed. I'll try to be careful if it will happen again but it's definitely not reproducible easily and I have no idea how.

Comment 13 Alex Wood 2014-01-21 16:28:23 UTC
(In reply to Deepak Bhole from comment #11)
> Looks like there was an error during JDWP initialization. Is this
> reproducible? If so, can you kill all Java processes (or reboot) and try
> again?

Yes, this is reproducible.  I've rebooted since I submitted this bug and stopped all Java processes.  I can then start Tomcat, allow it to finish initialization, and stop it.  Every time I stop Tomcat, I get this issue.

Comment 14 Deepak Bhole 2014-01-21 16:34:07 UTC
(In reply to Alex Wood from comment #13)
> (In reply to Deepak Bhole from comment #11)
> > Looks like there was an error during JDWP initialization. Is this
> > reproducible? If so, can you kill all Java processes (or reboot) and try
> > again?
> 
> Yes, this is reproducible.  I've rebooted since I submitted this bug and
> stopped all Java processes.  I can then start Tomcat, allow it to finish
> initialization, and stop it.  Every time I stop Tomcat, I get this issue.

Does the error happen when it is actually stopping, or when you try to restart after stopping?

Comment 15 Alex Wood 2014-01-21 17:04:42 UTC
> Does the error happen when it is actually stopping, or when you try to
> restart after stopping?

The error occurs when I stop Tomcat.

Comment 20 Deepak Bhole 2014-06-12 21:13:46 UTC
Hi Alex, sorry it took us to long to get back to this. I took another close look at the traces and I know why the error is occuring.

When Tomcat is started, it is almost certainly being started with the JDWP agent running on port 8000. When it is being shut down, another JVM appears to be initialiazed (with the "Stop" argument, as in the original report). The second JVM instance is also being told to run JDWP at port 8000, which is still in use by the original one.

This is not an error in OpenJDK, but rather in Tomcat configuration. Have you modified the Tomcat configuration in any way on the system where this is happening?

Comment 21 Alex Wood 2014-06-16 16:06:20 UTC
Ah ha!  Yes, I have modified tomcat.conf to enable JDWP.

CATALINA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000"

However, I looked at the difference between CATALINA_OPTS and JAVA_OPTS and found
http://stackoverflow.com/a/11222367 which puzzles me because according to that answer, the JVM run with the "stop" argument shouldn't be receiving the CATALINA_OPTS string as arguments.

Any ideas on how to avoid this issue?  I use the JDWP connection frequently, so I'd like to keep it available.

Comment 22 Alex Wood 2014-06-16 16:12:07 UTC
Looking at /usr/libexec/tomcat/server, I see

MAIN_CLASS=org.apache.catalina.startup.Bootstrap

FLAGS="$JAVA_OPTS $CATALINA_OPTS"
OPTIONS="-Dcatalina.base=$CATALINA_BASE \
-Dcatalina.home=$CATALINA_HOME \
-Djava.endorsed.dirs=$JAVA_ENDORSED_DIRS \
-Djava.io.tmpdir=$CATALINA_TMPDIR \
-Djava.util.logging.config.file=${LOGGING_PROPERTIES} \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager"

if [ "$1" = "start" ] ; then
  if [ "${SECURITY_MANAGER}" = "true" ] ; then
    OPTIONS="${OPTIONS} \
    -Djava.security.manager \
    -Djava.security.policy==${CATALINA_BASE}/conf/catalina.policy"
  fi
  run start
elif [ "$1" = "stop" ] ; then
  run stop
fi

So it looks like the startup/shutdown script simply doesn't honor the distinction between JAVA_OPTS and CATALINA_OPTS.  Perhaps we can transfer this bug to the Tomcat component?

Comment 23 Deepak Bhole 2014-06-16 20:04:57 UTC
Hi Alex, thank you for investigating it further! I agree, this is definitely a bug as users should be allowed to use JDWP without having that exception show up each time. Furthermore, that exception means that the JVM that is supposed to do a safe shutdown never runs, which could lead to problems caused by improper shutdown/serialization.

Transferring this to tomcat component.

To tomcat maintainers:

Please see comment 20 onward for why this is a tomcat issue. Basically, the tomcat package provides no way to set startup only JVM args. Any JVM args provided for startup are also used during shutdown it seems and it can cause problems when using services like JDWP.

Comment 24 Ivan Afonichev 2014-06-16 20:14:07 UTC
Nevertheless is it good approach for JVM to fail with stacktrace if it is simply cannot bind the port?

Comment 25 Deepak Bhole 2014-06-16 20:17:21 UTC
I agree that it would be nicer, but unfortunately this is by design afaik. 

JDWP is the debug wire protocol. It needs to be started very early during JVM startup to debug initialization and what not -- in order to provide a safe shutdown/stack trace as you suggest, the JVM would need to be fully initialized first.

Comment 26 Fedora Update System 2014-09-16 21:36:48 UTC
tomcat-7.0.52-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/tomcat-7.0.52-1.fc20

Comment 27 Fedora Update System 2014-09-19 10:13:40 UTC
Package tomcat-7.0.52-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing tomcat-7.0.52-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-11048/tomcat-7.0.52-1.fc20
then log in and leave karma (feedback).

Comment 28 Fedora Update System 2014-09-26 09:02:23 UTC
tomcat-7.0.52-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 29 Chris Hubick 2014-10-08 18:17:34 UTC
My tomcat.conf CATALINA_OPTS for JDWP in F20 just stopped working altogether with this tomcat-7.0.52 update, via restart *and* independent stop/start.

Thanks to the yum.log and a 'yum downgrade' to 7.0.47, I'm back to debugging again.

Comment 30 Alex Wood 2014-10-09 20:57:13 UTC
(In reply to Chris Hubick from comment #29)
> My tomcat.conf CATALINA_OPTS for JDWP in F20 just stopped working altogether
> with this tomcat-7.0.52 update, via restart *and* independent stop/start.
> 
> Thanks to the yum.log and a 'yum downgrade' to 7.0.47, I'm back to debugging
> again.

Chris, there was an issue with this update.  I've filed bug 1147105 and mentioned a work-around.


Note You need to log in before you can comment on or make changes to this bug.