Hide Forgot
Description of problem: tomcat-7.0.47-1.fc20.noarch running with CATALINA_OPTS set to "-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n" Stopping the service with 'service tomcat stop' yields the reported error. Version-Release number of selected component: java-1.7.0-openjdk-headless-1.7.0.60-2.4.3.0.fc20 Additional info: reporter: libreport-2.1.10 backtrace_rating: 4 cmdline: java -agentpath:/usr/lib64/libabrt-java-connector.so=abrt=on -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap stop crash_function: jni_FatalError executable: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/jre-abrt/bin/java kernel: 3.12.6-300.fc20.x86_64 runlevel: N 5 type: CCpp uid: 91 Truncated backtrace: Thread no. 1 (10 frames) #3 jni_FatalError at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/prims/jni.cpp:870 #4 jniFatalError at ../../../src/share/back/debugInit.c:671 #5 debugInit_exit at ../../../src/share/back/debugInit.c:1334 #6 initialize at ../../../src/share/back/debugInit.c:750 #7 cbEarlyVMInit at ../../../src/share/back/debugInit.c:458 #8 JvmtiExport::post_vm_initialized at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiExport.cpp:466 #9 Threads::create_vm at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3635 #10 JNI_CreateJavaVM at /usr/src/debug/java-1.7.0-openjdk-1.7.0.60-2.4.3.0.fc20.x86_64/openjdk/hotspot/src/share/vm/prims/jni.cpp:5128 #11 InitializeJVM at ../../../src/share/bin/java.c:1120 #12 JavaMain at ../../../src/share/bin/java.c:358
Created attachment 847799 [details] File: backtrace
Created attachment 847800 [details] File: cgroup
Created attachment 847801 [details] File: core_backtrace
Created attachment 847802 [details] File: dso_list
Created attachment 847803 [details] File: environ
Created attachment 847804 [details] File: limits
Created attachment 847805 [details] File: maps
Created attachment 847806 [details] File: open_fds
Created attachment 847807 [details] File: proc_pid_status
Created attachment 847808 [details] File: var_log_messages
Looks like there was an error during JDWP initialization. Is this reproducible? If so, can you kill all Java processes (or reboot) and try again?
Sadly I have no idea. I was using Netbeans and Glassfish 3.1.2.2 for development. Actually I didn't notice anything special (I think the computer load was a bit high but I don't recall the details), and at some point the fedora bug report tool notified me that Java had crashed and prompted me to report it. Honestly, had it not popped up, I would not have noticed. I'll try to be careful if it will happen again but it's definitely not reproducible easily and I have no idea how.
(In reply to Deepak Bhole from comment #11) > Looks like there was an error during JDWP initialization. Is this > reproducible? If so, can you kill all Java processes (or reboot) and try > again? Yes, this is reproducible. I've rebooted since I submitted this bug and stopped all Java processes. I can then start Tomcat, allow it to finish initialization, and stop it. Every time I stop Tomcat, I get this issue.
(In reply to Alex Wood from comment #13) > (In reply to Deepak Bhole from comment #11) > > Looks like there was an error during JDWP initialization. Is this > > reproducible? If so, can you kill all Java processes (or reboot) and try > > again? > > Yes, this is reproducible. I've rebooted since I submitted this bug and > stopped all Java processes. I can then start Tomcat, allow it to finish > initialization, and stop it. Every time I stop Tomcat, I get this issue. Does the error happen when it is actually stopping, or when you try to restart after stopping?
> Does the error happen when it is actually stopping, or when you try to > restart after stopping? The error occurs when I stop Tomcat.
Hi Alex, sorry it took us to long to get back to this. I took another close look at the traces and I know why the error is occuring. When Tomcat is started, it is almost certainly being started with the JDWP agent running on port 8000. When it is being shut down, another JVM appears to be initialiazed (with the "Stop" argument, as in the original report). The second JVM instance is also being told to run JDWP at port 8000, which is still in use by the original one. This is not an error in OpenJDK, but rather in Tomcat configuration. Have you modified the Tomcat configuration in any way on the system where this is happening?
Ah ha! Yes, I have modified tomcat.conf to enable JDWP. CATALINA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000" However, I looked at the difference between CATALINA_OPTS and JAVA_OPTS and found http://stackoverflow.com/a/11222367 which puzzles me because according to that answer, the JVM run with the "stop" argument shouldn't be receiving the CATALINA_OPTS string as arguments. Any ideas on how to avoid this issue? I use the JDWP connection frequently, so I'd like to keep it available.
Looking at /usr/libexec/tomcat/server, I see MAIN_CLASS=org.apache.catalina.startup.Bootstrap FLAGS="$JAVA_OPTS $CATALINA_OPTS" OPTIONS="-Dcatalina.base=$CATALINA_BASE \ -Dcatalina.home=$CATALINA_HOME \ -Djava.endorsed.dirs=$JAVA_ENDORSED_DIRS \ -Djava.io.tmpdir=$CATALINA_TMPDIR \ -Djava.util.logging.config.file=${LOGGING_PROPERTIES} \ -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" if [ "$1" = "start" ] ; then if [ "${SECURITY_MANAGER}" = "true" ] ; then OPTIONS="${OPTIONS} \ -Djava.security.manager \ -Djava.security.policy==${CATALINA_BASE}/conf/catalina.policy" fi run start elif [ "$1" = "stop" ] ; then run stop fi So it looks like the startup/shutdown script simply doesn't honor the distinction between JAVA_OPTS and CATALINA_OPTS. Perhaps we can transfer this bug to the Tomcat component?
Hi Alex, thank you for investigating it further! I agree, this is definitely a bug as users should be allowed to use JDWP without having that exception show up each time. Furthermore, that exception means that the JVM that is supposed to do a safe shutdown never runs, which could lead to problems caused by improper shutdown/serialization. Transferring this to tomcat component. To tomcat maintainers: Please see comment 20 onward for why this is a tomcat issue. Basically, the tomcat package provides no way to set startup only JVM args. Any JVM args provided for startup are also used during shutdown it seems and it can cause problems when using services like JDWP.
Nevertheless is it good approach for JVM to fail with stacktrace if it is simply cannot bind the port?
I agree that it would be nicer, but unfortunately this is by design afaik. JDWP is the debug wire protocol. It needs to be started very early during JVM startup to debug initialization and what not -- in order to provide a safe shutdown/stack trace as you suggest, the JVM would need to be fully initialized first.
tomcat-7.0.52-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/tomcat-7.0.52-1.fc20
Package tomcat-7.0.52-1.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing tomcat-7.0.52-1.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-11048/tomcat-7.0.52-1.fc20 then log in and leave karma (feedback).
tomcat-7.0.52-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
My tomcat.conf CATALINA_OPTS for JDWP in F20 just stopped working altogether with this tomcat-7.0.52 update, via restart *and* independent stop/start. Thanks to the yum.log and a 'yum downgrade' to 7.0.47, I'm back to debugging again.
(In reply to Chris Hubick from comment #29) > My tomcat.conf CATALINA_OPTS for JDWP in F20 just stopped working altogether > with this tomcat-7.0.52 update, via restart *and* independent stop/start. > > Thanks to the yum.log and a 'yum downgrade' to 7.0.47, I'm back to debugging > again. Chris, there was an issue with this update. I've filed bug 1147105 and mentioned a work-around.