Description of problem: When shutting down the server, the following error is logged and the server hangs during shutdown: ERROR [STDERR] Exception in thread "AWT-Windows" ERROR [STDERR] java.lang.IllegalStateException: Shutdown in progress ERROR [STDERR] at java.lang.ApplicationShutdownHooks.add(Unknown Source) ERROR [STDERR] at java.lang.Runtime.addShutdownHook(Unknown Source) ERROR [STDERR] at sun.awt.windows.WToolkit$2.run(Unknown Source) ERROR [STDERR] at java.security.AccessController.doPrivileged(Native Method) ERROR [STDERR] at sun.awt.windows.WToolkit.run(Unknown Source) ERROR [STDERR] at java.lang.Thread.run(Unknown Source) The result is that the JVM process does not exit on its own. Instead, the java.exe process must be terminated or if the Java Service Wrapper was used to launch the server, wait for its wrapper.jvm_exit.timeout value to expire (which by default is 180 seconds). However, even then the JVM is not being shutdown properly/cleanly. Version-Release number of selected component (if applicable): 4.4.0.JON311GA How reproducible: Always Steps to Reproduce: 1. Install Windows Server 2008 2. Install PostgreSQL database 3. Initialize RHQ PostgreSQL role and database createuser -U postgres -SRD rhqadmin createdb -U postgres -O rhqadmin rhq psql -U postgres -c "ALTER USER rhqadmin WITH PASSWORD 'rhqadmin';" 4. Install Java 6 32-bit JRE This is required because JBoss ON on Windows can only run as a service in a 32-bit JRE due to us using a third-party service wrapper. 5. Install JBoss ON 3.1.1 server using rhq-server console 6. After configured and installed, from the terminal windows where rhq-server console was executed, issue CTRL-C Actual results: Server log contains the following exception and JVM doesn't shutdown until the Java Service Wrapper kills it 3 minuts later. ERROR [STDERR] Exception in thread "AWT-Windows" ERROR [STDERR] java.lang.IllegalStateException: Shutdown in progress ERROR [STDERR] at java.lang.ApplicationShutdownHooks.add(Unknown Source) ERROR [STDERR] at java.lang.Runtime.addShutdownHook(Unknown Source) ERROR [STDERR] at sun.awt.windows.WToolkit$2.run(Unknown Source) ERROR [STDERR] at java.security.AccessController.doPrivileged(Native Method) ERROR [STDERR] at sun.awt.windows.WToolkit.run(Unknown Source) ERROR [STDERR] at java.lang.Thread.run(Unknown Source) Expected results: Server should shutdown as normal and no execption should be logged and JVM should not hang. Additional info: This appears to be a result of statically initializing AWT from org.rhq.enterprise.gui.image.WebImage. Part of this may be a bug in AWT and its inability to understand that a shutdown is already underway (i.e. it doesn't handle the exception thrown and simply waits). However, to prevent it, it is recommended that the graphics library is loaded when needed instead of statically in the class loader. One possible workaround for this issue is to use the following JVM argument to force Tomcat not to free up static classes: -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false The down-side to this is that memory leaks will be introduced in the event that classes are unloaded/reloaded which use thes static references.
Created attachment 666498 [details] RHQ server log showing shutdown hang
Created attachment 666499 [details] RHQ server wrapper log showing failed shutdown w/ JVM termination
Created attachment 666500 [details] RHQ server thread dumps at time of hang
Created attachment 666501 [details] RHQ server boot log
This may no longer be a problem given that we are now based on a completely different AS version and also have removed reliance on JSF charting. We'll need to re-test using an actual JON32 build. We should still use a Java6 JRE when retesting. I've not had any problems with Ctrl-C but I've been using a dev env, java7 and Win7.
Created task 995541 which asks that the old unused code be removed to ensure this is no longer an issue. Once that work is done, this can be re-tested to verify no AWT shutdown thread blocking occurs.
Larry, can you try again with "-Djava.awt.headless=true" added to the java options? This option is set in Linux scripts but not in wrapper.conf
"-Djava.awt.headless=true" is already present in modules/enterprise/server/appserver/src/main/bin-resources/bin/wrapper/rhq-server-wrapper.conf
Moving to ON_QA as available for testing in the following brew build: https://brewweb.devel.redhat.com//buildinfo?buildID=336752 Note: the installed version is still JON 3.2.0.GA by design and this represents part of the payload for JON 3.2.1 also known as cumulative patch 1 for 3.2.0.GA. How this will be delivered to customers is still being discussed.
Tested on JON 3.2.1 DR01 build (Build Number :c758688:4c03150) on Windows server 2008 with 32 bit JAVA 6 JRE. 1.Installed JBoss ON 3.2.1 DR01 build with - rhqctl.bat install 2.Started the services with - rhqctl.bat start 3.Stopped the server using - rhqctl.bat stop --server 4.Started server in console mode with rhqctl.bat console --server 5. After starting server in console mode on windows terminal, issued <CTRL-C> 6. It displayed - Terminate batch job <Y/N>? ( pressed Y) There are no exceptions in server log and server does not hang, however, issuing <CTRL-C> on terminal does not stop the server. The server can be stopped after killing the java.exe process using windows task manager. Reopening the bug as issuing <CTRL-C> on windows terminal does not stop the server.
Moving back to ON_QA so that QE can verify they have the test case defined for the original issue. The new issue identified in comment 13 has been captured in Bug 1071379.
Test case run for the original issue: https://tcms.engineering.redhat.com/run/118611/
JON 3.2.1 released week of 5/5/2014