Bug 889012 - Server shutdown hangs on Windows
Server shutdown hangs on Windows
Status: CLOSED CURRENTRELEASE
Product: JBoss Operations Network
Classification: JBoss
Component: Core Server (Show other bugs)
JON 3.1.1,JON 3.1.2
All Windows
high Severity high
: DR01
: JON 3.2.1
Assigned To: Thomas Segismont
Mike Foley
: TestCaseNeeded
Depends On: 995541
Blocks:
  Show dependency treegraph
 
Reported: 2012-12-19 21:38 EST by Larry O'Leary
Modified: 2014-05-08 13:43 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-05-08 13:43:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
RHQ server log showing shutdown hang (28.95 KB, application/octet-stream)
2012-12-19 22:14 EST, Larry O'Leary
no flags Details
RHQ server wrapper log showing failed shutdown w/ JVM termination (187 bytes, application/octet-stream)
2012-12-19 22:15 EST, Larry O'Leary
no flags Details
RHQ server thread dumps at time of hang (66.00 KB, application/octet-stream)
2012-12-19 22:15 EST, Larry O'Leary
no flags Details
RHQ server boot log (79.48 KB, application/octet-stream)
2012-12-19 22:16 EST, Larry O'Leary
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 282073 None None None Never

  None (edit)
Description Larry O'Leary 2012-12-19 21:38:10 EST
Description of problem:
When shutting down the server, the following error is logged and the server hangs during shutdown:

    ERROR [STDERR] Exception in thread "AWT-Windows" 
    ERROR [STDERR] java.lang.IllegalStateException: Shutdown in progress
    ERROR [STDERR] 	at java.lang.ApplicationShutdownHooks.add(Unknown Source)
    ERROR [STDERR] 	at java.lang.Runtime.addShutdownHook(Unknown Source)
    ERROR [STDERR] 	at sun.awt.windows.WToolkit$2.run(Unknown Source)
    ERROR [STDERR] 	at java.security.AccessController.doPrivileged(Native Method)
    ERROR [STDERR] 	at sun.awt.windows.WToolkit.run(Unknown Source)
    ERROR [STDERR] 	at java.lang.Thread.run(Unknown Source)

The result is that the JVM process does not exit on its own. Instead, the java.exe process must be terminated or if the Java Service Wrapper was used to launch the server, wait for its wrapper.jvm_exit.timeout value to expire (which by default is 180 seconds). However, even then the JVM is not being shutdown properly/cleanly.


Version-Release number of selected component (if applicable):
4.4.0.JON311GA

How reproducible:
Always

Steps to Reproduce:
1.  Install Windows Server 2008
2.  Install PostgreSQL database
3.  Initialize RHQ PostgreSQL role and database

    createuser -U postgres -SRD rhqadmin
    createdb -U postgres -O rhqadmin rhq
    psql -U postgres -c "ALTER USER rhqadmin WITH PASSWORD 'rhqadmin';"

4.  Install Java 6 32-bit JRE

    This is required because JBoss ON on Windows can only run as a service in a 32-bit JRE due to us using a third-party service wrapper.
    
5.  Install JBoss ON 3.1.1 server using rhq-server console
6.  After configured and installed, from the terminal windows where rhq-server console was executed, issue CTRL-C
  

Actual results:
Server log contains the following exception and JVM doesn't shutdown until the Java Service Wrapper kills it 3 minuts later.
    ERROR [STDERR] Exception in thread "AWT-Windows" 
    ERROR [STDERR] java.lang.IllegalStateException: Shutdown in progress
    ERROR [STDERR] 	at java.lang.ApplicationShutdownHooks.add(Unknown Source)
    ERROR [STDERR] 	at java.lang.Runtime.addShutdownHook(Unknown Source)
    ERROR [STDERR] 	at sun.awt.windows.WToolkit$2.run(Unknown Source)
    ERROR [STDERR] 	at java.security.AccessController.doPrivileged(Native Method)
    ERROR [STDERR] 	at sun.awt.windows.WToolkit.run(Unknown Source)
    ERROR [STDERR] 	at java.lang.Thread.run(Unknown Source)


Expected results:
Server should shutdown as normal and no execption should be logged and JVM should not hang.


Additional info:
This appears to be a result of statically initializing AWT from org.rhq.enterprise.gui.image.WebImage. Part of this may be a bug in AWT and its inability to understand that a shutdown is already underway (i.e. it doesn't handle the exception thrown and simply waits). However, to prevent it, it is recommended that the graphics library is loaded when needed instead of statically in the class loader.

One possible workaround for this issue is to use the following JVM argument to force Tomcat not to free up static classes:

    -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
    
The down-side to this is that memory leaks will be introduced in the event that classes are unloaded/reloaded which use thes static references.
Comment 1 Larry O'Leary 2012-12-19 22:14:17 EST
Created attachment 666498 [details]
RHQ server log showing shutdown hang
Comment 2 Larry O'Leary 2012-12-19 22:15:16 EST
Created attachment 666499 [details]
RHQ server wrapper log showing failed shutdown w/ JVM termination
Comment 3 Larry O'Leary 2012-12-19 22:15:57 EST
Created attachment 666500 [details]
RHQ server thread dumps at time of hang
Comment 4 Larry O'Leary 2012-12-19 22:16:20 EST
Created attachment 666501 [details]
RHQ server boot log
Comment 5 Jay Shaughnessy 2013-02-27 16:44:54 EST
This may no longer be a problem given that we are now based on a completely different AS version and also have removed reliance on JSF charting.

We'll need to re-test using an actual JON32 build.  We should still use a Java6 JRE when retesting.  

I've not had any problems with Ctrl-C but I've been using a dev env, java7 and Win7.
Comment 6 Larry O'Leary 2013-08-09 12:44:53 EDT
Created task 995541 which asks that the old unused code be removed to ensure this is no longer an issue. Once that work is done, this can be re-tested to verify no AWT shutdown thread blocking occurs.
Comment 7 Thomas Segismont 2013-09-20 10:55:58 EDT
Larry, can you try again with "-Djava.awt.headless=true" added to the java options?

This option is set in Linux scripts but not in wrapper.conf
Comment 11 Thomas Segismont 2014-01-29 11:51:19 EST
"-Djava.awt.headless=true" is already present in modules/enterprise/server/appserver/src/main/bin-resources/bin/wrapper/rhq-server-wrapper.conf
Comment 12 Simeon Pinder 2014-02-18 10:08:22 EST
Moving to ON_QA as available for testing in the following brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=336752

Note: the installed version is still JON 3.2.0.GA by design and this represents part of the payload for JON 3.2.1 also known as cumulative patch 1 for 3.2.0.GA.  How this will be delivered to customers is still being discussed.
Comment 13 Sunil Kondkar 2014-02-28 06:27:40 EST
Tested on JON 3.2.1 DR01 build (Build Number :c758688:4c03150) on Windows server 2008 with 32 bit JAVA 6 JRE.

1.Installed JBoss ON 3.2.1 DR01 build with - rhqctl.bat install
2.Started the services with - rhqctl.bat start
3.Stopped the server using - rhqctl.bat stop --server
4.Started server in console mode with rhqctl.bat console --server

5. After starting server in console mode on windows terminal, issued <CTRL-C>
6. It displayed - Terminate batch job <Y/N>? ( pressed Y)

There are no exceptions in server log and server does not hang, however, issuing <CTRL-C> on terminal does not stop the server. The server can be stopped after killing the java.exe process using windows task manager. 

Reopening the bug as issuing <CTRL-C> on windows terminal does not stop the server.
Comment 15 Larry O'Leary 2014-02-28 11:00:48 EST
Moving back to ON_QA so that QE can verify they have the test case defined for the original issue.

The new issue identified in comment 13 has been captured in Bug 1071379.
Comment 16 Sunil Kondkar 2014-03-03 08:10:16 EST
Test case run for the original issue:
https://tcms.engineering.redhat.com/run/118611/
Comment 17 Mike Foley 2014-05-08 13:43:52 EDT
JON 3.2.1 released week of 5/5/2014

Note You need to log in before you can comment on or make changes to this bug.