Bug 957689

Summary:	Unable to complete tasks when server is shutdown
Product:	[Other] RHQ Project	Reporter:	Thomas Segismont <tsegismo>
Component:	Core Server	Assignee:	Thomas Segismont <tsegismo>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Mike Foley <mfoley>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.6	CC:	hrupp, mazz
Target Milestone:	---
Target Release:	RHQ 4.9
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	963167 (view as bug list)		Environment:
Last Closed:	2014-03-26 08:31:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	958217
Bug Blocks:	963167

Description Thomas Segismont 2013-04-29 09:13:11 UTC

Description of problem:
When the server is shutdown, several components (datasources, ejbs) go down before RHQ's shutdown listener code completes.

Version-Release number of selected component (if applicable):
4.6

How reproducible:
Always

Steps to Reproduce:
1. Add a 'Thread.sleep' statement in Data Purge Job code
2. Change the scheduling code to start the Data Purge Job 30 seconds after server startup
2. Rebuild, start, wait and stop the server when purge job is executed
  
Actual results:
Errors in the server log: http://pastebin.test.redhat.com/138892

Expected results:
The server should not go down until data purge job and shutdown listener code finish.

Additional info:
To be verified: it looks like AS7, unlike AS 4.2.3, does not wait for our Shutdown listener code to complete execution before shutting down all resources (datasources, EJBs, etc...)

Comment 1 Thomas Segismont 2013-04-30 13:29:02 UTC

Before RHQ4.6, shutdown listener was implemented as a JMX notification listener. Now it is implemented as a Singleton bean and shutdown logic is in the @PreDestroy method.

It's not really needed to hack the Data Purge Job code to show this. Almost always, when you hit Ctrl-C on a server started in console mode, you'll see the same exception message:

javax.ejb.EJBException: JBAS014559: Invocation cannot proceed as component is shutting down 

This message was introduced in this AS7 pull request:
https://github.com/wildfly/wildfly/pull/1889

Calling @EJB methods in @PreDestroy callbacks is normally allowed by the spec. Somebody reported similar (but not same) problem and the issue was accepted and resolved:
https://community.jboss.org/message/764240
https://issues.jboss.org/browse/AS7-4808

Note: shutdown notification should come back in Wildfly
http://lists.jboss.org/pipermail/jboss-as7-dev/2012-April/005834.html

Comment 2 Thomas Segismont 2013-06-04 16:22:42 UTC

Remember to check that integration tests are not affected by the change

Comment 3 Thomas Segismont 2013-06-25 14:18:02 UTC

commit adedcb46e352d6fb08a43352447b748b9459c899
Author: Thomas Segismont <tsegismo>
Date:   Thu Jun 6 22:56:11 2013 +0200

    Bug 957689 - Unable to complete tasks when server is shutdown
    
    Inject the NoTxRHQDS as a @Resource in the ShutdownListener
    Convert SAR services into @Singleton beans
    Tweak RHQ EAR deployment with a set of DeploymentUnitProcessor classes
    
    EAR modules now deployed in order
    RHQ Startup Subsystem installs a DUP which make the ShutdownListener bean automatically depend on all other session beans.

Comment 4 Thomas Segismont 2013-06-28 10:49:31 UTC

Second commit in master. There is no longer need to restart the server after install.

commit 0bddf58297213e94fe7d8ab426c07c1ad10dfe70
Author: Thomas Segismont <tsegismo>
Date:   Fri Jun 28 12:45:50 2013 +0200

    Bug 957689 - Unable to complete tasks when server is shutdown
    
    Removed DUPs
    Made Core GUI last component to get deployed and first to get undeployed
    Added CoreGuiServletContextListener to initialize or shutdown RHQ server

Comment 5 Heiko W. Rupp 2014-03-26 08:31:55 UTC

Bulk closing now that 4.10 is out.

If you think an issue is not resolved, please open a new BZ and link to the existing one.