Bug 1090147

Summary: Thread locking problem on server that is going to shutdown
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: EJBAssignee: David M. Lloyd <david.lloyd>
Status: CLOSED CURRENTRELEASE QA Contact: Ondrej Chaloupka <ochaloup>
Severity: medium Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.3.0CC: hhovsepy, jcacek, jkudrnac, kkhan, pjelinek, rhatlapa, rsvoboda, smumford
Target Milestone: ER3   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-28 15:29:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1051640    
Attachments:
Description Flags
server.log
none
server.stacktrace.dump none

Description Ondrej Chaloupka 2014-04-22 17:37:08 UTC
Created attachment 888605 [details]
server.log

I face troubles of threads being deadlocked when server is going to shutdown.
This happens in half of the cases when I run my integration arquillian tests again EAP 6.3.0.ER1.
I deploy one testing application packaged as jar file and then jdbc driver is deployed by copying to deployments folder.

I'm attaching server log file and thread dump at the time when threads are stuck.

I was trying to find out where the lock occurs and from the stack trace it happens on two places. One is lock on remoting queue and other is lock on DeploymentRepository object.

I did a quick fix which seems to work for me that I removed 'synchronized' keyword from the methods which remove listeners in DeploymentRepository.
It means from:
https://github.com/jbossas/jboss-eap/blob/6.x/ejb3/src/main/java/org/jboss/as/ejb3/deployment/DeploymentRepository.java#L103
and
https://github.com/jbossas/jboss-eap/blob/6.x/ejb3/src/main/java/org/jboss/as/ejb3/deployment/DeploymentRepository.java#L107
I'm really not sure whether this is 'a correct' fix but I just put my observation here to have some starting point.

Comment 1 Ondrej Chaloupka 2014-04-22 17:39:35 UTC
Created attachment 888606 [details]
server.stacktrace.dump

Comment 2 David M. Lloyd 2014-04-22 18:24:21 UTC
Definitely a bug.  Needs upstream too though.

Comment 3 Ondrej Chaloupka 2014-04-23 07:22:13 UTC
The issue was as well hit for EAP integration testsuite running against databases in our test lab (e.g. postgresql, oracle etc.). It seems that the undeploying of the driver could somelike play some role in this.

Comment 6 Ondrej Chaloupka 2014-04-23 13:59:11 UTC
I need to point out that I'm now running my testing with the 'hot fix' and I have to say that it seems that it causes a warning being written to log (time to time).
The server is correctly stopped and then it could be started without problem but the log contains a bit unlovely message.

06:29:42,923 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000004: Failure during stop of service jboss.deployment.unit."dsdriver.jar".moduleDeploymentRuntimeInformation: java.util.ConcurrentModificationException
 at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) [rt.jar:1.7.0_51]
 at java.util.ArrayList$Itr.next(ArrayList.java:831) [rt.jar:1.7.0_51]
 at org.jboss.as.ejb3.deployment.DeploymentRepository.remove(DeploymentRepository.java:111)
 at org.jboss.as.ejb3.deployment.ModuleDeployment.stop(ModuleDeployment.java:74)
 at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2088) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
 at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2049) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
 at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

Comment 7 David M. Lloyd 2014-04-23 14:06:29 UTC
The line numbers don't seem to match up with my local copy, and I don't see anyplace where this should even be possible?

Comment 8 David M. Lloyd 2014-04-23 14:11:17 UTC
Ah if you're testing your "remove synch" patch - that is definitely not a viable solution, though it was helpful in diagnosing the problem.

You should use the attached fix instead.

Comment 9 Ondrej Chaloupka 2014-04-23 14:32:42 UTC
OK, I see. I will take it from your pull request. Sorry for the buzz.

Comment 10 Ondrej Chaloupka 2014-05-06 07:58:24 UTC
Verified on 6.3.0.ER3.

The hang up troubles on shutdown disappeared and the change is part of the code (checked the decompiled jar jboss-as-ejb3-7.4.0.Final-redhat-11)

Comment 11 Scott Mumford 2014-05-14 00:43:12 UTC
Marking for exclusion from 6.3.0 Beta release notes as both 'affects' and 'fix' versions are listed as 6.3.0, suggesting this was not a customer-facing issue.