Created attachment 888605 [details] server.log I face troubles of threads being deadlocked when server is going to shutdown. This happens in half of the cases when I run my integration arquillian tests again EAP 6.3.0.ER1. I deploy one testing application packaged as jar file and then jdbc driver is deployed by copying to deployments folder. I'm attaching server log file and thread dump at the time when threads are stuck. I was trying to find out where the lock occurs and from the stack trace it happens on two places. One is lock on remoting queue and other is lock on DeploymentRepository object. I did a quick fix which seems to work for me that I removed 'synchronized' keyword from the methods which remove listeners in DeploymentRepository. It means from: https://github.com/jbossas/jboss-eap/blob/6.x/ejb3/src/main/java/org/jboss/as/ejb3/deployment/DeploymentRepository.java#L103 and https://github.com/jbossas/jboss-eap/blob/6.x/ejb3/src/main/java/org/jboss/as/ejb3/deployment/DeploymentRepository.java#L107 I'm really not sure whether this is 'a correct' fix but I just put my observation here to have some starting point.
Created attachment 888606 [details] server.stacktrace.dump
Definitely a bug. Needs upstream too though.
The issue was as well hit for EAP integration testsuite running against databases in our test lab (e.g. postgresql, oracle etc.). It seems that the undeploying of the driver could somelike play some role in this.
I need to point out that I'm now running my testing with the 'hot fix' and I have to say that it seems that it causes a warning being written to log (time to time). The server is correctly stopped and then it could be started without problem but the log contains a bit unlovely message. 06:29:42,923 WARN [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000004: Failure during stop of service jboss.deployment.unit."dsdriver.jar".moduleDeploymentRuntimeInformation: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) [rt.jar:1.7.0_51] at java.util.ArrayList$Itr.next(ArrayList.java:831) [rt.jar:1.7.0_51] at org.jboss.as.ejb3.deployment.DeploymentRepository.remove(DeploymentRepository.java:111) at org.jboss.as.ejb3.deployment.ModuleDeployment.stop(ModuleDeployment.java:74) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2088) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2049) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
The line numbers don't seem to match up with my local copy, and I don't see anyplace where this should even be possible?
Ah if you're testing your "remove synch" patch - that is definitely not a viable solution, though it was helpful in diagnosing the problem. You should use the attached fix instead.
OK, I see. I will take it from your pull request. Sorry for the buzz.
Verified on 6.3.0.ER3. The hang up troubles on shutdown disappeared and the change is part of the code (checked the decompiled jar jboss-as-ejb3-7.4.0.Final-redhat-11)
Marking for exclusion from 6.3.0 Beta release notes as both 'affects' and 'fix' versions are listed as 6.3.0, suggesting this was not a customer-facing issue.