Bug 1090147 - Thread locking problem on server that is going to shutdown
Summary: Thread locking problem on server that is going to shutdown
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: EJB
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ER3
: EAP 6.3.0
Assignee: David M. Lloyd
QA Contact: Ondrej Chaloupka
Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks: eap63-beta-blockers
TreeView+ depends on / blocked
 
Reported: 2014-04-22 17:37 UTC by Ondrej Chaloupka
Modified: 2014-06-28 15:29 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-28 15:29:54 UTC
Type: Bug


Attachments (Terms of Use)
server.log (373.71 KB, text/x-log)
2014-04-22 17:37 UTC, Ondrej Chaloupka
no flags Details
server.stacktrace.dump (35.25 KB, text/plain)
2014-04-22 17:39 UTC, Ondrej Chaloupka
no flags Details


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker WFLY-3280 Major Resolved Thread locking problem when app server is going to shutdown 2016-03-16 16:23:32 UTC

Description Ondrej Chaloupka 2014-04-22 17:37:08 UTC
Created attachment 888605 [details]
server.log

I face troubles of threads being deadlocked when server is going to shutdown.
This happens in half of the cases when I run my integration arquillian tests again EAP 6.3.0.ER1.
I deploy one testing application packaged as jar file and then jdbc driver is deployed by copying to deployments folder.

I'm attaching server log file and thread dump at the time when threads are stuck.

I was trying to find out where the lock occurs and from the stack trace it happens on two places. One is lock on remoting queue and other is lock on DeploymentRepository object.

I did a quick fix which seems to work for me that I removed 'synchronized' keyword from the methods which remove listeners in DeploymentRepository.
It means from:
https://github.com/jbossas/jboss-eap/blob/6.x/ejb3/src/main/java/org/jboss/as/ejb3/deployment/DeploymentRepository.java#L103
and
https://github.com/jbossas/jboss-eap/blob/6.x/ejb3/src/main/java/org/jboss/as/ejb3/deployment/DeploymentRepository.java#L107
I'm really not sure whether this is 'a correct' fix but I just put my observation here to have some starting point.

Comment 1 Ondrej Chaloupka 2014-04-22 17:39:35 UTC
Created attachment 888606 [details]
server.stacktrace.dump

Comment 2 David M. Lloyd 2014-04-22 18:24:21 UTC
Definitely a bug.  Needs upstream too though.

Comment 3 Ondrej Chaloupka 2014-04-23 07:22:13 UTC
The issue was as well hit for EAP integration testsuite running against databases in our test lab (e.g. postgresql, oracle etc.). It seems that the undeploying of the driver could somelike play some role in this.

Comment 6 Ondrej Chaloupka 2014-04-23 13:59:11 UTC
I need to point out that I'm now running my testing with the 'hot fix' and I have to say that it seems that it causes a warning being written to log (time to time).
The server is correctly stopped and then it could be started without problem but the log contains a bit unlovely message.

06:29:42,923 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000004: Failure during stop of service jboss.deployment.unit."dsdriver.jar".moduleDeploymentRuntimeInformation: java.util.ConcurrentModificationException
 at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) [rt.jar:1.7.0_51]
 at java.util.ArrayList$Itr.next(ArrayList.java:831) [rt.jar:1.7.0_51]
 at org.jboss.as.ejb3.deployment.DeploymentRepository.remove(DeploymentRepository.java:111)
 at org.jboss.as.ejb3.deployment.ModuleDeployment.stop(ModuleDeployment.java:74)
 at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2088) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
 at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2049) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
 at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

Comment 7 David M. Lloyd 2014-04-23 14:06:29 UTC
The line numbers don't seem to match up with my local copy, and I don't see anyplace where this should even be possible?

Comment 8 David M. Lloyd 2014-04-23 14:11:17 UTC
Ah if you're testing your "remove synch" patch - that is definitely not a viable solution, though it was helpful in diagnosing the problem.

You should use the attached fix instead.

Comment 9 Ondrej Chaloupka 2014-04-23 14:32:42 UTC
OK, I see. I will take it from your pull request. Sorry for the buzz.

Comment 10 Ondrej Chaloupka 2014-05-06 07:58:24 UTC
Verified on 6.3.0.ER3.

The hang up troubles on shutdown disappeared and the change is part of the code (checked the decompiled jar jboss-as-ejb3-7.4.0.Final-redhat-11)

Comment 11 Scott Mumford 2014-05-14 00:43:12 UTC
Marking for exclusion from 6.3.0 Beta release notes as both 'affects' and 'fix' versions are listed as 6.3.0, suggesting this was not a customer-facing issue.


Note You need to log in before you can comment on or make changes to this bug.