Bug 739332

Summary: Qpid JCA Adapter -- Thread Deadlock in QpidActivation
Product: Red Hat Enterprise MRG Reporter: Weston M. Price <wprice>
Component: qpid-jcaAssignee: Weston M. Price <wprice>
Status: CLOSED UPSTREAM QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 2.0CC: astitcher, cdewolf, iboverma, jpechane, jross, tross
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-07 11:40:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
jstack - <pid> trace showing deadlock condition for BZ-739332 none

Description Weston M. Price 2011-09-17 19:33:05 UTC
Description of problem:

There is a deadlock condition in the Qpid JCA adapter. For inbound messaging, endpoint activation is handled by the QpidActivation.SetupActivation class. This class is a Work instance and is submitted to the JCA WorkManager, as such it is run in a separate thread. 

In the case where the initial activation call blocks (invalid durable subscription names), the IOReceiver thread receives an exception prior to the QpidActivation class every being property initialized. On an exception we attempt to teardown and setup the activation again. However, at this point the QpidActivation instance is locked by the JCA Work manager, as such, the IOReceiver thread (or any other that received the exception first), can't make forward progress because the QpidActivation class has been locked. Both the setup() and teardown() methods are synchronized and prohibit  forward progress since the JCA Work Manager already has a lock on the instance.

How reproducible:
Immediate

Steps to Reproduce:
1. Deploy the Qpid JCA adapter
2. Create a Topic Destination via *-ds.xml file.
3. Deploy an MDB, using a durable subscription with a name that does not exist.
  
Actual results:
JBoss EAP locks up and as such, does not allow a graceful shutdown without resorting to kill - <pid> or other extraordinary measures. Run jstack <pid> to review the deadlock condition.


The QpidActivation class contains a a SetupActivation class that is only responsible for setting up the initial connection/session etc. It shares the same code with the QpidActivation as a whole most notably the onException(), handleFailure() code. If the initial setup fails, or blocks, other threads receive the onException() invocation and attempt to teardown() and setup() the activation again. Since the instance is blocked, a deadlock occurs.

Comment 1 Weston M. Price 2011-09-17 19:47:10 UTC
Created attachment 523723 [details]
jstack - <pid> trace showing deadlock condition for BZ-739332

Comment 3 Weston M. Price 2012-01-12 13:15:16 UTC
This really shouldn't be marked as a blocker as it only occurs under misconfiguration conditions and does not apply to correctly configured endpoints. However, I agree that it should be fixed is possible before release. The fix will be in the current builds.

Comment 7 Weston M. Price 2013-01-07 11:40:28 UTC
Not sure why this is still open, this was fixed in a previous release (0.16). Closing it as a result.

Comment 8 Weston M. Price 2013-01-07 11:40:46 UTC
Not sure why this is still open, this was fixed in a previous release (0.16). Closing it as a result.