Bug 1023053

Summary: Services calling XAResourceRecoveryRegistry.removeXAResourceRecovery in start/stop must use MSC async API
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Brian Stansberry <brian.stansberry>
Component: ServerAssignee: Paul Ferraro <paul.ferraro>
Status: CLOSED CURRENTRELEASE QA Contact: Ondrej Chaloupka <ochaloup>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: dandread, lthon, myarboro, rsvoboda
Target Milestone: CR1   
Target Release: EAP 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-15 16:18:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1022054    
Attachments:
Description Flags
standalone-full-ha.xml none

Description Brian Stansberry 2013-10-24 13:58:19 UTC 
XAResourceRecoveryRegistry.removeXAResourceRecovery is a blocking API, so MSC services that call it in start()/stop() must use Start|StopContext.asynchronous().

Services must not do blocking tasks with the MSC thread that calls the lifecycle methods, as discussed in the class javadoc at http://grepcode.com/file/repository.jboss.org/nexus/content/repositories/releases/org.jboss.msc/jboss-msc/1.0.0.Beta8/org/jboss/msc/service/Service.java. This is because the MSC thread pool is a very limited resource (4 threads), and blocking those threads has the potential to lock up the entire service container.

The issue discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1020209 shows that XAResourceRecoveryRegistry.removeXAResourceRecovery is a blocking call. From discussions with Tom Jenkinson it seems some sort of blocking behavior existed all the way back to 6.0.

The fix is to use the MSC asynchronous APIs in the relevant methods. Backport of fix for https://issues.jboss.org/browse/WFLY-2350.
Comment 1 Brian Stansberry 2013-10-25 22:15:35 UTC 
https://github.com/jbossas/jboss-eap/pull/621 completes most of this, but https://github.com/wildfly/wildfly/pull/5330 needs to be backported.
Comment 2 Paul Ferraro 2013-10-28 18:59:28 UTC 
https://github.com/jbossas/jboss-eap/pull/641
Comment 3 Ondrej Chaloupka 2013-11-05 14:33:20 UTC 
The fix seems to be in the ER7 release (lthon compiled sources, I've checked the decompiled jar files) but the server stucks at the start.

Adding standalone-full-ha.xml config file that simulates the problem.
Comment 4 Ondrej Chaloupka 2013-11-05 14:33:41 UTC 
Created attachment 819801 [details]
standalone-full-ha.xml
Comment 5 Ladislav Thon 2013-11-05 14:48:20 UTC 
Let me comment on this a bit too, as I was helping Ondra with the [failed] attempt to reproduce & verify.

Adding the following bits to the "infinispan" subsystem configuration in the XML makes EAP 6.2.0.ER7 hang during startup:

<cache-container name="aaa" default-cache="default" start="EAGER">
    <transport lock-timeout="60000"/>
    <replicated-cache name="default" mode="SYNC" batching="true">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="FULL_XA"/>
    </replicated-cache>
</cache-container>
<cache-container name="bbb" default-cache="default" start="EAGER">
    <transport lock-timeout="60000"/>
    <replicated-cache name="default" mode="SYNC" batching="true">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="FULL_XA"/>
    </replicated-cache>
</cache-container>
<cache-container name="ccc" default-cache="default" start="EAGER">
    <transport lock-timeout="60000"/>
    <replicated-cache name="default" mode="SYNC" batching="true">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="FULL_XA"/>
    </replicated-cache>
</cache-container>
<cache-container name="eee" default-cache="default" start="EAGER">
    <transport lock-timeout="60000"/>
    <replicated-cache name="default" mode="SYNC" batching="true">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="FULL_XA"/>
    </replicated-cache>
</cache-container>
<cache-container name="fff" default-cache="default" start="EAGER">
    <transport lock-timeout="60000"/>
    <replicated-cache name="default" mode="SYNC" batching="true">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="FULL_XA"/>
    </replicated-cache>
</cache-container>
<cache-container name="ggg" default-cache="default" start="EAGER">
    <transport lock-timeout="60000"/>
    <replicated-cache name="default" mode="SYNC" batching="true">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="FULL_XA"/>
    </replicated-cache>
</cache-container>

I'm not sure how much this is/isn't related to this particular issue, but it's definitely not OK.
Comment 6 Dimitris Andreadis 2013-11-05 17:57:11 UTC 
Paul, are you looking at it? This is the final week to get fixes for CR, which should hopefully be the last build.
Comment 8 Paul Ferraro 2013-11-07 19:51:06 UTC 
Deadlocking on startup is indeed an issue, but a separate issue nonetheless.
While caches service startup is asynchronous, the cache configuration service startup is synchronous.  The cache configuration can depend on the TransactionManager or TransactionSychronizationRegistry, which are probably blocking - causing the deadlock.  I will see if this fixes the issue.
Comment 9 Paul Ferraro 2013-11-07 20:23:06 UTC 
https://github.com/jbossas/jboss-eap/pull/678
Comment 10 Ladislav Thon 2013-11-08 09:36:16 UTC 
I built the tip of EAP branch and can confirm that the startup issue is fixed. Thanks Paul :-)
Comment 11 Ondrej Chaloupka 2013-11-11 15:26:05 UTC 
OK, the server does not stuck. It seems fine for me.
Thanks