Hide Forgot
XAResourceRecoveryRegistry.removeXAResourceRecovery is a blocking API, so MSC services that call it in start()/stop() must use Start|StopContext.asynchronous(). Services must not do blocking tasks with the MSC thread that calls the lifecycle methods, as discussed in the class javadoc at http://grepcode.com/file/repository.jboss.org/nexus/content/repositories/releases/org.jboss.msc/jboss-msc/1.0.0.Beta8/org/jboss/msc/service/Service.java. This is because the MSC thread pool is a very limited resource (4 threads), and blocking those threads has the potential to lock up the entire service container. The issue discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1020209 shows that XAResourceRecoveryRegistry.removeXAResourceRecovery is a blocking call. From discussions with Tom Jenkinson it seems some sort of blocking behavior existed all the way back to 6.0. The fix is to use the MSC asynchronous APIs in the relevant methods. Backport of fix for https://issues.jboss.org/browse/WFLY-2350.
https://github.com/jbossas/jboss-eap/pull/621 completes most of this, but https://github.com/wildfly/wildfly/pull/5330 needs to be backported.
https://github.com/jbossas/jboss-eap/pull/641
The fix seems to be in the ER7 release (lthon compiled sources, I've checked the decompiled jar files) but the server stucks at the start. Adding standalone-full-ha.xml config file that simulates the problem.
Created attachment 819801 [details] standalone-full-ha.xml
Let me comment on this a bit too, as I was helping Ondra with the [failed] attempt to reproduce & verify. Adding the following bits to the "infinispan" subsystem configuration in the XML makes EAP 6.2.0.ER7 hang during startup: <cache-container name="aaa" default-cache="default" start="EAGER"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC" batching="true"> <locking isolation="REPEATABLE_READ"/> <transaction mode="FULL_XA"/> </replicated-cache> </cache-container> <cache-container name="bbb" default-cache="default" start="EAGER"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC" batching="true"> <locking isolation="REPEATABLE_READ"/> <transaction mode="FULL_XA"/> </replicated-cache> </cache-container> <cache-container name="ccc" default-cache="default" start="EAGER"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC" batching="true"> <locking isolation="REPEATABLE_READ"/> <transaction mode="FULL_XA"/> </replicated-cache> </cache-container> <cache-container name="eee" default-cache="default" start="EAGER"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC" batching="true"> <locking isolation="REPEATABLE_READ"/> <transaction mode="FULL_XA"/> </replicated-cache> </cache-container> <cache-container name="fff" default-cache="default" start="EAGER"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC" batching="true"> <locking isolation="REPEATABLE_READ"/> <transaction mode="FULL_XA"/> </replicated-cache> </cache-container> <cache-container name="ggg" default-cache="default" start="EAGER"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC" batching="true"> <locking isolation="REPEATABLE_READ"/> <transaction mode="FULL_XA"/> </replicated-cache> </cache-container> I'm not sure how much this is/isn't related to this particular issue, but it's definitely not OK.
Paul, are you looking at it? This is the final week to get fixes for CR, which should hopefully be the last build.
Deadlocking on startup is indeed an issue, but a separate issue nonetheless. While caches service startup is asynchronous, the cache configuration service startup is synchronous. The cache configuration can depend on the TransactionManager or TransactionSychronizationRegistry, which are probably blocking - causing the deadlock. I will see if this fixes the issue.
https://github.com/jbossas/jboss-eap/pull/678
I built the tip of EAP branch and can confirm that the startup issue is fixed. Thanks Paul :-)
OK, the server does not stuck. It seems fine for me. Thanks