Bug 901245 (JBPAPP6-1695)
| Summary: | Strange XA-related exceptions after network failure | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Ladislav Thon <lthon> |
| Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> |
| Status: | CLOSED DUPLICATE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.1.0 | CC: | jkudrnac, lthon, rsvoboda |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | http://jira.jboss.org/jira/browse/JBPAPP6-1695 | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-03-21 14:04:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hmm, looks like it doesn't only happen after network failure, also when shutting down EAP properly. See https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-shutdown-repl-sync/16/console-perf20/ Affects EAP 6.1.0.ER1. Also seen this kind of exceptions in a scenario where the failure type is undeploy: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/19/console-perf20/ Maybe this is a dupe of 901016. So I'm confident this is actually a dupe of 901016 and 901161. Marking as such. *** This bug has been marked as a duplicate of bug 901016 *** |
project_key: JBPAPP6 After network failure, Infinispan obviously isn't able to end a transaction successfully. I've seen this situation resulting in a sequence of strange exceptions: {code} 10:15:10,551 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EJB default - 4) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [{[43, 126, 22, -68, 108, 112, 70, 102, -125, -22, -2, -107, -70, -39, -30, 59]}] for requestor [GlobalTransaction:<perf20/ejb>:136789:remote]! Lock held by [GlobalTransaction:<perf19/ejb>:118557:local] at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:216) at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:189) at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:179) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:209) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:136) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:324) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:119) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:213) at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:85) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:132) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:70) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:128) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:459) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:366) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:602) at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) at org.jgroups.JChannel.up(JChannel.java:716) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) at org.jgroups.protocols.RSVP.up(RSVP.java:188) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD.up(FD.java:273) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.stack.Protocol.up(Protocol.java:363) at org.jgroups.protocols.TP.passMessageUp(TP.java:1185) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1733) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1715) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [... more like this ...] 10:15:10,572 ERROR [org.infinispan.transaction.tm.DummyTransaction] (EJB default - 3) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=136798}, status=1}, lockedKeys=null, backupKeyLocks=[MarshalledValue{instance={[-32, -31, 24, 96, -106, -76, 78, 24, -127, -60, 47, 35, -79, -4, 48, 78]}, serialized=ByteArray{size=128, array=0x0301fe03ff810200..}, cachedHashCode=796054578}@284071f3], viewId=6} org.infinispan.transaction.synchronization.SyncLocalTransaction@2165a} org.infinispan.transaction.synchronization.SynchronizationAdapter@21679: org.infinispan.CacheException: Could not prepare. at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81) at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:218) at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:90) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:230) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:302) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:188) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:43) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:321) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_30] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] at org.jboss.threads.JBossThread.run(JBossThread.java:122) Caused by: javax.transaction.xa.XAException at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) ... 58 more [... more like this ...] 10:15:10,582 WARN [com.arjuna.ats.jta] (EJB default - 3) ARJUNA016029: SynchronizationImple.afterCompletion - failed for org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization@50599b59 with exception: org.infinispan.CacheException: Unable to end batch at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:107) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81) at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:218) at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:90) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:230) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:302) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:188) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:43) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:321) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_30] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] at org.jboss.threads.JBossThread.run(JBossThread.java:122) Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) ... 52 more [... more like this ...] {code} Seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/10/console-perf20/ After speaking about this with [~rhusar] (as I found the same exceptions in his gist https://gist.github.com/4079891), I agree that the exceptions kinda sorta make sense, but I still believe that they are quite hard to decipher. Therefore filing a minor priority bug. If we aren't able to make this better, feel free to reject.