Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 901016 (JBPAPP6-1271)

Summary: CacheException: Could not prepare after failover
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Jitka Kozana <jkudrnac>
Component: ClusteringAssignee: Paul Ferraro <paul.ferraro>
Status: CLOSED NOTABUG QA Contact: Jitka Kozana <jkudrnac>
Severity: high Docs Contact:
Priority: high    
Version: 6.1.1CC: jkudrnac, lthon, myarboro, rjanik, santosh.sharma, smumford, tremes
Target Milestone: ---   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/JBPAPP6-1271
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-10 19:22:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Comment none

Description Jitka Kozana 2012-10-05 14:27:23 UTC 
project_key: JBPAPP6

After restoring the network traffic (all multicast traffic was blocked and then brought back up), we are seeing many infinispan exceptions, which decreases cluster responsivnes.
{code}
08:00:34,249 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (notification-thread-15) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 13
	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:201)
	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:81)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:132)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:260)
	at org.jboss.as.clustering.registry.RegistryService.viewChanged(RegistryService.java:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_30]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_30]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_30]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_30]
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:200)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{code}
followed by: 
{code}

08:00:34,256 ERROR [org.infinispan.transaction.tm.DummyTransaction] (notification-thread-15) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=19604}, status=1}, lockedKeys=null, backupKeyLocks=null, viewId=5} org.infinispan.transaction.synchronization.SyncLocalTransaction@4c8e} org.infinispan.transaction.synchronization.SynchronizationAdapter@4cad: org.infinispan.CacheException: Could not prepare. 
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:260)
	at org.jboss.as.clustering.registry.RegistryService.viewChanged(RegistryService.java:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_30]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_30]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_30]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_30]
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:200)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
Caused by: javax.transaction.xa.XAException
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	... 19 more
{code}
 and 
{code}
08:00:34,258 WARN  [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] (notification-thread-15) ISPN000134: Unable to invoke method public void org.jboss.as.clustering.registry.RegistryService.viewChanged(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent) on Object instance org.jboss.as.clustering.registry.RegistryService@4c5d1e86 - removing this target object from list of listeners!: org.infinispan.CacheException: Unable to end batch
{code}

The network was restored at 07:59:55:001, see the full server log here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-http-session-netDown-repl-sync/1/artifact/report/config/jboss-perf18/server.log
Comment 1 Jitka Kozana 2012-11-05 10:53:18 UTC 
With ASYNC we are seeing "Could not commit" instead of "Could not prepare".
Comment 2 Anne-Louise Tangring 2012-11-13 20:53:12 UTC 
Docs QE Status: Removed: NEW
Comment 3 Ladislav Thon 2012-11-30 15:28:30 UTC 
Affects 6.0.1.ER4.1 as well. Added 6.0.1.ER4 to _Affects Version/s_, because we don't have 6.0.1.ER4.1 in JIRA (oh those point ER releases...).
Comment 4 Ladislav Thon 2012-12-03 10:23:41 UTC 
I'm also seeing these exceptions, but I believe that they are only a slightly different manifestation of this exact issue:

{code}
10:45:40,073 ERROR [org.infinispan.transaction.TransactionCoordinator] (EJB default - 1) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 34
	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:201)
	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:81)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:132)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
	at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
	at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.insert(InfinispanBackingCacheEntryStore.java:180)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.insert(SerializationGroupMemberContainer.java:304)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.insert(SerializationGroupMemberContainer.java:51)
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.create(PassivatingBackingCacheImpl.java:122)
	at org.jboss.as.ejb3.cache.impl.GroupAwareCache.create(GroupAwareCache.java:67)
	at org.jboss.as.ejb3.cache.impl.GroupAwareCache.create(GroupAwareCache.java:41)
	at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSession(StatefulSessionComponent.java:241)
	at org.jboss.as.ejb3.remote.protocol.versionone.SessionOpenRequestHandler$SessionIDGeneratorTask.run(SessionOpenRequestHandler.java:146)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_30]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
{code}

See https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-netDown-repl-async/9/console-perf21/
Comment 5 Ladislav Thon 2013-03-01 12:22:41 UTC 
Seeing this on EAP 6.1.0.ER1 too (with a failure type of undeploy):

ERROR [org.infinispan.transaction.tm.DummyTransaction] (EJB default - 4) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MarshalledValue{instance={[12, -6, -102, 85, -30, 33, 69, -91, -123, 123, 79, -106, 95, -26, 99, 58]}, serialized=ByteArray{size=128, array=0x0301fe03ff810200..}, cachedHashCode=1815227588}@3a673b65], topologyId=7, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@11daa} org.infinispan.transaction.synchronization.SynchronizationAdapter@11dc9: org.infinispan.CacheException: Could not prepare. 
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:661)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:214)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:65) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:321) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_38]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_38]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: javax.transaction.xa.XAException
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	... 58 more

Full log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/19/console-perf21/
Comment 6 Richard Janík 2013-03-04 12:42:10 UTC 
And also seen with jvmkill scenario:

[JBossINF] [0m[31m07:24:13,581 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-perf18/10.16.90.54:8009-42) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[AvHWeDC9n6p5RRSmQrAWbTPC], topologyId=13, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@3f95} org.infinispan.transaction.synchronization.SynchronizationAdapter@3fb4: org.infinispan.CacheException: Could not prepare. 
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
[JBossINF] 	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
[JBossINF] 	at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1527) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:142) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
[JBossINF] Caused by: javax.transaction.xa.XAException
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
[JBossINF] 	... 21 more

Full log:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-jvmkill-repl-sync/47/console-perf18/
Comment 7 Ladislav Thon 2013-03-04 13:07:49 UTC 
Created attachment 915677 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Comment 8 Jitka Kozana 2013-03-07 06:46:16 UTC 
Affects 6.1.0.ER2, basically the same stuff as Ladislav wrote in comment #7.

See server logs here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/16/artifact/report/config/jboss-perf19/server.log
Link to job:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/16/
Comment 9 Ladislav Thon 2013-03-21 14:04:05 UTC 
*** Bug 901245 has been marked as a duplicate of this bug. ***
Comment 10 Ladislav Thon 2013-05-14 12:48:47 UTC 
Still seeing this in EAP 6.1.0.ER8. For example:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-repl-sync/54/artifact/report/config/jboss-perf21/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/26/artifact/report/config/jboss-perf21/server.log
Comment 11 Jitka Kozana 2013-08-22 07:38:49 UTC 
Seen again during EAP 6.1.1.ER7 testing. 

Originally, the issue was seen only in network failover tests (e.g. where failure type was network outage), however, this issue started to appear in tests with graceful shutdown too. I have updated the title to better reflect this new status.

Link to job (graceful shutdown, cache REPL SYNC):
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-repl-sync/60/
Comment 12 Paul Ferraro 2013-08-29 18:57:12 UTC 
This could be fixed by upgrading to Infinispan 5.3 and replacing the @ViewChanged listener in RegistryService with a @TopologyChanged listener.  This is the approach taken upstream.  Recheduling for 6.3.
Comment 13 Paul Ferraro 2014-02-10 19:22:06 UTC 
The "netDown" scenario should not be used during functional testing.  EAP will never handle this scenario perfectly.  The attached stacktrace can be expected following this scenario.
Comment 14 Scott Mumford 2014-04-23 02:37:31 UTC 
Marking for exclusion from Release Notes documentation as not a bug