901016 – (JBPAPP6-1271) CacheException: Could not prepare after failover

Bug 901016 (JBPAPP6-1271) - CacheException: Could not prepare after failover

Summary: CacheException: Could not prepare after failover

Keywords:
Status:	CLOSED NOTABUG
Alias:	JBPAPP6-1271
Product:	JBoss Enterprise Application Platform 6
Classification:	JBoss
Component:	Clustering
Sub Component:
Version:	6.1.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	EAP 6.3.0
Assignee:	Paul Ferraro
QA Contact:	Jitka Kozana
Docs Contact:
URL:	http://jira.jboss.org/jira/browse/JBP...
Whiteboard:
Duplicates (1):	JBPAPP6-1695 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-10-05 14:27 UTC by Jitka Kozana
Modified:	2014-04-23 02:37 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2014-02-10 19:22:06 UTC
Type:	Bug
Embargoed:

Attachments	(Terms of Use)
Comment (64.13 KB, text/plain) 2013-03-04 13:07 UTC, Ladislav Thon	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	JBPAPP6-1271	0	Major	Closed	CacheException: Could not prepare after network failover	2018-04-09 05:51:08 UTC

Description Jitka Kozana 2012-10-05 14:27:23 UTC

project_key: JBPAPP6

After restoring the network traffic (all multicast traffic was blocked and then brought back up), we are seeing many infinispan exceptions, which decreases cluster responsivnes.
{code}
08:00:34,249 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (notification-thread-15) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 13
	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:201)
	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:81)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:132)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:260)
	at org.jboss.as.clustering.registry.RegistryService.viewChanged(RegistryService.java:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_30]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_30]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_30]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_30]
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:200)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{code}
followed by: 
{code}

08:00:34,256 ERROR [org.infinispan.transaction.tm.DummyTransaction] (notification-thread-15) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=19604}, status=1}, lockedKeys=null, backupKeyLocks=null, viewId=5} org.infinispan.transaction.synchronization.SyncLocalTransaction@4c8e} org.infinispan.transaction.synchronization.SynchronizationAdapter@4cad: org.infinispan.CacheException: Could not prepare. 
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.registry.RegistryService.invoke(RegistryService.java:260)
	at org.jboss.as.clustering.registry.RegistryService.viewChanged(RegistryService.java:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_30]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_30]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_30]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_30]
	at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:200)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
Caused by: javax.transaction.xa.XAException
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	... 19 more
{code}
 and 
{code}
08:00:34,258 WARN  [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] (notification-thread-15) ISPN000134: Unable to invoke method public void org.jboss.as.clustering.registry.RegistryService.viewChanged(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent) on Object instance org.jboss.as.clustering.registry.RegistryService@4c5d1e86 - removing this target object from list of listeners!: org.infinispan.CacheException: Unable to end batch
{code}

The network was restored at 07:59:55:001, see the full server log here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-http-session-netDown-repl-sync/1/artifact/report/config/jboss-perf18/server.log

Comment 1 Jitka Kozana 2012-11-05 10:53:18 UTC

With ASYNC we are seeing "Could not commit" instead of "Could not prepare".

Comment 2 Anne-Louise Tangring 2012-11-13 20:53:12 UTC

Docs QE Status: Removed: NEW

Comment 3 Ladislav Thon 2012-11-30 15:28:30 UTC

Affects 6.0.1.ER4.1 as well. Added 6.0.1.ER4 to _Affects Version/s_, because we don't have 6.0.1.ER4.1 in JIRA (oh those point ER releases...).

Comment 4 Ladislav Thon 2012-12-03 10:23:41 UTC

I'm also seeing these exceptions, but I believe that they are only a slightly different manifestation of this exact issue:

{code}
10:45:40,073 ERROR [org.infinispan.transaction.TransactionCoordinator] (EJB default - 1) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 34
	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:201)
	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:81)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:132)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
	at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
	at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:615)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.insert(InfinispanBackingCacheEntryStore.java:180)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.insert(SerializationGroupMemberContainer.java:304)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.insert(SerializationGroupMemberContainer.java:51)
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.create(PassivatingBackingCacheImpl.java:122)
	at org.jboss.as.ejb3.cache.impl.GroupAwareCache.create(GroupAwareCache.java:67)
	at org.jboss.as.ejb3.cache.impl.GroupAwareCache.create(GroupAwareCache.java:41)
	at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSession(StatefulSessionComponent.java:241)
	at org.jboss.as.ejb3.remote.protocol.versionone.SessionOpenRequestHandler$SessionIDGeneratorTask.run(SessionOpenRequestHandler.java:146)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_30]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
{code}

See https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-netDown-repl-async/9/console-perf21/

Comment 5 Ladislav Thon 2013-03-01 12:22:41 UTC

Seeing this on EAP 6.1.0.ER1 too (with a failure type of undeploy):

ERROR [org.infinispan.transaction.tm.DummyTransaction] (EJB default - 4) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MarshalledValue{instance={[12, -6, -102, 85, -30, 33, 69, -91, -123, 123, 79, -106, 95, -26, 99, 58]}, serialized=ByteArray{size=128, array=0x0301fe03ff810200..}, cachedHashCode=1815227588}@3a673b65], topologyId=7, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@11daa} org.infinispan.transaction.synchronization.SynchronizationAdapter@11dc9: org.infinispan.CacheException: Could not prepare. 
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:661)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:214)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:65) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:321) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_38]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_38]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: javax.transaction.xa.XAException
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	... 58 more

Full log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/19/console-perf21/

Comment 6 Richard Janík 2013-03-04 12:42:10 UTC

And also seen with jvmkill scenario:

[JBossINF] [0m[31m07:24:13,581 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-perf18/10.16.90.54:8009-42) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[AvHWeDC9n6p5RRSmQrAWbTPC], topologyId=13, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@3f95} org.infinispan.transaction.synchronization.SynchronizationAdapter@3fb4: org.infinispan.CacheException: Could not prepare. 
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
[JBossINF] 	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
[JBossINF] 	at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1527) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:142) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
[JBossINF] Caused by: javax.transaction.xa.XAException
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
[JBossINF] 	... 21 more

Full log:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-jvmkill-repl-sync/47/console-perf18/

Comment 7 Ladislav Thon 2013-03-04 13:07:49 UTC

Created attachment 915677 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).

Comment 8 Jitka Kozana 2013-03-07 06:46:16 UTC

Affects 6.1.0.ER2, basically the same stuff as Ladislav wrote in comment #7.

See server logs here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/16/artifact/report/config/jboss-perf19/server.log
Link to job:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/16/

Comment 9 Ladislav Thon 2013-03-21 14:04:05 UTC

*** Bug 901245 has been marked as a duplicate of this bug. ***

Comment 10 Ladislav Thon 2013-05-14 12:48:47 UTC

Still seeing this in EAP 6.1.0.ER8. For example:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-repl-sync/54/artifact/report/config/jboss-perf21/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/26/artifact/report/config/jboss-perf21/server.log

Comment 11 Jitka Kozana 2013-08-22 07:38:49 UTC

Seen again during EAP 6.1.1.ER7 testing. 

Originally, the issue was seen only in network failover tests (e.g. where failure type was network outage), however, this issue started to appear in tests with graceful shutdown too. I have updated the title to better reflect this new status.

Link to job (graceful shutdown, cache REPL SYNC):
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-repl-sync/60/

Comment 12 Paul Ferraro 2013-08-29 18:57:12 UTC

This could be fixed by upgrading to Infinispan 5.3 and replacing the @ViewChanged listener in RegistryService with a @TopologyChanged listener.  This is the approach taken upstream.  Recheduling for 6.3.

Comment 13 Paul Ferraro 2014-02-10 19:22:06 UTC

The "netDown" scenario should not be used during functional testing.  EAP will never handle this scenario perfectly.  The attached stacktrace can be expected following this scenario.

Comment 14 Scott Mumford 2014-04-23 02:37:31 UTC

Marking for exclusion from Release Notes documentation as not a bug

Note You need to log in before you can comment on or make changes to this bug.