Bug 1119350 - RollbackCommands should ignore leavers
Summary: RollbackCommands should ignore leavers
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: CR4
: 6.3.0
Assignee: Tristan Tarrant
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks: 1104639
TreeView+ depends on / blocked
 
Reported: 2014-07-14 14:58 UTC by Dan Berindei
Modified: 2015-01-26 14:03 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-26 14:03:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-4517 0 Critical Resolved RollbackCommands should ignore leavers 2014-08-04 14:31:23 UTC

Description Dan Berindei 2014-07-14 14:58:38 UTC
When one of the targets of a PrepareCommand leaves, the originator receives a SuspectException and tries to roll back the transaction. However, the 

RollbackCommand can also fail with with a SuspectException if:
* syncRollbackPhase = true (the default, since ISPN-4137)
* The cache topology hasn't been updated to exclude the leaver yet (maybe because it was the old coordinator that left)

In that case, we could throw a SuspectException in JGroupsTransport.invokeRemotely without sending the RollbackCommand to the other owner:
23:34:01,219 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (PrivateLogChecker-4) ISPN000136: Execution error
org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command RollbackCommand {gtx=GlobalTransaction:<edg-perf08-52473>:1077:local, cacheName='testCache', topologyId=8}
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:486)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
	at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:223)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:101)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:51)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:50)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:207)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:101)
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:66)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:222)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:153)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:91)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:101)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
	at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:101)
	at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
	at org.infinispan.transaction.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:237)
	at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:172)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:104)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:44)
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1436)
	at org.infinispan.CacheImpl.putInternal(CacheImpl.java:904)
	at org.infinispan.CacheImpl.put(CacheImpl.java:896)
	at org.infinispan.CacheImpl.put(CacheImpl.java:1471)
	at org.infinispan.CacheImpl.put(CacheImpl.java:231)
	at org.radargun.cachewrappers.InfinispanBasicOperations.put(InfinispanBasicOperations.java:25)
	at org.radargun.cachewrappers.Infinispan51BasicOperations.put(Infinispan51BasicOperations.java:31)
	at org.radargun.cachewrappers.Infinispan52BasicOperations.put(Infinispan52BasicOperations.java:16)
	at org.radargun.cachewrappers.InfinispanWrapper.put(InfinispanWrapper.java:185)
	at org.radargun.stressors.LogChecker.run(LogChecker.java:106)

The RollbackCommand should have the SYNCHRONOUS_IGNORE_LEAVERS ResponseMode, so that the owner still alive receives the the command. Otherwise, that stale transaction will never be completed.

Comment 2 Alan Field 2014-07-15 13:36:14 UTC
Executed the elasticity test in Hyperion 7 times without a failure, and the resilience test 13 times without a failure with JDG 6.3.0 CR4. VERIFIED


Note You need to log in before you can comment on or make changes to this bug.