Bug 1022602

Summary:	Intermittent failures in ReplicationWebFailoverTestCase
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Richard Janík <rjanik>
Component:	Clustering	Assignee:	Paul Ferraro <paul.ferraro>
Status:	CLOSED EOL	QA Contact:	Michal Vinkler <mvinkler>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.2.0
Target Milestone:	---
Target Release:	EAP 6.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	Clustering testsuite
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-19 12:48:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Richard Janík 2013-10-23 15:31:14 UTC

Description of problem:

EAP 6.2.0.ER6

There are occasional failures in org.jboss.as.test.clustering.cluster.web.ReplicationWebFailoverTestCase(ASYNC-udp), the tests in question are:
testGracefulSimpleFailover
testGracefulUndeployFailover
testCleanup

testCleanup failed due to: "java.lang.IllegalArgumentException: Deployment with name deployment-0 could not be undeployed. Container container-0 must be still running.", not much there.
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/lastCompletedBuild/jdk=ibm17,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.web/ReplicationWebFailoverTestCase%28ASYNC-udp%29/testCleanup/


testGracefulSimpleFailover and testGracefulUndeployFailover fail with


java.lang.AssertionError: Session failed to replicate after container 1 was shutdown. expected:<3> but was:<2>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.jboss.as.test.clustering.cluster.web.ClusteredWebFailoverAbstractCase.testGracefulSimpleFailover(ClusteredWebFailoverAbstractCase.java:125)
	...


or


org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:8080 refused
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at org.jboss.as.test.clustering.ClusterHttpClientUtil.establishView(ClusterHttpClientUtil.java:51)
	at org.jboss.as.test.clustering.cluster.web.ClusteredWebFailoverAbstractCase.establishView(ClusteredWebFailoverAbstractCase.java:284)
	at org.jboss.as.test.clustering.cluster.web.ClusteredWebFailoverAbstractCase.testGracefulUndeployFailover(ClusteredWebFailoverAbstractCase.java:198)
	...


respectively.


https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/lastCompletedBuild/jdk=ibm17,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.web/ReplicationWebFailoverTestCase%28ASYNC-udp%29/testGracefulSimpleFailover/
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/lastCompletedBuild/jdk=ibm17,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.web/ReplicationWebFailoverTestCase%28ASYNC-udp%29/testGracefulUndeployFailover/


The standard output contains an exception as well, the same one for both tests:


&#27;[0m&#27;[31m00:31:11,674 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Incoming-4,shared=udp) ISPN000136: Execution error: org.infinispan.transaction.xa.InvalidTransactionException: This remote transaction GlobalTransaction:<node-0/web>:2:remote is already rolled back
	at org.infinispan.transaction.RemoteTransaction.checkIfRolledBack(RemoteTransaction.java:138) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.transaction.RemoteTransaction.putLookedUpEntry(RemoteTransaction.java:73) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.context.impl.RemoteTxInvocationContext.putLookedUpEntry(RemoteTxInvocationContext.java:99) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.container.EntryFactoryImpl.wrapInternalCacheEntryForPut(EntryFactoryImpl.java:242) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:166) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:330) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:103) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:119) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.RSVP.up(RSVP.java:188) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:797) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:645) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FD.up(FD.java:253) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1260) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156) [rt.jar:1.7.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626) [rt.jar:1.7.0]
	at java.lang.Thread.run(Thread.java:780) [vm.jar:1.7.0]

&#27;[0m&#27;[31m00:31:11,768 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (Incoming-4,shared=udp) Exception executing command: org.infinispan.transaction.xa.InvalidTransactionException: This remote transaction GlobalTransaction:<node-0/web>:2:remote is already rolled back
	at org.infinispan.transaction.RemoteTransaction.checkIfRolledBack(RemoteTransaction.java:138) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.transaction.RemoteTransaction.putLookedUpEntry(RemoteTransaction.java:73) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.context.impl.RemoteTxInvocationContext.putLookedUpEntry(RemoteTxInvocationContext.java:99) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.container.EntryFactoryImpl.wrapInternalCacheEntryForPut(EntryFactoryImpl.java:242) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:166) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor$EntryWrappingVisitor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:330) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:103) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:119) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.RSVP.up(RSVP.java:188) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:797) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:645) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FD.up(FD.java:253) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1260) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156) [rt.jar:1.7.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626) [rt.jar:1.7.0]
	at java.lang.Thread.run(Thread.java:780) [vm.jar:1.7.0]

Comment 1 Richard Janík 2013-10-23 15:39:43 UTC

Uh, the links are dangerous - they take you to the last completed build rather than to the build in question and also contain views that might change in the future. Sorry for that, here are safer versions:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/67/jdk=ibm17,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.web/ReplicationWebFailoverTestCase%28ASYNC-udp%29/testCleanup/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/67/jdk=ibm17,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.web/ReplicationWebFailoverTestCase%28ASYNC-udp%29/testGracefulSimpleFailover/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/67/jdk=ibm17,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.web/ReplicationWebFailoverTestCase%28ASYNC-udp%29/testGracefulUndeployFailover/

Comment 3 Paul Ferraro 2014-09-02 14:14:40 UTC

These failures are simply due to this failover test assuming SYNC replication.  Running with ASYNC requires waiting for some configurable time period prior to shutting down a server to increase the probability that the session replicates in time.