1023469 – TimeoutException: Could not acquire lock on MarshalledValue after failover in cluster

Bug 1023469 - TimeoutException: Could not acquire lock on MarshalledValue after failover in cluster

Summary: TimeoutException: Could not acquire lock on MarshalledValue after failover in...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	JBoss Enterprise Application Platform 6
Classification:	JBoss
Component:	Clustering
Sub Component:
Version:	6.2.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	EAP 6.2.0
Assignee:	Paul Ferraro
QA Contact:	Jitka Kozana
Docs Contact:	Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-10-25 13:40 UTC by Jitka Kozana
Modified:	2013-12-16 23:31 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-12-12 16:47:46 UTC
Type:	Bug

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Jitka Kozana 2013-10-25 13:40:06 UTC

EAP 6.2.0.ER6.
Setup: 4 node cluster, cache REPL SYNC. Failover tests with failover type undeploy. 

In 4 node cluster, every node is failed and brought back once. After the last failover (4th), the first node (perf18) received an exception from the second one (perf19). Nothing to be seen on client side. 

perf18:
06:28:15,315 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf18/10.16.90.54:8009-323) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf19/ejb, see cause for remote stack trace
	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:72)
	at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:102)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
	at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208)
	at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109)
	at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:216)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:276) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.stateful.StatefulComponentIdInterceptor.processInvocation(StatefulComponentIdInterceptor.java:52) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9]
	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$$$view11.getSerialAndIncrement(Unknown Source) [clusterbench-ee6-ejb.jar:]
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) [:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:267) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:263) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:115) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]
	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) [clusterbench-ee6-ejb.jar:]
	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_WeldClientProxy.java) [clusterbench-ee6-ejb.jar:]
	at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:21)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
	at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134)
	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99)
	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92)
	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488)
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
Caused by: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=1114424370}@15656bf7 on behalf of transaction GlobalTransaction:<perf18/ejb>:735992:remote. Lock is being held by GlobalTransaction:<perf18/ejb>:156082:remote
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:218)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:211)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:175)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:124)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.lockAndRecord(OptimisticLockingInterceptor.java:254)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitSingleKeyCommand(OptimisticLockingInterceptor.java:249)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:335)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:123)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
	at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:119)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600)
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
	at org.jgroups.JChannel.up(JChannel.java:707)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)
	at org.jgroups.protocols.RSVP.up(RSVP.java:188)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453)
	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:751)
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:609)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
	at org.jgroups.protocols.FD.up(FD.java:253)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1260)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	... 1 more

perf19:
06:28:15,076 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-312,shared=udp) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=158916681}@7d669b24 on behalf of transaction GlobalTransaction:<perf18/ejb>:735779:remote. Lock is being held by GlobalTransaction:<perf18/ejb>:156080:remote
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:218) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:211) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:175) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.lockAndRecord(OptimisticLockingInterceptor.java:254) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitSingleKeyCommand(OptimisticLockingInterceptor.java:249) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:335) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:123) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:119) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.RSVP.up(RSVP.java:188) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:751) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:609) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FD.up(FD.java:253) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1260) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

Links to server logs:
perf18: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/43/artifact/report/config/jboss-perf18/server.log
perf19:https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/43/artifact/report/config/jboss-perf19/server.log
Link to job: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/43/

Comment 1 Paul Ferraro 2013-11-05 21:00:15 UTC

This is acceptable behavior in a failover scenario.

Comment 3 Paul Ferraro 2013-12-04 00:38:24 UTC

@Jitka Does this test employ mod_cluster?

Comment 4 Jitka Kozana 2013-12-05 09:35:41 UTC

No, this test uses mod_jk. But it could be easily rerun with modcluster.

Comment 5 Paul Ferraro 2013-12-12 16:47:46 UTC

mod_cluster is the only supported mechanism for clean undeploy/shutdown.  Unless your test uses mod_cluster (instead of mod_jk), you can expect these failures.

Comment 6 Radoslav Husar 2013-12-16 23:31:44 UTC

Graceful shutdown is not support in the affected version of EAP 6. The 
described behavior is to be expected.

The RFE to implement graceful shutdown is 
https://issues.jboss.org/browse/EAP6-7

Issues relating to HTTP can be avoided using mod_cluster and session 
draining prior to shutdown/undeploy.

Note You need to log in before you can comment on or make changes to this bug.