Bug 1023469
| Summary: | TimeoutException: Could not acquire lock on MarshalledValue after failover in cluster | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Jitka Kozana <jkudrnac> |
| Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> |
| Status: | CLOSED NOTABUG | QA Contact: | Jitka Kozana <jkudrnac> |
| Severity: | medium | Docs Contact: | Russell Dickenson <rdickens> |
| Priority: | unspecified | ||
| Version: | 6.2.0 | CC: | jkudrnac, rhusar, rjanik |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | EAP 6.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-12-12 16:47:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This is acceptable behavior in a failover scenario. @Jitka Does this test employ mod_cluster? No, this test uses mod_jk. But it could be easily rerun with modcluster. mod_cluster is the only supported mechanism for clean undeploy/shutdown. Unless your test uses mod_cluster (instead of mod_jk), you can expect these failures. Graceful shutdown is not support in the affected version of EAP 6. The described behavior is to be expected. The RFE to implement graceful shutdown is https://issues.jboss.org/browse/EAP6-7 Issues relating to HTTP can be avoided using mod_cluster and session draining prior to shutdown/undeploy. |
EAP 6.2.0.ER6. Setup: 4 node cluster, cache REPL SYNC. Failover tests with failover type undeploy. In 4 node cluster, every node is failed and brought back once. After the last failover (4th), the first node (perf18) received an exception from the second one (perf19). Nothing to be seen on client side. perf18: 06:28:15,315 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf18/10.16.90.54:8009-323) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf19/ejb, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:72) at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:102) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82) at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:216) at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:276) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.stateful.StatefulComponentIdInterceptor.processInvocation(StatefulComponentIdInterceptor.java:52) [jboss-as-ejb3-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.3.0.Final-redhat-9.jar:7.3.0.Final-redhat-9] at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$$$view11.getSerialAndIncrement(Unknown Source) [clusterbench-ee6-ejb.jar:] at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) [:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45] at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:267) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:263) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:115) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1] at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) [clusterbench-ee6-ejb.jar:] at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_WeldClientProxy.java) [clusterbench-ee6-ejb.jar:] at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:21) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91) at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] Caused by: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=1114424370}@15656bf7 on behalf of transaction GlobalTransaction:<perf18/ejb>:735992:remote. Lock is being held by GlobalTransaction:<perf18/ejb>:156082:remote at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:218) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:211) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:175) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:124) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.lockAndRecord(OptimisticLockingInterceptor.java:254) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitSingleKeyCommand(OptimisticLockingInterceptor.java:249) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:335) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:123) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:119) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) at org.jgroups.JChannel.up(JChannel.java:707) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025) at org.jgroups.protocols.RSVP.up(RSVP.java:188) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:751) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:609) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) at org.jgroups.protocols.FD.up(FD.java:253) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607) at org.jgroups.protocols.TP.passMessageUp(TP.java:1260) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] ... 1 more perf19: 06:28:15,076 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-312,shared=udp) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=158916681}@7d669b24 on behalf of transaction GlobalTransaction:<perf18/ejb>:735779:remote. Lock is being held by GlobalTransaction:<perf18/ejb>:156080:remote at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.newTimeoutException(AbstractTxLockingInterceptor.java:218) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.waitForTransactionsToComplete(AbstractTxLockingInterceptor.java:211) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:175) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.lockAndRecord(OptimisticLockingInterceptor.java:254) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitSingleKeyCommand(OptimisticLockingInterceptor.java:249) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:335) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:123) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:119) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.7.Final-redhat-1.jar:5.2.7.Final-redhat-1] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.RSVP.up(RSVP.java:188) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:751) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:609) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.FD.up(FD.java:253) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.TP.passMessageUp(TP.java:1260) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] Links to server logs: perf18: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/43/artifact/report/config/jboss-perf18/server.log perf19:https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/43/artifact/report/config/jboss-perf19/server.log Link to job: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/43/