Bug 949440
Summary: | Intermittent failure of SingletonTestCase.testSingletonService | ||||||
---|---|---|---|---|---|---|---|
Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Richard Janík <rjanik> | ||||
Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> | ||||
Status: | CLOSED EOL | QA Contact: | Petr Kremensky <pkremens> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 6.1.0, 6.2.0, 6.1.1, 6.3.0 | CC: | bbaranow, jkudrnac, pkremens, rhusar | ||||
Target Milestone: | ER3 | Keywords: | Reopened | ||||
Target Release: | EAP 6.4.0 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | Clustering testsuite | ||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2019-08-19 12:47:55 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 996500 | ||||||
Attachments: |
|
Description
Richard Janík
2013-04-08 08:02:18 UTC
The ISE from the testing harness is not very interesting in this context, the actual issue is a timeout on sending a message: 03:44:35,963 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-2) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to node-1/singleton As these tests rely heavily on the network, this seems be just environmental. Also seen in 6.1.1.ER3. Test fails, either due to container not comming up or some sort deadlock/timeout( it seems). However there is one familiar thing: Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) ... 13 more This may or not be related to: https://issues.jboss.org/browse/WFLY-1507 Test output: Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 66.625 sec <<< FAILURE! testSingletonService(org.jboss.as.test.clustering.cluster.singleton.SingletonTestCase) Time elapsed: 46.205 sec <<< ERROR! java.lang.IllegalArgumentException: Deployment with name deployment-1 could not be undeployed. Container container-1 must be still running. at org.jboss.arquillian.container.test.impl.client.deployment.ClientDeployer.undeploy(ClientDeployer.java:128) at org.jboss.as.test.clustering.cluster.singleton.SingletonTestCase.testSingletonService(SingletonTestCase.java:204) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.jboss.arquillian.junit.Arquillian$6$1.invoke(Arquillian.java:270) at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:53) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:89) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:75) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:60) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111) at org.jboss.arquillian.junit.Arquillian$6.evaluate(Arquillian.java:263) at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:226) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:240) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:185) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:199) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:147) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) Log: java:global/singleton/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:app/singleton/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:module/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:jboss/exported/singleton/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:global/singleton/ViewChangeListenerBean java:app/singleton/ViewChangeListenerBean java:module/ViewChangeListenerBean [0m[0m15:35:17,014 INFO [org.jboss.web] (ServerService Thread Pool -- 55) JBAS018210: Register web context: /singleton [0m[0m15:35:17,528 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000078: Starting JGroups Channel [0m[0m15:35:17,548 INFO [stdout] (ServerService Thread Pool -- 52) [0m[0m15:35:17,549 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m15:35:17,549 INFO [stdout] (ServerService Thread Pool -- 52) GMS: address=node-1/singleton, cluster=singleton, physical address=127.0.0.1:7700 [0m[0m15:35:17,549 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m15:35:17,677 INFO [org.jboss.as.clustering] (Incoming-8,shared=tcp) JBAS010225: New cluster view for partition singleton (id: 3, delta: 1, merge: false) : [node-0/singleton, node-1/singleton] [0m[0m15:35:17,678 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-8,shared=tcp) ISPN000094: Received new cluster view: [node-0/singleton|3] [node-0/singleton, node-1/singleton] [0m[0m15:35:17,693 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000094: Received new cluster view: [node-0/singleton|3] [node-0/singleton, node-1/singleton] [0m[0m15:35:17,750 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000079: Cache local address is node-1/singleton, physical addresses are [127.0.0.1:7700] [0m[0m15:35:17,757 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 52) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final [0m[0m15:35:17,807 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 52) ISPN000161: Using a batchMode transaction manager [0m[0m15:35:17,948 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 52) ISPN000031: MBeans were successfully registered to the platform MBean server. [0m[0m15:35:18,097 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 52) JBAS010281: Started default cache from singleton container [0m[0m15:35:18,110 INFO [org.jboss.as.clustering] (MSC service thread 1-8) JBAS010238: Number of cluster members: 2 [0m[0m15:35:18,133 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: node-1/singleton elected as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,134 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010341: This node will no longer operate as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,138 INFO [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 52) JBAS010342: node-1/singleton elected as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,139 INFO [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 52) JBAS010340: This node will now operate as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,197 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "singleton.war" (runtime-name : "singleton.war") [0m[0m15:35:18,292 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10090/management [0m[0m15:35:18,293 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10090 [0m[0m15:35:18,294 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.1.1.Alpha1 (AS 7.2.2-internal-SNAPSHOT) started in 5894ms - Started 185 of 302 services (115 services are passive or on-demand) [0m[0m15:35:18,486 INFO [stdout] (http-/127.0.0.1:8080-1) Cluster 'singleton' successfully established view [node-0/singleton, node-1/singleton] within 0 ms. [0m15:35:18,694 INFO [org.jboss.arquillian.container.test.impl.client.container.ClientContainerController] (main) Manual stopping of a server instance [0m15:35:18,721 INFO [org.apache.coyote.http11] (MSC service thread 1-7) JBWEB003075: Coyote HTTP/1.1 pausing on: http-/127.0.0.1:8180 [0m[0m15:35:18,722 INFO [org.apache.coyote.http11] (MSC service thread 1-7) JBWEB003077: Coyote HTTP/1.1 stopping on : http-/127.0.0.1:8180 [0m[0m15:35:18,733 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 11) MODCLUSTER000002: Initiating mod_cluster shutdown [0m[0m15:35:18,739 INFO [org.apache.catalina.core] (MSC service thread 1-4) JBWEB001079: Container org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/] has not been started [0m[0m15:35:18,750 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: node-0/singleton elected as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,751 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010340: This node will now operate as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,758 INFO [org.jboss.web] (ServerService Thread Pool -- 11) JBAS018224: Unregister web context: /singleton [0m[0m15:35:18,769 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010409: Unbound data source [java:jboss/datasources/ExampleDS] [0m[0m15:35:18,772 INFO [org.apache.coyote.ajp] (MSC service thread 1-5) JBWEB003048: Pausing Coyote AJP/1.3 on ajp-/127.0.0.1:8109 [0m[0m15:35:18,774 INFO [org.apache.coyote.ajp] (MSC service thread 1-5) JBWEB003051: Stopping Coyote AJP/1.3 on ajp-/127.0.0.1:8109 [0m[0m15:35:18,813 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped default cache from singleton container [0m[0m15:35:18,817 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting and closing JGroups Channel [0m[0m15:35:18,854 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment singleton.war (runtime-name: singleton.war) in 144ms [0m[0m15:35:19,129 INFO [org.jboss.as.clustering] (Incoming-15,shared=tcp) JBAS010225: New cluster view for partition singleton (id: 4, delta: -1, merge: false) : [node-0/singleton] [0m[0m15:35:19,130 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-15,shared=tcp) ISPN000094: Received new cluster view: [node-0/singleton|4] [node-0/singleton] [0m[0m15:35:19,155 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher [0m[0m15:35:19,163 INFO [org.jboss.as] (MSC service thread 1-4) JBAS015950: JBoss EAP 6.1.1.Alpha1 (AS 7.2.2-internal-SNAPSHOT) stopped in 413ms [0m[0m15:35:19,183 INFO [org.jboss.as.clustering] (Incoming-17,shared=tcp) JBAS010225: New cluster view for partition singleton (id: 5, delta: 1, merge: false) : [node-0/singleton, node-1/singleton] [0m[0m15:35:19,183 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,shared=tcp) ISPN000094: Received new cluster view: [node-0/singleton|5] [node-0/singleton, node-1/singleton] [0m15:35:19,522 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopped deployment=NONE, container=container-1 [0m15:35:19,525 INFO [stdout] (http-/127.0.0.1:8080-1) [node-0/singleton] != [node-0/singleton, node-1/singleton], waiting for a view change event... [0m[33m15:35:24,185 WARN [org.jgroups.protocols.pbcast.GMS] (ViewHandler,singleton,node-0/singleton) node-0/singleton: failed to collect all ACKs (expected=1) for unicast view [node-0/singleton|5] [node-0/singleton, node-1/singleton] after 5000ms, missing ACKs from [node-1/singleton] [0m[31m15:35:34,552 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/singleton].[org.jboss.as.test.clustering.ViewChangeListenerServlet]] (http-/127.0.0.1:8080-1) JBWEB000236: Servlet.service() for servlet org.jboss.as.test.clustering.ViewChangeListenerServlet threw exception: java.lang.InterruptedException: Cluster 'singleton' failed to establish view [node-0/singleton] within 15000 ms. Current view is: [node-0/singleton, node-1/singleton] at org.jboss.as.test.clustering.ViewChangeListenerBean.establishView(ViewChangeListenerBean.java:77) [classes:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:248) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:249) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocation(EJBObjectInterceptor.java:58) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocation(EJBHomeInterceptor.java:83) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:42) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:125) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:253) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at com.sun.proxy.$Proxy30.establishView(Unknown Source) at org.jboss.as.test.clustering.ViewChangeListenerServlet.doGet(ViewChangeListenerServlet.java:70) [classes:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final.jar:1.0.2.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final.jar:1.0.2.Final] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91) at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] [0m[0m15:35:34,606 INFO [org.jboss.web] (ServerService Thread Pool -- 53) JBAS018224: Unregister web context: /singleton [0m[31m15:35:52,112 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-4) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 70 more [0m[31m15:35:52,143 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-4) Error while processing prepare: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 70 more [0m[31m15:35:52,158 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-4) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=6, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@7} org.infinispan.transaction.synchronization.SynchronizationAdapter@26: org.infinispan.CacheException: Could not prepare. at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: javax.transaction.xa.XAException at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) ... 19 more [0m[33m15:35:52,165 WARN [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC00004: Failure during stop of service jboss.test.myservice: org.infinispan.CacheException: Unable to end batch at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:107) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) ... 13 more Update from EAP 6.1.1.ER7 testing: the test still fails, the interesting part here is the: [0m[33m05:30:50,489 WARN [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC000004: Failure during stop of service jboss.test.myservice: org.infinispan.CacheException: Unable to end batch at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:107) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) Link to test output: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/jdk=openjdk-1.7.0-local,label=RHEL5_x86_64/60/testReport/junit/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase(SYNC-udp)/testSingletonService/ For the "unable to end batch" error, there is already a BZ 901161, that was seen during our integration testing. We will plan to port the upstream fix for this test for 6.2. The addition of the GlobalComponentRegistryService should improve the reliability of this test. Please retest against EAP 6.2.0.ER2. Not seen in EAP 6.2.0.ER3, nor ER1. And it's back. Though it is more probable that it is just rare and wasn't seen previously. EAP 6.2.0.ER6 Log is long and will be attached (similar to what baranowb posted, though there is a SuspectException instead of TimeoutException). Link: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/lastCompletedBuild/jdk=java16_default,label=RHEL5_x86/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28ASYNC-tcp%29/testSingletonService/ Created attachment 815443 [details]
standard-output.txt
Fixing the link from my last comment - correct build and no views: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/67/jdk=java16_default,label=RHEL5_x86/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28ASYNC-tcp%29/testSingletonService/ Seen also with EAP 6.3.1.CP.CR1 jdk=ibm1.7 RHEL5 && x86_64 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-63x-patched-testsuite-rhel/jdk=ibm1.7,label_exp=eap-sustaining%20&&%20RHEL5%20&&%20x86_64/5/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28SYNC-tcp%29/testSingletonService/ |