Description of problem: EAP 6.1.0.ER4 Test org.jboss.as.test.clustering.cluster.singleton.SingletonTestCase(SYNC-tcp).testSingletonService randomly fails; tcp is configured to use TCPGOSSIP, as in https://bugzilla.redhat.com/show_bug.cgi?id=922662 Error Message Deployment with name deployment-1 could not be undeployed. Container container-1 must be still running. Stacktrace java.lang.IllegalArgumentException: Deployment with name deployment-1 could not be undeployed. Container container-1 must be still running. at org.jboss.arquillian.container.test.impl.client.deployment.ClientDeployer.undeploy(ClientDeployer.java:128) at org.jboss.as.test.clustering.cluster.singleton.SingletonTestCase.testSingletonService(SingletonTestCase.java:236) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.jboss.arquillian.junit.Arquillian$6$1.invoke(Arquillian.java:270) at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:53) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111) at org.jboss.arquillian.junit.Arquillian$6.evaluate(Arquillian.java:263) at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:226) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:240) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:185) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:199) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:147) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/43/jdk=ibm16,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28SYNC-tcp%29/testSingletonService/
The ISE from the testing harness is not very interesting in this context, the actual issue is a timeout on sending a message: 03:44:35,963 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-2) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to node-1/singleton As these tests rely heavily on the network, this seems be just environmental.
Also seen in 6.1.1.ER3.
Test fails, either due to container not comming up or some sort deadlock/timeout( it seems). However there is one familiar thing: Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) ... 13 more This may or not be related to: https://issues.jboss.org/browse/WFLY-1507 Test output: Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 66.625 sec <<< FAILURE! testSingletonService(org.jboss.as.test.clustering.cluster.singleton.SingletonTestCase) Time elapsed: 46.205 sec <<< ERROR! java.lang.IllegalArgumentException: Deployment with name deployment-1 could not be undeployed. Container container-1 must be still running. at org.jboss.arquillian.container.test.impl.client.deployment.ClientDeployer.undeploy(ClientDeployer.java:128) at org.jboss.as.test.clustering.cluster.singleton.SingletonTestCase.testSingletonService(SingletonTestCase.java:204) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.jboss.arquillian.junit.Arquillian$6$1.invoke(Arquillian.java:270) at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:53) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:89) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:75) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:60) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111) at org.jboss.arquillian.junit.Arquillian$6.evaluate(Arquillian.java:263) at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:226) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:240) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:185) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:199) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:147) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) Log: java:global/singleton/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:app/singleton/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:module/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:jboss/exported/singleton/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:global/singleton/ViewChangeListenerBean java:app/singleton/ViewChangeListenerBean java:module/ViewChangeListenerBean [0m[0m15:35:17,014 INFO [org.jboss.web] (ServerService Thread Pool -- 55) JBAS018210: Register web context: /singleton [0m[0m15:35:17,528 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000078: Starting JGroups Channel [0m[0m15:35:17,548 INFO [stdout] (ServerService Thread Pool -- 52) [0m[0m15:35:17,549 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m15:35:17,549 INFO [stdout] (ServerService Thread Pool -- 52) GMS: address=node-1/singleton, cluster=singleton, physical address=127.0.0.1:7700 [0m[0m15:35:17,549 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m15:35:17,677 INFO [org.jboss.as.clustering] (Incoming-8,shared=tcp) JBAS010225: New cluster view for partition singleton (id: 3, delta: 1, merge: false) : [node-0/singleton, node-1/singleton] [0m[0m15:35:17,678 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-8,shared=tcp) ISPN000094: Received new cluster view: [node-0/singleton|3] [node-0/singleton, node-1/singleton] [0m[0m15:35:17,693 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000094: Received new cluster view: [node-0/singleton|3] [node-0/singleton, node-1/singleton] [0m[0m15:35:17,750 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000079: Cache local address is node-1/singleton, physical addresses are [127.0.0.1:7700] [0m[0m15:35:17,757 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 52) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final [0m[0m15:35:17,807 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 52) ISPN000161: Using a batchMode transaction manager [0m[0m15:35:17,948 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 52) ISPN000031: MBeans were successfully registered to the platform MBean server. [0m[0m15:35:18,097 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 52) JBAS010281: Started default cache from singleton container [0m[0m15:35:18,110 INFO [org.jboss.as.clustering] (MSC service thread 1-8) JBAS010238: Number of cluster members: 2 [0m[0m15:35:18,133 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: node-1/singleton elected as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,134 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010341: This node will no longer operate as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,138 INFO [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 52) JBAS010342: node-1/singleton elected as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,139 INFO [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 52) JBAS010340: This node will now operate as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,197 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "singleton.war" (runtime-name : "singleton.war") [0m[0m15:35:18,292 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10090/management [0m[0m15:35:18,293 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10090 [0m[0m15:35:18,294 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.1.1.Alpha1 (AS 7.2.2-internal-SNAPSHOT) started in 5894ms - Started 185 of 302 services (115 services are passive or on-demand) [0m[0m15:35:18,486 INFO [stdout] (http-/127.0.0.1:8080-1) Cluster 'singleton' successfully established view [node-0/singleton, node-1/singleton] within 0 ms. [0m15:35:18,694 INFO [org.jboss.arquillian.container.test.impl.client.container.ClientContainerController] (main) Manual stopping of a server instance [0m15:35:18,721 INFO [org.apache.coyote.http11] (MSC service thread 1-7) JBWEB003075: Coyote HTTP/1.1 pausing on: http-/127.0.0.1:8180 [0m[0m15:35:18,722 INFO [org.apache.coyote.http11] (MSC service thread 1-7) JBWEB003077: Coyote HTTP/1.1 stopping on : http-/127.0.0.1:8180 [0m[0m15:35:18,733 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 11) MODCLUSTER000002: Initiating mod_cluster shutdown [0m[0m15:35:18,739 INFO [org.apache.catalina.core] (MSC service thread 1-4) JBWEB001079: Container org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/] has not been started [0m[0m15:35:18,750 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: node-0/singleton elected as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,751 INFO [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010340: This node will now operate as the singleton provider of the jboss.test.myservice service [0m[0m15:35:18,758 INFO [org.jboss.web] (ServerService Thread Pool -- 11) JBAS018224: Unregister web context: /singleton [0m[0m15:35:18,769 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010409: Unbound data source [java:jboss/datasources/ExampleDS] [0m[0m15:35:18,772 INFO [org.apache.coyote.ajp] (MSC service thread 1-5) JBWEB003048: Pausing Coyote AJP/1.3 on ajp-/127.0.0.1:8109 [0m[0m15:35:18,774 INFO [org.apache.coyote.ajp] (MSC service thread 1-5) JBWEB003051: Stopping Coyote AJP/1.3 on ajp-/127.0.0.1:8109 [0m[0m15:35:18,813 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped default cache from singleton container [0m[0m15:35:18,817 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting and closing JGroups Channel [0m[0m15:35:18,854 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment singleton.war (runtime-name: singleton.war) in 144ms [0m[0m15:35:19,129 INFO [org.jboss.as.clustering] (Incoming-15,shared=tcp) JBAS010225: New cluster view for partition singleton (id: 4, delta: -1, merge: false) : [node-0/singleton] [0m[0m15:35:19,130 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-15,shared=tcp) ISPN000094: Received new cluster view: [node-0/singleton|4] [node-0/singleton] [0m[0m15:35:19,155 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher [0m[0m15:35:19,163 INFO [org.jboss.as] (MSC service thread 1-4) JBAS015950: JBoss EAP 6.1.1.Alpha1 (AS 7.2.2-internal-SNAPSHOT) stopped in 413ms [0m[0m15:35:19,183 INFO [org.jboss.as.clustering] (Incoming-17,shared=tcp) JBAS010225: New cluster view for partition singleton (id: 5, delta: 1, merge: false) : [node-0/singleton, node-1/singleton] [0m[0m15:35:19,183 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,shared=tcp) ISPN000094: Received new cluster view: [node-0/singleton|5] [node-0/singleton, node-1/singleton] [0m15:35:19,522 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopped deployment=NONE, container=container-1 [0m15:35:19,525 INFO [stdout] (http-/127.0.0.1:8080-1) [node-0/singleton] != [node-0/singleton, node-1/singleton], waiting for a view change event... [0m[33m15:35:24,185 WARN [org.jgroups.protocols.pbcast.GMS] (ViewHandler,singleton,node-0/singleton) node-0/singleton: failed to collect all ACKs (expected=1) for unicast view [node-0/singleton|5] [node-0/singleton, node-1/singleton] after 5000ms, missing ACKs from [node-1/singleton] [0m[31m15:35:34,552 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/singleton].[org.jboss.as.test.clustering.ViewChangeListenerServlet]] (http-/127.0.0.1:8080-1) JBWEB000236: Servlet.service() for servlet org.jboss.as.test.clustering.ViewChangeListenerServlet threw exception: java.lang.InterruptedException: Cluster 'singleton' failed to establish view [node-0/singleton] within 15000 ms. Current view is: [node-0/singleton, node-1/singleton] at org.jboss.as.test.clustering.ViewChangeListenerBean.establishView(ViewChangeListenerBean.java:77) [classes:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:248) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final.jar:1.1.2.Final] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:249) [jboss-as-ejb3-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocation(EJBObjectInterceptor.java:58) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocation(EJBHomeInterceptor.java:83) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:42) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:125) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:253) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144) [jboss-ejb-client-1.0.23.Final.jar:1.0.23.Final] at com.sun.proxy.$Proxy30.establishView(Unknown Source) at org.jboss.as.test.clustering.ViewChangeListenerServlet.doGet(ViewChangeListenerServlet.java:70) [classes:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final.jar:1.0.2.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final.jar:1.0.2.Final] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91) at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final.jar:7.2.2.Final] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] [0m[0m15:35:34,606 INFO [org.jboss.web] (ServerService Thread Pool -- 53) JBAS018224: Unregister web context: /singleton [0m[31m15:35:52,112 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-4) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 70 more [0m[31m15:35:52,143 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-4) Error while processing prepare: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: org.jgroups.TimeoutException: timeout sending message to node-1/singleton at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 70 more [0m[31m15:35:52,158 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-4) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=6, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@7} org.infinispan.transaction.synchronization.SynchronizationAdapter@26: org.infinispan.CacheException: Could not prepare. at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70) at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230) at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.2-internal-SNAPSHOT.jar:7.2.2-internal-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: javax.transaction.xa.XAException at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123) at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) ... 19 more [0m[33m15:35:52,165 WARN [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC00004: Failure during stop of service jboss.test.myservice: org.infinispan.CacheException: Unable to end batch at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:107) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA.jar:1.0.4.GA] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) ... 13 more
Update from EAP 6.1.1.ER7 testing: the test still fails, the interesting part here is the: [0m[33m05:30:50,489 WARN [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC000004: Failure during stop of service jboss.test.myservice: org.infinispan.CacheException: Unable to end batch at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:107) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53) at org.jboss.as.clustering.service.ServiceProviderRegistryService.unregister(ServiceProviderRegistryService.java:110) at org.jboss.as.clustering.singleton.SingletonService.stop(SingletonService.java:145) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9 at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:89) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102) at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) Link to test output: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/jdk=openjdk-1.7.0-local,label=RHEL5_x86_64/60/testReport/junit/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase(SYNC-udp)/testSingletonService/ For the "unable to end batch" error, there is already a BZ 901161, that was seen during our integration testing.
We will plan to port the upstream fix for this test for 6.2.
The addition of the GlobalComponentRegistryService should improve the reliability of this test. Please retest against EAP 6.2.0.ER2.
Not seen in EAP 6.2.0.ER3, nor ER1.
And it's back. Though it is more probable that it is just rare and wasn't seen previously. EAP 6.2.0.ER6 Log is long and will be attached (similar to what baranowb posted, though there is a SuspectException instead of TimeoutException). Link: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/lastCompletedBuild/jdk=java16_default,label=RHEL5_x86/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28ASYNC-tcp%29/testSingletonService/
Created attachment 815443 [details] standard-output.txt
Fixing the link from my last comment - correct build and no views: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/67/jdk=java16_default,label=RHEL5_x86/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28ASYNC-tcp%29/testSingletonService/
Seen also with EAP 6.3.1.CP.CR1 jdk=ibm1.7 RHEL5 && x86_64 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-63x-patched-testsuite-rhel/jdk=ibm1.7,label_exp=eap-sustaining%20&&%20RHEL5%20&&%20x86_64/5/testReport/org.jboss.as.test.clustering.cluster.singleton/SingletonTestCase%28SYNC-tcp%29/testSingletonService/