Created attachment 907573 [details] Server TRACE logs This started failing with 6.3.0.ER6, ER5 was OK. When the 3rd node joins the cluster, it hangs for ~ 4 minutes, after which this exception occurs on the 3rd node: 12:00:43,839 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_06] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_06] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_06] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:170) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) at org.infinispan.CacheImpl.start(CacheImpl.java:696) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:574) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:529) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:409) at org.infinispan.registry.ClusterRegistryImpl.startRegistryCache(ClusterRegistryImpl.java:136) at org.infinispan.registry.ClusterRegistryImpl.addListener(ClusterRegistryImpl.java:97) at org.infinispan.query.remote.ProtobufMetadataManager.ensureInit(ProtobufMetadataManager.java:65) at org.infinispan.query.remote.ProtobufMetadataManager.getSerializationContext(ProtobufMetadataManager.java:132) at org.infinispan.query.remote.LifecycleManager.cacheStarting(LifecycleManager.java:114) at org.infinispan.factories.ComponentRegistry.notifyCacheStarting(ComponentRegistry.java:228) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:214) at org.infinispan.CacheImpl.start(CacheImpl.java:696) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:574) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:529) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:409) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80) at org.jboss.as.clustering.infinispan.subsystem.SecurityActions$5.run(SecurityActions.java:101) at org.jboss.as.clustering.infinispan.subsystem.SecurityActions$5.run(SecurityActions.java:98) at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_06] at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.startCache(SecurityActions.java:109) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] ... 3 more Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache __cluster_registry_cache__ on node2/clustered at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_06] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_06] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_06] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_06] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) ... 33 more The tests fail with arquillian error though, because arquillian gives the server only 60s to start. TRACE logs attached.
The trace logs are from the run of this class https://github.com/infinispan/infinispan/blob/master/server/integration/testsuite/src/test/java/org/infinispan/server/test/jmx/suppress/statetransfer/StateTransferSuppressForHotrodIT.java and this test case only: https://github.com/infinispan/infinispan/blob/master/server/integration/testsuite/src/test/java/org/infinispan/server/test/jmx/suppress/statetransfer/AbstractStateTransferSuppressIT.java#L133 (fails on line 196, when starting the 3rd node)
This has been verified a long time ago.