Description of problem: ExampleConfigsIT.testJDBCCacheStoreConfig fails randomly with Error Message Could not retrieve HotRod host Stacktrace java.lang.RuntimeException: Could not retrieve HotRod host at org.infinispan.arquillian.utils.MBeanUtils.getMBeanAttribute(MBeanUtils.java:59) at org.infinispan.arquillian.core.HotRodEndpoint.getInetAddress(HotRodEndpoint.java:65) at org.infinispan.server.test.util.ITestUtils.createCacheManager(ITestUtils.java:47) at org.infinispan.server.test.util.RemoteCacheManagerFactory.createManager(RemoteCacheManagerFactory.java:50) at org.infinispan.server.test.util.RemoteCacheManagerFactory.createCache(RemoteCacheManagerFactory.java:34) at org.infinispan.server.test.util.RemoteCacheManagerFactory.createCache(RemoteCacheManagerFactory.java:30) at org.infinispan.server.test.configs.ExampleConfigsIT.createCache(ExampleConfigsIT.java:728) at org.infinispan.server.test.configs.ExampleConfigsIT.testJDBCCacheStoreConfig(ExampleConfigsIT.java:312) Caused by: javax.management.InstanceNotFoundException: jboss.infinispan:type=Server,name=HotRod,component=Transport at org.jboss.as.jmx.PluggableMBeanServerImpl.findDelegate(PluggableMBeanServerImpl.java:1093) at org.jboss.as.jmx.PluggableMBeanServerImpl.getAttribute(PluggableMBeanServerImpl.java:388) at org.jboss.as.jmx.BlockingNotificationMBeanServer.getAttribute(BlockingNotificationMBeanServer.java:148) at org.jboss.remotingjmx.protocol.v2.ServerProxy$GetAttributeHandler.handle(ServerProxy.java:691) at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1$1.run(ServerCommon.java:153) at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:75) at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:70) at java.security.AccessController.doPrivileged(AccessController.java:362) at javax.security.auth.Subject.doAs(Subject.java:573) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor.handleEvent(ServerInterceptorFactory.java:70) at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:149) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:761)
I believe that the underlying problem here is the same as in https://bugzilla.redhat.com/show_bug.cgi?id=1181715
This can be replicated just by starting the server with the clustered-ccl.xml config, although with low probability. Turning trace logging on whole org.infinispan seemed to suppress it, I was able to only get logging for org.infinispan.persistence. I couldn't hit it with debugging either... Happens with 6.5.0.ER2. The server error looks like this: 13:59:30,457 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.infinispan.clustered.memcachedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1936) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) at org.infinispan.CacheImpl.start(CacheImpl.java:779) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:587) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:542) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:420) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:434) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80) at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104) at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101) at org.infinispan.security.Security.doPrivileged(Security.java:76) at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48) at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:85) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] ... 3 more Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31] at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) ... 23 more Caused by: java.lang.NullPointerException at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.processStoreConfiguration(CacheStoreFactoryRegistry.java:48) at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:529) at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:114) ... 28 more
Created attachment 1017405 [details] server trace log
Executed ExampleConfigsIT.testJDBCCacheStoreConfig multiple times without any error with JDG 6.5.0 ER3
Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-5403 to Reopened