EAP 6.1.0.ER1. Seen in pure ipv6 environment, though this may be just coincidence. At server shutdown, were are seeing this exception: 01:48:51,766 WARN [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000032: Problems un-registering MBeans: org.infinispan.CacheException: Failure while unregistering mbeans at org.infinispan.jmx.ComponentsJmxRegistration.unregisterMBeans(ComponentsJmxRegistration.java:105) at org.infinispan.jmx.AbstractJmxRegistration.unregisterMBeans(AbstractJmxRegistration.java:70) at org.infinispan.jmx.CacheJmxRegistration.stop(CacheJmxRegistration.java:107) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_38] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_38] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:690) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:568) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:224) at org.infinispan.CacheImpl.stop(CacheImpl.java:581) at org.infinispan.CacheImpl.stop(CacheImpl.java:576) at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38] Caused by: javax.management.InstanceNotFoundException: jboss.infinispan:type=Cache,name="default-host/clusterbench(repl_async)",manager="web",component=Activation at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) [rt.jar:1.6.0_38] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415) [rt.jar:1.6.0_38] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403) [rt.jar:1.6.0_38] at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) [rt.jar:1.6.0_38] at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.unregisterMBean(PluggableMBeanServerImpl.java:613) at org.jboss.as.jmx.PluggableMBeanServerImpl.unregisterMBean(PluggableMBeanServerImpl.java:360) at org.infinispan.jmx.JmxUtil.unregisterMBean(JmxUtil.java:112) at org.infinispan.jmx.ComponentsJmxRegistration.unregisterMBeans(ComponentsJmxRegistration.java:101) ... 21 more Cache: repl async. whole server log is here: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-repl-async-modcluster-ipv6/4/artifact/report/config/jboss-vmg36.mw.lab.eng.bos.redhat.com/server.log Can this be AS7-4247?
Here we have it again in EAP 6.1.0.ER2, this time in ipv4 environment: link to server logs: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbstateless-shutdown-repl-async/13/artifact/report/config/perf21/server.log
It also happens in EAP 6.1.0.ER3 and indeed, it is NOT specific to IPv6: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-shutdown-repl-async/14/console-perf21/
Still seeing this with EAP 6.1.0.ER8. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/18/artifact/report/config/jboss-perf21/server.log
Still seeing this with EAP 6.1.1.ER7. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/25/
Radoslav Husar <rhusar> updated the status of jira AS7-4247 to Closed
This should no longer be an issue.
Still seeing this issue (EAP 6.2.0.CR3), for example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/34/artifact/report/config/jboss-perf19/server.log
The cause seems to be that we are getting multiple cache stop request on a single cache instance during server shutdown: 08:56:18,692 DEBUG [org.infinispan.CacheImpl] (ServerService Thread Pool -- 5) Stopping cache repl on localhost/web: java.lang.RuntimeException: StackTrace at org.infinispan.CacheImpl.stop(CacheImpl.java:603) at org.infinispan.CacheImpl.stop(CacheImpl.java:599) at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:97) at org.jboss.as.clustering.msc.AsynchronousService$2.run(AsynchronousService.java:113) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1] 08:56:18,790 DEBUG [org.infinispan.CacheImpl] (MSC service thread 1-7) Stopping cache repl on localhost/web: java.lang.RuntimeException: StackTrace at org.infinispan.CacheImpl.stop(CacheImpl.java:603) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1] at org.infinispan.CacheImpl.stop(CacheImpl.java:599) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1] at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:732) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1] at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1] at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76) [jboss-as-clustering-infinispan-7.4.0.Final-redhat-SNAPSHOT.jar:7.4.0.Final-redhat-SNAPSHOT] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2088) [jboss-msc-1.1.5.Final.jar:1.1.5.Final] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2049) [jboss-msc-1.1.5.Final.jar:1.1.5.Final] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29] The first comes from CacheService instance running the cache stop method and the second comes from EmbeddedCacheManagerService containing the cache stop method. It seems that Infinispan cachemanager and MSC service management is interleaving. The CacheImpl.stop is not synchronized so there is potential conflict.
Dominik Pospisil <dpospisi> updated the status of jira ISPN-4127 to Coding In Progress
Mircea Markus <mmarkus> updated the status of jira ISPN-4127 to Resolved
We will need to revisit https://bugzilla.redhat.com/show_bug.cgi?id=1036889 for Final to pick up this and other possible fixes.
This was seen in 6.3.0.DR4 and was NOT seen in 6.3.0.DR6. I will keep this ON_QA for at least one more build and if we don't see it again, we will mark VERIFIED soon.
No problem, sounds like a good approach for issues that can be rarely hit. You can also verify the code changes if you want.
This wasn't seen in the last 3 builds (DR6, ER1 and ER2), marking as verified.
Changed <literal></literal> tags in Doc Text to ticks (`) and removed <programlisting> tag to fix Bug 1096865