Bug 917010

Summary: CacheException: Failure while unregistering mbeans at server shutdown
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Jitka Kozana <jkudrnac>
Component: ClusteringAssignee: Dominik Pospisil <dpospisi>
Status: CLOSED CURRENTRELEASE QA Contact: Jitka Kozana <jkudrnac>
Severity: medium Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.1.1CC: cdewolf, dpospisi, jkudrnac, lthon, myarboro, rhusar, rjanik, sgilda, smumford
Target Milestone: DR6   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previous versions of JBoss EAP 6 contained a bug in the Infinispan component that could cause the following exception at server shut down: ---- WARN [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000032: Problems un-registering MBeans: org.infinispan.CacheException: Failure while unregistering mbeans ---- The error did not prevent successful shut down of the server and was caused by multiple stop requests (from `CacheService` and `EmbeddedCacheManagerService`) being sent to single cache instances. This issue was resolved with an upgrade to the Infinispan component.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-28 15:29:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1078158    
Bug Blocks:    

Description Jitka Kozana 2013-03-01 13:04:01 UTC
EAP 6.1.0.ER1.
Seen in pure ipv6 environment, though this may be just coincidence. 

At server shutdown, were are seeing this exception:
01:48:51,766 WARN  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000032: Problems un-registering MBeans: org.infinispan.CacheException: Failure while unregistering mbeans
	at org.infinispan.jmx.ComponentsJmxRegistration.unregisterMBeans(ComponentsJmxRegistration.java:105)
	at org.infinispan.jmx.AbstractJmxRegistration.unregisterMBeans(AbstractJmxRegistration.java:70)
	at org.infinispan.jmx.CacheJmxRegistration.stop(CacheJmxRegistration.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_38]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_38]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:690)
	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:568)
	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:224)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:581)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:576)
	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
	at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96)
	at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
Caused by: javax.management.InstanceNotFoundException: jboss.infinispan:type=Cache,name="default-host/clusterbench(repl_async)",manager="web",component=Activation
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) [rt.jar:1.6.0_38]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415) [rt.jar:1.6.0_38]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403) [rt.jar:1.6.0_38]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) [rt.jar:1.6.0_38]
	at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.unregisterMBean(PluggableMBeanServerImpl.java:613)
	at org.jboss.as.jmx.PluggableMBeanServerImpl.unregisterMBean(PluggableMBeanServerImpl.java:360)
	at org.infinispan.jmx.JmxUtil.unregisterMBean(JmxUtil.java:112)
	at org.infinispan.jmx.ComponentsJmxRegistration.unregisterMBeans(ComponentsJmxRegistration.java:101)
	... 21 more


Cache: repl async. 
whole server log is here: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-repl-async-modcluster-ipv6/4/artifact/report/config/jboss-vmg36.mw.lab.eng.bos.redhat.com/server.log

Can this be AS7-4247?

Comment 2 Ladislav Thon 2013-03-25 09:49:13 UTC
It also happens in EAP 6.1.0.ER3 and indeed, it is NOT specific to IPv6: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-shutdown-repl-async/14/console-perf21/

Comment 4 Ladislav Thon 2013-08-26 11:12:58 UTC
Still seeing this with EAP 6.1.1.ER7. For example:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/25/

Comment 5 JBoss JIRA Server 2013-12-09 13:38:50 UTC
Radoslav Husar <rhusar> updated the status of jira AS7-4247 to Closed

Comment 6 Paul Ferraro 2013-12-09 13:44:46 UTC
This should no longer be an issue.

Comment 8 Dominik Pospisil 2014-03-14 08:34:20 UTC
The cause seems to be that we are getting multiple cache stop request on a single cache instance during server shutdown:

08:56:18,692 DEBUG [org.infinispan.CacheImpl] (ServerService Thread Pool -- 5) Stopping cache repl on localhost/web: java.lang.RuntimeException: StackTrace
        at org.infinispan.CacheImpl.stop(CacheImpl.java:603)
        at org.infinispan.CacheImpl.stop(CacheImpl.java:599)
        at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
        at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:97)
        at org.jboss.as.clustering.msc.AsynchronousService$2.run(AsynchronousService.java:113)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

08:56:18,790 DEBUG [org.infinispan.CacheImpl] (MSC service thread 1-7) Stopping cache repl on localhost/web: java.lang.RuntimeException: StackTrace
        at org.infinispan.CacheImpl.stop(CacheImpl.java:603) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.infinispan.CacheImpl.stop(CacheImpl.java:599) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:732) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76) [jboss-as-clustering-infinispan-7.4.0.Final-redhat-SNAPSHOT.jar:7.4.0.Final-redhat-SNAPSHOT]
        at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2088) [jboss-msc-1.1.5.Final.jar:1.1.5.Final]
        at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2049) [jboss-msc-1.1.5.Final.jar:1.1.5.Final]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]

The first comes from CacheService instance running the cache stop method and the second comes from EmbeddedCacheManagerService containing the cache stop method. It seems that Infinispan cachemanager and MSC service management is interleaving. The CacheImpl.stop is not synchronized so there is potential conflict.

Comment 9 JBoss JIRA Server 2014-03-18 08:58:21 UTC
Dominik Pospisil <dpospisi> updated the status of jira ISPN-4127 to Coding In Progress

Comment 10 JBoss JIRA Server 2014-03-18 15:21:26 UTC
Mircea Markus <mmarkus> updated the status of jira ISPN-4127 to Resolved

Comment 11 Radoslav Husar 2014-03-18 15:39:39 UTC
We will need to revisit https://bugzilla.redhat.com/show_bug.cgi?id=1036889 for Final to pick up this and other possible fixes.

Comment 12 Ladislav Thon 2014-04-03 12:58:08 UTC
This was seen in 6.3.0.DR4 and was NOT seen in 6.3.0.DR6. I will keep this ON_QA for at least one more build and if we don't see it again, we will mark VERIFIED soon.

Comment 13 Radoslav Husar 2014-04-03 15:39:21 UTC
No problem, sounds like a good approach for issues that can be rarely hit. You can also verify the code changes if you want.

Comment 14 Ladislav Thon 2014-04-29 11:26:50 UTC
This wasn't seen in the last 3 builds (DR6, ER1 and ER2), marking as verified.

Comment 15 sgilda 2014-05-12 19:10:11 UTC
Changed <literal></literal> tags in Doc Text to ticks (`) and removed <programlisting> tag to fix Bug 1096865