Bug 917010 - CacheException: Failure while unregistering mbeans at server shutdown
Summary: CacheException: Failure while unregistering mbeans at server shutdown
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering
Version: 6.1.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: DR6
: EAP 6.3.0
Assignee: Dominik Pospisil
QA Contact: Jitka Kozana
Russell Dickenson
URL:
Whiteboard:
Depends On: 1078158
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-01 13:04 UTC by Jitka Kozana
Modified: 2014-06-28 15:29 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previous versions of JBoss EAP 6 contained a bug in the Infinispan component that could cause the following exception at server shut down: ---- WARN [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000032: Problems un-registering MBeans: org.infinispan.CacheException: Failure while unregistering mbeans ---- The error did not prevent successful shut down of the server and was caused by multiple stop requests (from `CacheService` and `EmbeddedCacheManagerService`) being sent to single cache instances. This issue was resolved with an upgrade to the Infinispan component.
Clone Of:
Environment:
Last Closed: 2014-06-28 15:29:26 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker AS7-4247 Minor Closed Server shutdown: Problems un-registering MBeans: CacheException: Failure while unregistering mbeans 2018-07-30 07:40:28 UTC
JBoss Issue Tracker ISPN-4127 Major Resolved CacheJmxRegistration.unregisterCacheMBean unregisters cache mbean of all caches 2018-07-30 07:40:27 UTC

Description Jitka Kozana 2013-03-01 13:04:01 UTC
EAP 6.1.0.ER1.
Seen in pure ipv6 environment, though this may be just coincidence. 

At server shutdown, were are seeing this exception:
01:48:51,766 WARN  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000032: Problems un-registering MBeans: org.infinispan.CacheException: Failure while unregistering mbeans
	at org.infinispan.jmx.ComponentsJmxRegistration.unregisterMBeans(ComponentsJmxRegistration.java:105)
	at org.infinispan.jmx.AbstractJmxRegistration.unregisterMBeans(AbstractJmxRegistration.java:70)
	at org.infinispan.jmx.CacheJmxRegistration.stop(CacheJmxRegistration.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_38]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_38]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:690)
	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:568)
	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:224)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:581)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:576)
	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
	at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96)
	at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
Caused by: javax.management.InstanceNotFoundException: jboss.infinispan:type=Cache,name="default-host/clusterbench(repl_async)",manager="web",component=Activation
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) [rt.jar:1.6.0_38]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415) [rt.jar:1.6.0_38]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403) [rt.jar:1.6.0_38]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506) [rt.jar:1.6.0_38]
	at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.unregisterMBean(PluggableMBeanServerImpl.java:613)
	at org.jboss.as.jmx.PluggableMBeanServerImpl.unregisterMBean(PluggableMBeanServerImpl.java:360)
	at org.infinispan.jmx.JmxUtil.unregisterMBean(JmxUtil.java:112)
	at org.infinispan.jmx.ComponentsJmxRegistration.unregisterMBeans(ComponentsJmxRegistration.java:101)
	... 21 more


Cache: repl async. 
whole server log is here: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-repl-async-modcluster-ipv6/4/artifact/report/config/jboss-vmg36.mw.lab.eng.bos.redhat.com/server.log

Can this be AS7-4247?

Comment 2 Ladislav Thon 2013-03-25 09:49:13 UTC
It also happens in EAP 6.1.0.ER3 and indeed, it is NOT specific to IPv6: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-shutdown-repl-async/14/console-perf21/

Comment 4 Ladislav Thon 2013-08-26 11:12:58 UTC
Still seeing this with EAP 6.1.1.ER7. For example:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/25/

Comment 5 JBoss JIRA Server 2013-12-09 13:38:50 UTC
Radoslav Husar <rhusar@redhat.com> updated the status of jira AS7-4247 to Closed

Comment 6 Paul Ferraro 2013-12-09 13:44:46 UTC
This should no longer be an issue.

Comment 8 Dominik Pospisil 2014-03-14 08:34:20 UTC
The cause seems to be that we are getting multiple cache stop request on a single cache instance during server shutdown:

08:56:18,692 DEBUG [org.infinispan.CacheImpl] (ServerService Thread Pool -- 5) Stopping cache repl on localhost/web: java.lang.RuntimeException: StackTrace
        at org.infinispan.CacheImpl.stop(CacheImpl.java:603)
        at org.infinispan.CacheImpl.stop(CacheImpl.java:599)
        at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
        at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:97)
        at org.jboss.as.clustering.msc.AsynchronousService$2.run(AsynchronousService.java:113)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

08:56:18,790 DEBUG [org.infinispan.CacheImpl] (MSC service thread 1-7) Stopping cache repl on localhost/web: java.lang.RuntimeException: StackTrace
        at org.infinispan.CacheImpl.stop(CacheImpl.java:603) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.infinispan.CacheImpl.stop(CacheImpl.java:599) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:732) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179) [infinispan-core-5.2.8.CR1.jar:5.2.8.CR1]
        at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76) [jboss-as-clustering-infinispan-7.4.0.Final-redhat-SNAPSHOT.jar:7.4.0.Final-redhat-SNAPSHOT]
        at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2088) [jboss-msc-1.1.5.Final.jar:1.1.5.Final]
        at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2049) [jboss-msc-1.1.5.Final.jar:1.1.5.Final]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]

The first comes from CacheService instance running the cache stop method and the second comes from EmbeddedCacheManagerService containing the cache stop method. It seems that Infinispan cachemanager and MSC service management is interleaving. The CacheImpl.stop is not synchronized so there is potential conflict.

Comment 9 JBoss JIRA Server 2014-03-18 08:58:21 UTC
Dominik Pospisil <dpospisi@redhat.com> updated the status of jira ISPN-4127 to Coding In Progress

Comment 10 JBoss JIRA Server 2014-03-18 15:21:26 UTC
Mircea Markus <mmarkus@redhat.com> updated the status of jira ISPN-4127 to Resolved

Comment 11 Radoslav Husar 2014-03-18 15:39:39 UTC
We will need to revisit https://bugzilla.redhat.com/show_bug.cgi?id=1036889 for Final to pick up this and other possible fixes.

Comment 12 Ladislav Thon 2014-04-03 12:58:08 UTC
This was seen in 6.3.0.DR4 and was NOT seen in 6.3.0.DR6. I will keep this ON_QA for at least one more build and if we don't see it again, we will mark VERIFIED soon.

Comment 13 Radoslav Husar 2014-04-03 15:39:21 UTC
No problem, sounds like a good approach for issues that can be rarely hit. You can also verify the code changes if you want.

Comment 14 Ladislav Thon 2014-04-29 11:26:50 UTC
This wasn't seen in the last 3 builds (DR6, ER1 and ER2), marking as verified.

Comment 15 sgilda 2014-05-12 19:10:11 UTC
Changed <literal></literal> tags in Doc Text to ticks (`) and removed <programlisting> tag to fix Bug 1096865


Note You need to log in before you can comment on or make changes to this bug.