Bug 900946 (JBPAPP6-1274) - IllegalStateException: Cache is in 'TERMINATED'/'STOPPING' state.
Summary: IllegalStateException: Cache is in 'TERMINATED'/'STOPPING' state.
Keywords:
Status: ASSIGNED
Alias: JBPAPP6-1274
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering
Version: 6.1.1,6.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: TBD EAP 7
Assignee: Paul Ferraro
QA Contact: Michal Vinkler
URL: http://jira.jboss.org/jira/browse/JBP...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-09-27 10:43 UTC by Richard Janík
Modified: 2018-03-06 20:35 UTC (History)
8 users (show)

Doc Type: Known Issue
Doc Text:
This release of JBoss EAP 6 carries a bug that may cause an `IllegalStateException` to appear after an application is undeployed on one node within 5 seconds of having been deployed on another node within the cluster. This issue in under investigation and is expected to be resolved in a future release.
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-3743 Major Resolved Silence "IllegalStateException: Default cache is in 'STOPPING' state" exceptions 2018-06-20 17:52:34 UTC
JBoss Issue Tracker JBPAPP6-1274 Major Closed IllegalStateException: Cache is in 'TERMINATED'/'STOPPING' state. 2018-06-20 17:52:34 UTC

Description Richard Janík 2012-09-27 10:43:47 UTC
Steps to Reproduce: # node1 is running with clusterbench (https://github.com/rhusar/clusterbench) deployed
# node2 deploys clusterbench
# undeploy clusterbench from node1 soon after it is deployed on node2
# some of subsequent client requests will fail with 500 and this exception (node1)

I was able to reproduce this issue manually, roughly 1 in 7 tries will succeed at reproducing this.
project_key: JBPAPP6

There was an exception during manual ipv6 clustering failover testing, scenario ejbservlet-repl-async-undeploy. Full stack trace is below:
{code}
06:11:11,653 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host]] (ajp-/2620:52:0:105f:0:0:ffff:4%2:8009-4) Exception sending request initialized lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener: java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
	at org.infinispan.interceptors.IsMarshallableInterceptor.visitGetKeyValueCommand(IsMarshallableInterceptor.java:94)
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:80)
	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	at org.infinispan.CacheImpl.get(CacheImpl.java:278)
	at org.infinispan.CacheImpl.get(CacheImpl.java:270)
	at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:303)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:199)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:196)
	at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:19)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:60)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.invoke(InfinispanBackingCacheEntryStore.java:243)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.get(InfinispanBackingCacheEntryStore.java:202)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:290) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.peek(PassivatingBackingCacheImpl.java:205) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.contains(AbstractCache.java:78) [jboss-as-ejb3-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
	at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.isRemoved(StatefulSessionObjectReferenceImpl.java:164) [jboss-as-weld-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
	at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:123) [jboss-as-weld-7.1.3.Final-redhat-2.jar:7.1.3.Final-redhat-2]
	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:108) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.toString(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) [clusterbench-ee6-ejb.jar:]
	at java.lang.String.valueOf(String.java:2826) [rt.jar:1.6.0_30]
	at java.lang.StringBuilder.append(StringBuilder.java:115) [rt.jar:1.6.0_30]
	at org.jboss.weld.context.SerializableContextualInstanceImpl.toString(SerializableContextualInstanceImpl.java:60) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at java.lang.String.valueOf(String.java:2826) [rt.jar:1.6.0_30]
	at java.lang.StringBuilder.append(StringBuilder.java:115) [rt.jar:1.6.0_30]
	at org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:109) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at org.jboss.weld.context.AbstractBoundContext.activate(AbstractBoundContext.java:66) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:141) [weld-core-1.1.9.Final-redhat-1.jar:1.1.9.Final-redhat-1]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final-redhat-1.jar:]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{code}

This occurred for cache in TERMINATED as well as STOPPING state and triggered code 500 for the client. Although the issue occurred on RHEL6-x86_64 with ipv6 it may not be limited to this environment.

Comment 1 Richard Janík 2012-10-01 14:26:03 UTC
Update: issue replicated on dualstack machines while ipv6 testing as well (manual). This time it was seen after regular failover (4 nodes running, one gets its application undeployed, with the same scenario.

Comment 4 Anne-Louise Tangring 2012-11-13 20:53:13 UTC
Docs QE Status: Removed: NEW 


Comment 5 Jitka Kozana 2012-12-11 15:06:57 UTC
We hit this issue again in EAP 6.0.1.ER4.2 testing, cache REPL sync, failure type: shutdown.
The client side exception:
{code}

The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
	org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:115)
	org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
	org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
	org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
	org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
	org.infinispan.interceptors.IsMarshallableInterceptor.visitGetKeyValueCommand(IsMarshallableInterceptor.java:94)
	org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:80)
	org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
	org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	org.infinispan.CacheImpl.get(CacheImpl.java:278)
	org.infinispan.CacheImpl.get(CacheImpl.java:270)
	org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:303)
	org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:198)
	org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:195)
	org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
	org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
	org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
	org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.get(InfinispanBackingCacheEntryStore.java:201)
	org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:290)
	org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:51)
	org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.peek(PassivatingBackingCacheImpl.java:205)
	org.jboss.as.ejb3.cache.spi.impl.AbstractCache.contains(AbstractCache.java:78)
	org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.isRemoved(StatefulSessionObjectReferenceImpl.java:164)
	org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:123)
	org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:109)
	org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
	org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105)
	org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.toString(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java)
	java.lang.String.valueOf(String.java:2826)
	java.lang.StringBuilder.append(StringBuilder.java:115)
	org.jboss.weld.context.SerializableContextualInstanceImpl.toString(SerializableContextualInstanceImpl.java:60)
	java.lang.String.valueOf(String.java:2826)
	java.lang.StringBuilder.append(StringBuilder.java:115)
	org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:109)
	org.jboss.weld.context.AbstractBoundContext.activate(AbstractBoundContext.java:66)
	org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:141)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372)
	org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
	org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453)
	org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931)
	java.lang.Thread.run(Thread.java:662)
{code}
On server side:
{code}
08:53:28,780 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewTrigger,vmg34/web) Woke up, shouldRecoverViews=false
08:53:29,308 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-18,null) org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl: Received leave request from nodes [vmg36/ejb]
08:53:29,309 DEBUG [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-18,null) Blocking new write commands because we'll soon start a state transfer
08:53:29,309 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-18,null) Waking up cache view installer thread
08:53:29,310 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewTrigger,vmg34/ejb) Woke up, shouldRecoverViews=false
{code}
See the server log here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-sync-modcluster-ipv6/1/artifact/report/config/jboss-vmg34.mw.lab.eng.bos.redhat.com/server.log

Comment 6 jose vazquez 2013-02-06 17:28:34 UTC
Hello I have an enviroment with jboss 5 EAP with this issue, the OS is RHEL5-x86_64 with IPv4, this is the trace of the issue:

2013-02-06 09:42:59,638 WARN  - 159.16.237.114 - /negotiation/tnd/tenderConfiguration.do - [qroproas02#web-th-204435#01648] (ExceptionCatcher) Exception from exceptionCommand 'servlet-exception': javax.servlet.ServletException: java.lang.IllegalStateException: Default cache is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
	at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:286)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:840)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:622)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:560)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:488)
	at org.apache.struts.chain.commands.servlet.PerformForward.handleAsForward(PerformForward.java:113)
	at org.apache.struts.chain.commands.servlet.PerformForward.perform(PerformForward.java:96)
	at org.apache.struts.chain.commands.AbstractPerformForward.execute(AbstractPerformForward.java:54)
	at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51)
	at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
	at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304)
	at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
	at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.ResponseWrapperFilter.doFilter(ResponseWrapperFilter.java:78)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.SerializationCheckFilter.doFilter(SerializationCheckFilter.java:98)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.CrossSiteRequestForgeryFilter.doFilter(CrossSiteRequestForgeryFilter.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.ACLFilter.doFilter(ACLFilter.java:376)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.CacheFilter.doFilter(CacheFilter.java:163)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.EsopFilter.doFilterWork(EsopFilter.java:251)
	at com.bs.esop.foundation.servlet.filter.EsopFilter.doFilter(EsopFilter.java:148)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.DrainFilter.doFilter(DrainFilter.java:137)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at com.bs.esop.foundation.servlet.filter.ResourceGuardianFilter.doFilter(ResourceGuardianFilter.java:84)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:397)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:270)
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:397)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:876)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: Default cache is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:108)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89)
	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:95)
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
	at org.infinispan.CacheImpl.get(CacheImpl.java:272)
	at org.infinispan.CacheImpl.get(CacheImpl.java:264)
	at com.bs.lib.cluster.service.cache.impl.DistributedTreeCache.get(Unknown Source)
	at com.bs.esop.foundation.service.cluster.impl.CacheImpl.get(CacheImpl.java:64)
	at com.bs.lib.cluster.service.cache.util.PrefixedCache.get(Unknown Source)
	at com.bs.esop.foundation.context.visitor.VisitorReplicatedDataManager.getDistributedAttribute(VisitorReplicatedDataManager.java:194)
	at com.bs.esop.foundation.context.VisitorContext.getDistributedAttribute(VisitorContext.java:1218)
	at com.bs.esop.foundation.context.VisitorContext._pushMessages(VisitorContext.java:1285)
	at com.bs.esop.foundation.context.VisitorContext.pushErrors(VisitorContext.java:1327)
	at com.bs.esop.foundation.servlet.struts.BaseAction.addErrorString(BaseAction.java:845)
	at com.bs.esop.foundation.servlet.struts.BaseAction._execute(BaseAction.java:337)
	at com.bs.esop.foundation.servlet.struts.BaseAction.execute(BaseAction.java:141)
	at org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58)
	at org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67)
	at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51)
	at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
	at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304)
	at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
	at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283)
	... 63 more

_______________________________-
_______________________________

I didn't found any documentation about this kind of issue, exist any documentation for troubleshooting notes to fix this ?

How can I get the root cause of this issue?

Comment 7 jose vazquez 2013-02-06 17:30:25 UTC
I´m sorry the jboss version is 7; recently we update this enviroment

Comment 10 Jitka Kozana 2013-07-30 08:52:46 UTC
Seen in 6.1.1.ER3 testing cycle:
04:52:45,903 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (OOB-152,shared=udp) ISPN000016: Problem Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. encountered when applying state for key MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=264789768}@66e6f4c!: java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:115)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
	at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
	at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:470)
	at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:421)
	at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:85)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
	at org.jgroups.JChannel.up(JChannel.java:707)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
	at org.jgroups.protocols.RSVP.up(RSVP.java:172)
	at org.jgroups.protocols.FRAG2.unfragment(FRAG2.java:302)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:162)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765)
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420)
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:645)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
	at org.jgroups.protocols.FD.up(FD.java:253)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

This is most likely the same issue.

Link to server log:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/21/artifact/report/config/jboss-perf21/server.log

Comment 11 Ladislav Thon 2013-08-26 13:14:47 UTC
Still seeing this with EAP 6.1.1.ER7:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/24/

Comment 12 Paul Ferraro 2013-12-06 13:23:00 UTC
This might no longer be an issue.  Please retest.

Comment 13 Richard Janík 2013-12-09 07:07:07 UTC
Seen in 6.2.0.CR3 (~20MB file):

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/34/console-perf20/

It was only seen on this configuration (remote ejbs, netDown, DIST SYNC cache) through 6.2.0.

Stacktrace:

[JBossINF] [0m[33m11:25:18,703 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (OOB-70,shared=udp) ISPN000016: Problem Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. encountered when applying state for key MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=2042881233}@3ec53e!: java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:115)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
[JBossINF] 	at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
[JBossINF] 	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
[JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
[JBossINF] 	at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:470)
[JBossINF] 	at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:421)
[JBossINF] 	at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:85)
[JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
[JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
[JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600)
[JBossINF] 	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
[JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:707)
[JBossINF] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)
[JBossINF] 	at org.jgroups.protocols.RSVP.up(RSVP.java:172)
[JBossINF] 	at org.jgroups.protocols.FRAG2.unfragment(FRAG2.java:302)
[JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:162)
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
[JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
[JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
[JBossINF] 	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765)
[JBossINF] 	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420)
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:645)
[JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
[JBossINF] 	at org.jgroups.protocols.FD.up(FD.java:253)
[JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
[JBossINF] 	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
[JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
[JBossINF] 	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607)
[JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1260)
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822)
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

Comment 14 Ladislav Thon 2014-07-08 13:15:10 UTC
Still an issue (though very rare), moving to 6.4.

Comment 18 Paul Ferraro 2014-09-04 01:50:05 UTC
Deferring clean shutdown issues to EAP7.

Comment 19 JBoss JIRA Server 2015-05-25 20:38:59 UTC
Dan Berindei <dberinde@redhat.com> updated the status of jira ISPN-3743 to Resolved

Comment 20 Prasanth Pagolu 2015-12-17 12:42:46 UTC
Hi Paul,

One of our clients is experiencing similar exception in clustered and replicated JDG environment
version of jdg 6.4.1
version of eap 6.3

08:32:11,004  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704)) **Exception in MessageListener**:
08:32:11,004  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704)) org.infinispan.IllegalLifecycleStateException: ISPN000323: Cache 'jboss.cache:service=TBDGCache' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
08:32:11,004  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91)
08:32:11,005  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
08:32:11,005  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:76)
08:32:11,005  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:44)
08:32:11,005  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
08:32:11,006  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.CacheImpl.get(CacheImpl.java:439)
08:32:11,006  ERROR   [stderr] (TIBCO EMS Session Dispatcher (28132704))        at org.infinispan.CacheImpl.get(CacheImpl.java:431)

Could you please confirm if this issue will be resolved in EAP 6.4.0?
I am currently implementing a workaround to check if the cache status is in TERMINATED state and if yes, to start it manually before get() or put() operations are invoked on the cache. 

Regards,

Prasanth


Note You need to log in before you can comment on or make changes to this bug.