Bug 900946 (JBPAPP6-1274)
| Summary: | IllegalStateException: Cache is in 'TERMINATED'/'STOPPING' state. | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Richard Janík <rjanik> |
| Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> |
| Status: | CLOSED EOL | QA Contact: | Michal Vinkler <mvinkler> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.1.1, 6.2.0 | CC: | jkudrnac, jlvazquez, lthon, myarboro, ppagolu, rhusar, rjanik, rsvoboda |
| Target Milestone: | GA | ||
| Target Release: | TBD EAP 7 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | http://jira.jboss.org/jira/browse/JBPAPP6-1274 | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Known Issue | |
| Doc Text: |
This release of JBoss EAP 6 carries a bug that may cause an `IllegalStateException` to appear after an application is undeployed on one node within 5 seconds of having been deployed on another node within the cluster.
This issue in under investigation and is expected to be resolved in a future release.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-19 12:49:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Richard Janík
2012-09-27 10:43:47 UTC
Update: issue replicated on dualstack machines while ipv6 testing as well (manual). This time it was seen after regular failover (4 nodes running, one gets its application undeployed, with the same scenario. Docs QE Status: Removed: NEW We hit this issue again in EAP 6.0.1.ER4.2 testing, cache REPL sync, failure type: shutdown.
The client side exception:
{code}
The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:115)
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
org.infinispan.interceptors.IsMarshallableInterceptor.visitGetKeyValueCommand(IsMarshallableInterceptor.java:94)
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:80)
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:104)
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60)
org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
org.infinispan.CacheImpl.get(CacheImpl.java:278)
org.infinispan.CacheImpl.get(CacheImpl.java:270)
org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:303)
org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:198)
org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore$2.invoke(InfinispanBackingCacheEntryStore.java:195)
org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.get(InfinispanBackingCacheEntryStore.java:201)
org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:290)
org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.get(SerializationGroupMemberContainer.java:51)
org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.peek(PassivatingBackingCacheImpl.java:205)
org.jboss.as.ejb3.cache.spi.impl.AbstractCache.contains(AbstractCache.java:78)
org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.isRemoved(StatefulSessionObjectReferenceImpl.java:164)
org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:123)
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:109)
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105)
org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.toString(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java)
java.lang.String.valueOf(String.java:2826)
java.lang.StringBuilder.append(StringBuilder.java:115)
org.jboss.weld.context.SerializableContextualInstanceImpl.toString(SerializableContextualInstanceImpl.java:60)
java.lang.String.valueOf(String.java:2826)
java.lang.StringBuilder.append(StringBuilder.java:115)
org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:109)
org.jboss.weld.context.AbstractBoundContext.activate(AbstractBoundContext.java:66)
org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:141)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372)
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931)
java.lang.Thread.run(Thread.java:662)
{code}
On server side:
{code}
08:53:28,780 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewTrigger,vmg34/web) Woke up, shouldRecoverViews=false
08:53:29,308 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-18,null) org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl: Received leave request from nodes [vmg36/ejb]
08:53:29,309 DEBUG [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-18,null) Blocking new write commands because we'll soon start a state transfer
08:53:29,309 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-18,null) Waking up cache view installer thread
08:53:29,310 TRACE [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewTrigger,vmg34/ejb) Woke up, shouldRecoverViews=false
{code}
See the server log here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-sync-modcluster-ipv6/1/artifact/report/config/jboss-vmg34.mw.lab.eng.bos.redhat.com/server.log
Hello I have an enviroment with jboss 5 EAP with this issue, the OS is RHEL5-x86_64 with IPv4, this is the trace of the issue: 2013-02-06 09:42:59,638 WARN - 159.16.237.114 - /negotiation/tnd/tenderConfiguration.do - [qroproas02#web-th-204435#01648] (ExceptionCatcher) Exception from exceptionCommand 'servlet-exception': javax.servlet.ServletException: java.lang.IllegalStateException: Default cache is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container. at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:286) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462) at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:840) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:622) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:560) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:488) at org.apache.struts.chain.commands.servlet.PerformForward.handleAsForward(PerformForward.java:113) at org.apache.struts.chain.commands.servlet.PerformForward.perform(PerformForward.java:96) at org.apache.struts.chain.commands.AbstractPerformForward.execute(AbstractPerformForward.java:54) at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51) at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190) at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304) at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190) at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462) at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.ResponseWrapperFilter.doFilter(ResponseWrapperFilter.java:78) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.SerializationCheckFilter.doFilter(SerializationCheckFilter.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.CrossSiteRequestForgeryFilter.doFilter(CrossSiteRequestForgeryFilter.java:164) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.ACLFilter.doFilter(ACLFilter.java:376) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.CacheFilter.doFilter(CacheFilter.java:163) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.EsopFilter.doFilterWork(EsopFilter.java:251) at com.bs.esop.foundation.servlet.filter.EsopFilter.doFilter(EsopFilter.java:148) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.DrainFilter.doFilter(DrainFilter.java:137) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at com.bs.esop.foundation.servlet.filter.ResourceGuardianFilter.doFilter(ResourceGuardianFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:397) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:270) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:397) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:876) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.IllegalStateException: Default cache is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:108) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:95) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:60) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) at org.infinispan.CacheImpl.get(CacheImpl.java:272) at org.infinispan.CacheImpl.get(CacheImpl.java:264) at com.bs.lib.cluster.service.cache.impl.DistributedTreeCache.get(Unknown Source) at com.bs.esop.foundation.service.cluster.impl.CacheImpl.get(CacheImpl.java:64) at com.bs.lib.cluster.service.cache.util.PrefixedCache.get(Unknown Source) at com.bs.esop.foundation.context.visitor.VisitorReplicatedDataManager.getDistributedAttribute(VisitorReplicatedDataManager.java:194) at com.bs.esop.foundation.context.VisitorContext.getDistributedAttribute(VisitorContext.java:1218) at com.bs.esop.foundation.context.VisitorContext._pushMessages(VisitorContext.java:1285) at com.bs.esop.foundation.context.VisitorContext.pushErrors(VisitorContext.java:1327) at com.bs.esop.foundation.servlet.struts.BaseAction.addErrorString(BaseAction.java:845) at com.bs.esop.foundation.servlet.struts.BaseAction._execute(BaseAction.java:337) at com.bs.esop.foundation.servlet.struts.BaseAction.execute(BaseAction.java:141) at org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58) at org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67) at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51) at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190) at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304) at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190) at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283) ... 63 more _______________________________- _______________________________ I didn't found any documentation about this kind of issue, exist any documentation for troubleshooting notes to fix this ? How can I get the root cause of this issue? I´m sorry the jboss version is 7; recently we update this enviroment Seeing this in EAP 6.1.0.ER2 too: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/13/console-perf20/ Seen again in ER8: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-undeploy-dist-async/28/ Seen in 6.1.1.ER3 testing cycle:
04:52:45,903 WARN [org.infinispan.statetransfer.StateConsumerImpl] (OOB-152,shared=udp) ISPN000016: Problem Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. encountered when applying state for key MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=264789768}@66e6f4c!: java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:115)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:470)
at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:421)
at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:85)
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
at org.jgroups.JChannel.up(JChannel.java:707)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
at org.jgroups.protocols.RSVP.up(RSVP.java:172)
at org.jgroups.protocols.FRAG2.unfragment(FRAG2.java:302)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:162)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765)
at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:645)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
at org.jgroups.protocols.FD.up(FD.java:253)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
at org.jgroups.protocols.Discovery.up(Discovery.java:359)
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
This is most likely the same issue.
Link to server log:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/21/artifact/report/config/jboss-perf21/server.log
Still seeing this with EAP 6.1.1.ER7: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/24/ This might no longer be an issue. Please retest. Seen in 6.2.0.CR3 (~20MB file): https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-netDown-dist-sync/34/console-perf20/ It was only seen on this configuration (remote ejbs, netDown, DIST SYNC cache) through 6.2.0. Stacktrace: [JBossINF] [0m[33m11:25:18,703 WARN [org.infinispan.statetransfer.StateConsumerImpl] (OOB-70,shared=udp) ISPN000016: Problem Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. encountered when applying state for key MarshalledValue{instance=<serialized>, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=2042881233}@3ec53e!: java.lang.IllegalStateException: Cache 'org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container. [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:115) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [JBossINF] at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [JBossINF] at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:470) [JBossINF] at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:421) [JBossINF] at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:85) [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101) [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122) [JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [JBossINF] at org.jgroups.JChannel.up(JChannel.java:707) [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025) [JBossINF] at org.jgroups.protocols.RSVP.up(RSVP.java:172) [JBossINF] at org.jgroups.protocols.FRAG2.unfragment(FRAG2.java:302) [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:162) [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [JBossINF] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765) [JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:645) [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) [JBossINF] at org.jgroups.protocols.FD.up(FD.java:253) [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [JBossINF] at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607) [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1260) [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822) [JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] Still an issue (though very rare), moving to 6.4. Deferring clean shutdown issues to EAP7. Dan Berindei <dberinde> updated the status of jira ISPN-3743 to Resolved Hi Paul, One of our clients is experiencing similar exception in clustered and replicated JDG environment version of jdg 6.4.1 version of eap 6.3 08:32:11,004 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) **Exception in MessageListener**: 08:32:11,004 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) org.infinispan.IllegalLifecycleStateException: ISPN000323: Cache 'jboss.cache:service=TBDGCache' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container. 08:32:11,004 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91) 08:32:11,005 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) 08:32:11,005 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:76) 08:32:11,005 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:44) 08:32:11,005 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) 08:32:11,006 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.CacheImpl.get(CacheImpl.java:439) 08:32:11,006 ERROR [stderr] (TIBCO EMS Session Dispatcher (28132704)) at org.infinispan.CacheImpl.get(CacheImpl.java:431) Could you please confirm if this issue will be resolved in EAP 6.4.0? I am currently implementing a workaround to check if the cache status is in TERMINATED state and if yes, to start it manually before get() or put() operations are invoked on the cache. Regards, Prasanth |