Hide Forgot
Scenario: 4 node cluster, DIST SYNC cache with 2 owners, failover testing, failure type: undeploy. When one node leaves the cluster (= the application is redeployed), the other nodes starts logging this exception: 15:45:29,111 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/clusterbench].[LocalEjbServlet]] (ajp-perf19/10.16.90.56:8009-24) JBWEB000236: Servlet.service() for servlet LocalEjbServlet threw exception: javax.ejb.NoSuchEJBException: JBAS016055: EJB has been removed at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:124) [jboss-as-weld-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6] at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:113) [weld-core-1.1.15.Final-redhat-1.jar:2013-09-20 09:50] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.15.Final-redhat-1.jar:2013-09-20 09:50] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.15.Final-redhat-1.jar:2013-09-20 09:50] at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) [clusterbench-ee6-ejb.jar:] at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_WeldClientProxy.java) [clusterbench-ee6-ejb.jar:] at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:21) [classes:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91) at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6] at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6] at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6] at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] Not every node starts to log this exception, we have observed this exceptions in log in 3 of 4 server logs. On client side, clients gets 500 and a exception: Invalid response code: 500 Content: <html><head><title>JBoss Web/7.2.2.Final-redhat-1 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 500 - JBAS016055: EJB has been removed</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000066: Exception report</p><p><b>JBWEB000068: message</b> <u>JBAS016055: EJB has been removed</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000145: The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>JBWEB000070: exception</b> <pre>javax.ejb.NoSuchEJBException: JBAS016055: EJB has been removed org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:124) org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:113) org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_WeldClientProxy.java) org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:21) javax.servlet.http.HttpServlet.service(HttpServlet.java:734) javax.servlet.http.HttpServlet.service(HttpServlet.java:847) </pre></p><p><b>JBWEB000072: note</b> <u>JBWEB000073: The full stack trace of the root cause is available in the JBoss Web/7.2.2.Final-redhat-1 logs.& Link to job: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/ Links to server logs: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf18/server.log https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf19/server.log https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf20/server.log https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf21/server.log
From my understanding of this test, org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB is a @SessionScoped bean, which means that the container would normally remove the bean when the session is destroyed. However, on undeploy, session destruction/unbinding listeners are not invoked, so the EJB should not be getting removed. Does the infinispan subsystem configuration used by this test include a <file-store/> for the ejb cache container?
We are doing very little changes to the XML before we start the server[s] and we are not issuing any management operations later. Specifically, the only changes that we do in the infinispan subsystem configuration are: - set the default cache of some cache containers (repl or dist) - set the cache mode of some caches (sync or async) - change number of owners in the dist cache in certain tests So if the cache container configuration contains <file-store/> by default, which it does, then it is also included in the test. (We are also changing the cache store in session persistence tests, but these are unrelated.)
One more detail: the configuration XML can always be found in the test results under report/config/jboss-*/standalone-ha.xml; e.g. for https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/, the server configuration of node perf18 can be found at https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf18/standalone-ha.xml
This is the server.log on EAP 6.2.0.GA, cache dist sync, failure type: shutdown: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-dist-async/25/artifact/report/config/jboss-perf19/server.log
All pointers are leading to dist bug in Infinispan. Please revalidate this bug after the Infinispan upgrade.
We are still seeing this with EAP 6.3.0.DR2 and DR4 (didn't look at earlier DRs). In both these DRs, there is only one test that exhibits this problem: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-jvmkill-dist-sync/ Maybe it's expected with jvmkill? (Though I would certainly argue that it isn't.)
Still an issue, moving to 6.4.
Requires an Infinispan fix and is not feasible for 6.4.