Bug 1015127 - DIST: Exception: EJB has been removed [NEEDINFO]
DIST: Exception: EJB has been removed
Status: ASSIGNED
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high
: DR1
: EAP 6.4.0
Assigned To: jboss-set
Michal Vinkler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-03 09:33 EDT by Jitka Kozana
Modified: 2017-10-09 20:10 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rhusar: needinfo? (jkudrnac)
myarboro: needinfo? (paul.ferraro)


Attachments (Terms of Use)

  None (edit)
Description Jitka Kozana 2013-10-03 09:33:09 EDT
Scenario: 4 node cluster, DIST SYNC cache with 2 owners, failover testing, failure type: undeploy.

When one node leaves the cluster (= the application is redeployed), the other nodes starts logging this exception:

15:45:29,111 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/clusterbench].[LocalEjbServlet]] (ajp-perf19/10.16.90.56:8009-24) JBWEB000236: Servlet.service() for servlet LocalEjbServlet threw exception: javax.ejb.NoSuchEJBException: JBAS016055: EJB has been removed
	at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:124) [jboss-as-weld-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6]
	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:113) [weld-core-1.1.15.Final-redhat-1.jar:2013-09-20 09:50]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.15.Final-redhat-1.jar:2013-09-20 09:50]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.15.Final-redhat-1.jar:2013-09-20 09:50]
	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java) [clusterbench-ee6-ejb.jar:]
	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_WeldClientProxy.java) [clusterbench-ee6-ejb.jar:]
	at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:21) [classes:]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
	at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6]
	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6]
	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6]
	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-6.jar:7.3.0.Final-redhat-6]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

Not every node starts to log this exception, we have observed this exceptions in log in 3 of 4 server logs. 

On client side, clients gets 500 and a exception:

Invalid response code: 500 Content: <html><head><title>JBoss Web/7.2.2.Final-redhat-1 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 500 - JBAS016055: EJB has been removed</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000066: Exception report</p><p><b>JBWEB000068: message</b> <u>JBAS016055: EJB has been removed</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000145: The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>JBWEB000070: exception</b> <pre>javax.ejb.NoSuchEJBException: JBAS016055: EJB has been removed
        org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.getBusinessObject(StatefulSessionObjectReferenceImpl.java:124)
        org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:113)
        org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
        org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105)
        org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$Proxy$.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_Weld$Proxy$.java)
        org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(LocalStatefulSB$Proxy$_$$_WeldClientProxy.java)
        org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:21)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
</pre></p><p><b>JBWEB000072: note</b> <u>JBWEB000073: The full stack trace of the root cause is available in the JBoss Web/7.2.2.Final-redhat-1 logs.&

Link to job: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/
Links to server logs:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf18/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf19/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf20/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf21/server.log
Comment 1 Paul Ferraro 2013-11-05 17:00:09 EST
From my understanding of this test, org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB is a @SessionScoped bean, which means that the container would normally remove the bean when the session is destroyed.  However, on undeploy, session destruction/unbinding listeners are not invoked, so the EJB should not be getting removed.

Does the infinispan subsystem configuration used by this test include a <file-store/> for the ejb cache container?
Comment 2 Ladislav Thon 2013-11-20 09:46:43 EST
We are doing very little changes to the XML before we start the server[s] and we are not issuing any management operations later. Specifically, the only changes that we do in the infinispan subsystem configuration are:

- set the default cache of some cache containers (repl or dist)
- set the cache mode of some caches (sync or async)
- change number of owners in the dist cache in certain tests

So if the cache container configuration contains <file-store/> by default, which it does, then it is also included in the test.

(We are also changing the cache store in session persistence tests, but these are unrelated.)
Comment 3 Ladislav Thon 2013-11-20 09:55:48 EST
One more detail: the configuration XML can always be found in the test results under report/config/jboss-*/standalone-ha.xml; e.g. for https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/, the server configuration of node perf18 can be found at https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-dist-sync/22/artifact/report/config/jboss-perf18/standalone-ha.xml
Comment 6 Jitka Kozana 2013-12-13 03:48:22 EST
This is the server.log on EAP 6.2.0.GA, cache dist sync, failure type: shutdown:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-dist-async/25/artifact/report/config/jboss-perf19/server.log
Comment 8 Radoslav Husar 2014-02-10 09:48:57 EST
All pointers are leading to dist bug in Infinispan. Please revalidate this bug after the Infinispan upgrade.
Comment 9 Ladislav Thon 2014-03-20 10:55:45 EDT
We are still seeing this with EAP 6.3.0.DR2 and DR4 (didn't look at earlier DRs). In both these DRs, there is only one test that exhibits this problem: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-jvmkill-dist-sync/

Maybe it's expected with jvmkill? (Though I would certainly argue that it isn't.)
Comment 10 Ladislav Thon 2014-07-08 09:07:30 EDT
Still an issue, moving to 6.4.
Comment 11 Paul Ferraro 2014-10-17 10:15:33 EDT
Requires an Infinispan fix and is not feasible for 6.4.

Note You need to log in before you can comment on or make changes to this bug.