EAP 6.1.0.ER2. Soak test: cache REPL_ASYNC, http session passivation enabled. This came out in server logs during soak test (among many instaces of BZ 901028): perf20.log: [JBossINF] ESC[0mESC[33m03:38:18,865 WARN [org.jboss.as.clustering.web.infinispan] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS010322: Failed to load session uHl2LmYokjR0dOJ6jozR8uY8: java.lang.IllegalStateException: AtomicMap stored under key uHl2LmYokjR0dOJ6jozR8uY8 has been concurrently removed! [JBossINF] at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:172) [JBossINF] at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:110) [JBossINF] at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:220) [JBossINF] at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:210) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48) [JBossINF] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:542) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:236) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:194) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1387) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.processExpirationPassivation(DistributableSessionManager.java:1207) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2] [JBossINF] at org.jboss.as.web.session.AbstractSessionManager.processExpires(AbstractSessionManager.java:137) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2] [JBossINF] at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:367) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1302) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1588) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] [JBossINF] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1574) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43] [JBossINF] before this ISE a few of these were present: 14569478:[JBossINF] 03:38:08,558 ERROR [org.jboss.as.web.session.DistributableSessionManager.clusterbench-passivating] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS018079: Failed to passivate session uHl2LmYokjR0dOJ6jozR8uY8: java.lang.NullPointerException
*** Bug 923785 has been marked as a duplicate of this bug. ***
Also seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/23/console-perf18/ (and the exception is preceded by an exception from bug 923818).
Still seeing this with EAP 6.1.0.ER8. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-sync/24/artifact/report/config/jboss-perf21/server.log https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/54/artifact/report/config/jboss-perf20/server.log
Still seeing this with EAP 6.1.1.ER7. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/34/ https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-jvmkill-dist-sync/15/
The new web session clustering implementation in store for EAP 6.3 will render this BZ moot. It does not use atomic maps.
*** Bug 923789 has been marked as a duplicate of this bug. ***
This should be retested after BZ#1036889.
This needs to be retested.
We are still seeing this with EAP 6.3.0.DR2 and DR4. For DR2, it's here: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/45/console-perf18/ For DR4, it's here: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-sync/43/console-perf20/
Created attachment 893856 [details] IllegalStateException_server.log EAP 6.3.0.ER3. It's pretty rare though. The web app config is a rudimentary one [1] [1] https://github.com/Karm/clusterbench/blob/simplified-and-pure/clusterbench-ee6-web/src/main/webapp/WEB-INF/jboss-web.xml
Marking for exclusion from 6.3.0 Release Notes as per Paul's assertion in comment 5.
I don't think comment 5 is in sync with reality (the new clustering implementation didn't make it to 6.3, or did it?), so basing requires_doc_text- on that isn't exactly right. I think Paul should comment here on whether release notes are needed or not.
This issue is likely manifestation of Infinispan issue reported here: https://issues.jboss.org/browse/ISPN-2103 (added external tracking link). One of the workarounds for EAP might be just ignoring the exception whatsover, since this is really just a noise, the session is being removed.
Actually, that change from comment #5 was not approved and didn't make it to EAP 6.3 release. So we should provide the doc text as known issue, updating fields now.
Still an issue, moving to 6.4.
Changing the priority to low - as this is really an edge case (i.e. EJB accessed at the expiration/passivation boundary). This has already been address upstream in the new EJB clustering implementation.
Well just an info, as I think its not that rare. Today our customer has some production problem. And in the log there are some of those log entries, even this is not the reason for the problem...most probably. And Upgrade to 6.4 is planned already, so hopefully this will be solved then.
*** Bug 1377694 has been marked as a duplicate of this bug. ***
As I wrote in last comment [1]. The issue is reproducible (e.g. by this job [2]) but very rarely. [1] https://bugzilla.redhat.com/show_bug.cgi?id=922699#c28 [2] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/