Bug 922699 - [GSS](6.4.z) IllegalStateException: AtomicMap stored under key X has been concurrently removed!
Summary: [GSS](6.4.z) IllegalStateException: AtomicMap stored under key X has been con...
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering
Version: 6.1.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: EAP 6.4.0
Assignee: Radovan Netuka
QA Contact: Pavel Slavicek
Scott Mumford
: 923785 1377694 (view as bug list)
Depends On: 1036889
TreeView+ depends on / blocked
Reported: 2013-03-18 10:22 UTC by Jitka Kozana
Modified: 2021-03-11 14:11 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
An IllegalStateException can be thrown in rare cases when routine processing expiration and passivation accesses a session that is concurrently removed. This causes an exception to be thrown and logged. The exception can be ignored. However, disabling session passivation will ensure the problem is avoided.
Clone Of:
Last Closed: 2019-08-19 12:46:51 UTC
Type: Bug

Attachments (Terms of Use)
IllegalStateException_server.log (22.99 KB, text/plain)
2014-05-09 07:07 UTC, Michal Karm Babacek
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-2103 0 Major Closed Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other thr... 2019-08-01 22:14:57 UTC

Description Jitka Kozana 2013-03-18 10:22:03 UTC
EAP 6.1.0.ER2.

Soak test: cache REPL_ASYNC, http session passivation enabled. 

This came out in server logs during soak test (among many instaces of BZ 901028):
[JBossINF] ESC[0mESC[33m03:38:18,865 WARN  [org.jboss.as.clustering.web.infinispan] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS010322: Failed to load session uHl2LmYokjR0dOJ6jozR8uY8: java.lang.IllegalStateException: AtomicMap stored under key uHl2LmYokjR0dOJ6jozR8uY8 has been concurrently removed!
[JBossINF]      at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:172)
[JBossINF]      at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:110)
[JBossINF]      at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:220)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:210)
[JBossINF]      at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
[JBossINF]      at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
[JBossINF]      at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:542)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:236)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:194)
[JBossINF]      at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1387) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2]
[JBossINF]      at org.jboss.as.web.session.DistributableSessionManager.processExpirationPassivation(DistributableSessionManager.java:1207) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2]
[JBossINF]      at org.jboss.as.web.session.AbstractSessionManager.processExpires(AbstractSessionManager.java:137) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2]
[JBossINF]      at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:367) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1302) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1588) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1574) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]

before this ISE a few of these were present:
14569478:[JBossINF] 03:38:08,558 ERROR [org.jboss.as.web.session.DistributableSessionManager.clusterbench-passivating] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS018079: Failed to passivate  session uHl2LmYokjR0dOJ6jozR8uY8: java.lang.NullPointerException

Comment 1 Ladislav Thon 2013-03-20 12:43:18 UTC
*** Bug 923785 has been marked as a duplicate of this bug. ***

Comment 2 Ladislav Thon 2013-03-20 14:49:56 UTC
Also seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/23/console-perf18/ (and the exception is preceded by an exception from bug 923818).

Comment 5 Paul Ferraro 2013-08-29 21:43:19 UTC
The new web session clustering implementation in store for EAP 6.3 will render this BZ moot.  It does not use atomic maps.

Comment 6 Paul Ferraro 2013-12-09 14:51:07 UTC
*** Bug 923789 has been marked as a duplicate of this bug. ***

Comment 7 Radoslav Husar 2014-02-10 16:37:22 UTC
This should be retested after BZ#1036889.

Comment 8 Radoslav Husar 2014-03-12 14:15:01 UTC
This needs to be retested.

Comment 10 Michal Karm Babacek 2014-05-09 07:07:22 UTC
Created attachment 893856 [details]

EAP 6.3.0.ER3. It's pretty rare though. 

The web app config is a rudimentary one [1]

[1] https://github.com/Karm/clusterbench/blob/simplified-and-pure/clusterbench-ee6-web/src/main/webapp/WEB-INF/jboss-web.xml

Comment 11 Scott Mumford 2014-05-13 22:54:34 UTC
Marking for exclusion from 6.3.0 Release Notes as per Paul's assertion in comment 5.

Comment 12 Ladislav Thon 2014-05-15 07:33:39 UTC
I don't think comment 5 is in sync with reality (the new clustering implementation didn't make it to 6.3, or did it?), so basing requires_doc_text- on that isn't exactly right. I think Paul should comment here on whether release notes are needed or not.

Comment 13 Radoslav Husar 2014-05-15 11:42:20 UTC
This issue is likely manifestation of Infinispan issue reported here: https://issues.jboss.org/browse/ISPN-2103 (added external tracking link).

One of the workarounds for EAP might be just ignoring the exception whatsover, since this is really just a noise, the session is being removed.

Comment 14 Radoslav Husar 2014-05-15 11:52:01 UTC
Actually, that change from comment #5 was not approved and didn't make it to EAP 6.3 release. So we should provide the doc text as known issue, updating fields now.

Comment 15 Ladislav Thon 2014-07-08 13:32:32 UTC
Still an issue, moving to 6.4.

Comment 16 Paul Ferraro 2014-10-17 12:47:54 UTC
Changing the priority to low - as this is really an edge case (i.e. EJB accessed at the expiration/passivation boundary).  This has already been address upstream in the new EJB clustering implementation.

Comment 22 carsten.maneg 2015-10-17 13:37:03 UTC
Well just an info, as I think its not that rare.
Today our customer has some production problem.
And in the log there are some of those log entries, even this is not the reason for the problem...most probably.
And Upgrade to 6.4 is planned already, so hopefully this will be solved then.

Comment 25 Fedor Gavrilov 2016-11-28 15:06:19 UTC
*** Bug 1377694 has been marked as a duplicate of this bug. ***

Comment 29 Jiří Bílek 2017-02-01 11:51:38 UTC
As I wrote in last comment [1]. The issue is reproducible (e.g. by this job [2]) but very rarely.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=922699#c28
[2] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/

Note You need to log in before you can comment on or make changes to this bug.