Bug 922699

Summary: [GSS](6.4.z) IllegalStateException: AtomicMap stored under key X has been concurrently removed!
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Jitka Kozana <jkudrnac>
Component: ClusteringAssignee: Radovan Netuka <rnetuka>
Status: CLOSED EOL QA Contact: Pavel Slavicek <pslavice>
Severity: high Docs Contact: Scott Mumford <smumford>
Priority: high    
Version: 6.1.1CC: bmaxwell, carsten.maneg, dereed, lthon, mbabacek, mpark, msochure, mvinkler, myarboro, paul.ferraro, rhusar, rnetuka, sjacobs, smatasar
Target Milestone: ---   
Target Release: EAP 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
An IllegalStateException can be thrown in rare cases when routine processing expiration and passivation accesses a session that is concurrently removed. This causes an exception to be thrown and logged. The exception can be ignored. However, disabling session passivation will ensure the problem is avoided.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-19 12:46:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1036889    
Bug Blocks:    
Attachments:
Description Flags
IllegalStateException_server.log none

Description Jitka Kozana 2013-03-18 10:22:03 UTC
EAP 6.1.0.ER2.

Soak test: cache REPL_ASYNC, http session passivation enabled. 

This came out in server logs during soak test (among many instaces of BZ 901028):
perf20.log:
[JBossINF] ESC[0mESC[33m03:38:18,865 WARN  [org.jboss.as.clustering.web.infinispan] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS010322: Failed to load session uHl2LmYokjR0dOJ6jozR8uY8: java.lang.IllegalStateException: AtomicMap stored under key uHl2LmYokjR0dOJ6jozR8uY8 has been concurrently removed!
[JBossINF]      at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:172)
[JBossINF]      at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:110)
[JBossINF]      at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:220)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:210)
[JBossINF]      at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
[JBossINF]      at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
[JBossINF]      at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:542)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:236)
[JBossINF]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:194)
[JBossINF]      at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1387) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2]
[JBossINF]      at org.jboss.as.web.session.DistributableSessionManager.processExpirationPassivation(DistributableSessionManager.java:1207) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2]
[JBossINF]      at org.jboss.as.web.session.AbstractSessionManager.processExpires(AbstractSessionManager.java:137) [jboss-as-web-7.2.0.Final-redhat-2.jar:7.2.0.Final-redhat-2]
[JBossINF]      at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:367) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1302) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1588) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1574) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF]      at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
[JBossINF] 

before this ISE a few of these were present:
14569478:[JBossINF] 03:38:08,558 ERROR [org.jboss.as.web.session.DistributableSessionManager.clusterbench-passivating] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS018079: Failed to passivate  session uHl2LmYokjR0dOJ6jozR8uY8: java.lang.NullPointerException

Comment 1 Ladislav Thon 2013-03-20 12:43:18 UTC
*** Bug 923785 has been marked as a duplicate of this bug. ***

Comment 2 Ladislav Thon 2013-03-20 14:49:56 UTC
Also seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/23/console-perf18/ (and the exception is preceded by an exception from bug 923818).

Comment 5 Paul Ferraro 2013-08-29 21:43:19 UTC
The new web session clustering implementation in store for EAP 6.3 will render this BZ moot.  It does not use atomic maps.

Comment 6 Paul Ferraro 2013-12-09 14:51:07 UTC
*** Bug 923789 has been marked as a duplicate of this bug. ***

Comment 7 Radoslav Husar 2014-02-10 16:37:22 UTC
This should be retested after BZ#1036889.

Comment 8 Radoslav Husar 2014-03-12 14:15:01 UTC
This needs to be retested.

Comment 10 Michal Karm Babacek 2014-05-09 07:07:22 UTC
Created attachment 893856 [details]
IllegalStateException_server.log

EAP 6.3.0.ER3. It's pretty rare though. 

The web app config is a rudimentary one [1]


[1] https://github.com/Karm/clusterbench/blob/simplified-and-pure/clusterbench-ee6-web/src/main/webapp/WEB-INF/jboss-web.xml

Comment 11 Scott Mumford 2014-05-13 22:54:34 UTC
Marking for exclusion from 6.3.0 Release Notes as per Paul's assertion in comment 5.

Comment 12 Ladislav Thon 2014-05-15 07:33:39 UTC
I don't think comment 5 is in sync with reality (the new clustering implementation didn't make it to 6.3, or did it?), so basing requires_doc_text- on that isn't exactly right. I think Paul should comment here on whether release notes are needed or not.

Comment 13 Radoslav Husar 2014-05-15 11:42:20 UTC
This issue is likely manifestation of Infinispan issue reported here: https://issues.jboss.org/browse/ISPN-2103 (added external tracking link).

One of the workarounds for EAP might be just ignoring the exception whatsover, since this is really just a noise, the session is being removed.

Comment 14 Radoslav Husar 2014-05-15 11:52:01 UTC
Actually, that change from comment #5 was not approved and didn't make it to EAP 6.3 release. So we should provide the doc text as known issue, updating fields now.

Comment 15 Ladislav Thon 2014-07-08 13:32:32 UTC
Still an issue, moving to 6.4.

Comment 16 Paul Ferraro 2014-10-17 12:47:54 UTC
Changing the priority to low - as this is really an edge case (i.e. EJB accessed at the expiration/passivation boundary).  This has already been address upstream in the new EJB clustering implementation.

Comment 22 carsten.maneg 2015-10-17 13:37:03 UTC
Well just an info, as I think its not that rare.
Today our customer has some production problem.
And in the log there are some of those log entries, even this is not the reason for the problem...most probably.
And Upgrade to 6.4 is planned already, so hopefully this will be solved then.

Comment 25 Fedor Gavrilov 2016-11-28 15:06:19 UTC
*** Bug 1377694 has been marked as a duplicate of this bug. ***

Comment 29 Jiří Bílek 2017-02-01 11:51:38 UTC
As I wrote in last comment [1]. The issue is reproducible (e.g. by this job [2]) but very rarely.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=922699#c28
[2] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/