Bug 1377694

Summary:	IllegalStateException: AtomicMap stored under key X has been concurrently removed (high load, invalidation cache and shared cache store)
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Michal Vinkler <mvinkler>
Component:	Clustering	Assignee:	jboss-set
Status:	CLOSED DUPLICATE	QA Contact:	Michal Vinkler <mvinkler>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.4.8	CC:	fgavrilo, paul.ferraro
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-28 15:06:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Michal Vinkler 2016-09-20 11:54:17 UTC

We can see "IllegalStateException: AtomicMap stored under key X has been concurrently removed!" warning message in the server logs in our failover tests with invalidation cache and shared cache store.

Scenario description:
HTTP traffic accessing clustered web application that has replicated sessions (uses a mod_cluster load balancer). Delay between sending a new request after receiving a response is 1000 ms (for each client). Session size is 34 KB.
4-node EAP cluster + 4-node JDG cluster, one EAP node at time is shut down and after some time started again, while 6000 standalone clients keep calling the application.

Configuration:
 - 4-node EAP cluster with an invalidation cache + a shared cache store (remote JDG cluster)
 - 4-node JDG cluster with distributed cache
 - 4 nodes generating load (6000 clients in total)
 - cache mode: ASYNC (for both invalidation and distributed caches, also "write-behind" element is set for "remote-store" element accordingly)

Links to configuration files:
EAP http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stressfailover-remote-jdg-session-shutdown-invalidation-async-4nodes-perf17/4/artifact/report/config/jboss-perf18/standalone-ha.xml
JDG http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stressfailover-remote-jdg-session-shutdown-invalidation-async-4nodes-perf17/4/artifact/report/config/jboss-perf22/clustered.xml


When EAP server perf19 was shutdown, other servers received a new cluster view and started logging these warning few seconds after:

[JBossINF] [0m[33m22:15:59,287 WARN  [org.jboss.as.clustering.web.infinispan] (ajp-perf20/10.16.90.58:8009-2818) JBAS010322: Failed to load session -dttfB3ao4J2kK0ezrJxEilR: java.lang.IllegalStateException: AtomicMap stored under key -dttfB3ao4J2kK0ezrJxEilR has been concurrently removed!
[JBossINF] 	at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:149) [infinispan-core-5.2.18.Final-redhat-1.jar:5.2.18.Final-redhat-1]
[JBossINF] 	at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:92) [infinispan-core-5.2.18.Final-redhat-1.jar:5.2.18.Final-redhat-1]
[JBossINF] 	at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:197) [infinispan-core-5.2.18.Final-redhat-1.jar:5.2.18.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:212)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:85)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:550)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:238)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:196)
[JBossINF] 	at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:521) [jboss-as-web-7.5.8.Final-redhat-2-bz-1370648.jar:7.5.8.Final-redhat-SNAPSHOT]
[JBossINF] 	at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:488) [jboss-as-web-7.5.8.Final-redhat-2-bz-1370648.jar:7.5.8.Final-redhat-SNAPSHOT]
[JBossINF] 	at org.apache.catalina.connector.Request.doGetSession(Request.java:2638) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.connector.Request.getSession(Request.java:2382) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:791) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:73) [weld-core-1.1.33.Final-redhat-1.jar:1.1.33.Final-redhat-1]
[JBossINF] 	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:59) [weld-core-1.1.33.Final-redhat-1.jar:1.1.33.Final-redhat-1]
[JBossINF] 	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:39) [weld-core-1.1.33.Final-redhat-1.jar:1.1.33.Final-redhat-1]
[JBossINF] 	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:22) [weld-core-1.1.33.Final-redhat-1.jar:1.1.33.Final-redhat-1]
[JBossINF] 	at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:212) [weld-core-1.1.33.Final-redhat-1.jar:1.1.33.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1]
[JBossINF] 	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91]

Link to server log:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stressfailover-remote-jdg-session-shutdown-invalidation-async-4nodes-perf17/4/console-perf20/

Comment 1 Michal Vinkler 2016-09-26 07:57:40 UTC

After further investigation, this seems as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=922699