Seen this warning during EAP 6.1.0.ER3 testing: 07:07:57,798 WARN [org.infinispan.statetransfer.StateConsumerImpl] (OOB-9,shared=udp) Received unsolicited state from node perf20/web for segment 30 of cache default-host/clusterbench This happened immediately after 07:07:57,792 WARN [org.jboss.as.clustering.web.infinispan] (ajp-perf18/10.16.90.54:8009-40) JBAS010322: Failed to load session 4EZ5ZTuprRHO30+aZKaghPRw: java.lang.IllegalStateException: AtomicMap stored under key 4EZ5ZTuprRHO30+aZKaghPRw has been concurrently removed! at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:172) at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:110) at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:220) at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51) at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:219) at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:210) at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34) at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48) at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81) at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:542) at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:236) at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:194) at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:524) [jboss-as-web-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3] at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:491) [jboss-as-web-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3] at org.apache.catalina.connector.Request.doGetSession(Request.java:2612) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.connector.Request.getSession(Request.java:2357) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:790) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:73) [weld-core-1.1.11.Final-redhat-1.jar:1.1.11.Final-redhat-1] at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:59) [weld-core-1.1.11.Final-redhat-1.jar:1.1.11.Final-redhat-1] at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:31) [weld-core-1.1.11.Final-redhat-1.jar:1.1.11.Final-redhat-1] at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:16) [weld-core-1.1.11.Final-redhat-1.jar:1.1.11.Final-redhat-1] at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:134) [weld-core-1.1.11.Final-redhat-1.jar:1.1.11.Final-redhat-1] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:134) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43] which is bug 922699, but it doesn't look like something related. I can't find anything suspicious in logs of other servers around the same time. Maybe it's harmless? Seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/23/console-perf18/
Still seeing this with EAP 6.1.0.ER8. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-sync/24/artifact/report/config/jboss-perf21/server.log https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/54/artifact/report/config/jboss-perf20/server.log
Still seeing this with EAP 6.1.1.ER7. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-shutdown-dist-sync/26/ https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-dist-async/22/
*** This bug has been marked as a duplicate of bug 922699 ***
Marking for exclusion from Release Notes documentation as a duplicate.