Bug 917635

Summary: [GSS](6.4.z) Failed to load session: NullPointerException
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Richard Janík <rjanik>
Component: ClusteringAssignee: Radoslav Husar <rhusar>
Status: CLOSED CURRENTRELEASE QA Contact: Jiří Bílek <jbilek>
Severity: high Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.1.1, 6.2.0, 6.4.2CC: bmaxwell, jbilek, jkudrnac, lthon, mahesh_soori, myarboro, paul.ferraro, ppalaga, rhusar, rjanik, rstancel, smatasar, smumford
Target Milestone: GA   
Target Release: EAP 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
A Known Issue in this release can cause a NullPointerException with a 'Failed to load session' message to be encountered after application deployment in some circumstances. This issue is expected to be resolved in a later release of the product.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-26 14:13:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 918562, 920193, 959495    
Bug Blocks:    

Description Richard Janík 2013-03-04 12:38:34 UTC 
Description of problem:

After application deployment an exception gets thrown:

[JBossINF] [0m[33m07:20:02,192 WARN  [org.jboss.as.clustering.web.infinispan] (ServerService Thread Pool -- 58) JBAS010322: Failed to load session +bAA7z+EpDUUdd+WTTpBrps9: java.lang.NullPointerException
[JBossINF] 	at org.jboss.as.clustering.web.impl.IncomingDistributableSessionDataImpl.<init>(IncomingDistributableSessionDataImpl.java:44)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:221)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:210)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:542)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:236)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:206)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.initializeUnloadedSessions(DistributableSessionManager.java:287) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.start(DistributableSessionManager.java:176) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:452) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.JBossContextConfig.processWebMetaData(JBossContextConfig.java:282) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.JBossContextConfig.applicationWebConfig(JBossContextConfig.java:200) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.startup.ContextConfig.start(ContextConfig.java:359) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:139) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.JBossContextConfig.lifecycleEvent(JBossContextConfig.java:193) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardContext.start(StandardContext.java:3717) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.doStart(WebDeploymentService.java:156) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.access$000(WebDeploymentService.java:60) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService$1.run(WebDeploymentService.java:93) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
[JBossINF] 	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Seen here:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-granular-shutdown-repl-sync/17/artifact/report/config/jboss-perf18/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-jvmkill-repl-sync/47/console-perf18/

EAP version is EAP 6.1.0.ER1
Comment 1 Ladislav Thon 2013-03-26 14:03:30 UTC 
I'm still seeing this in EAP 6.1.0.ER3: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-sync/18/console-perf19/
Comment 2 Paul Ferraro 2013-04-29 14:56:30 UTC 
Please retest once BZ 900549 is resolved:
https://github.com/jbossas/jboss-eap/pull/122

BZ 900549 would have triggered symptoms like this.
Comment 3 Paul Ferraro 2013-04-30 15:48:16 UTC 
This might also be due to ISPN-2974.
Comment 4 Jitka Kozana 2013-05-13 13:30:08 UTC 
Still seeing in ER8. 

Link to job: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-undeploy-repl-sync/54/

Link to server.log:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-undeploy-repl-sync/54/artifact/report/config/jboss-perf18/server.log
Comment 5 Ladislav Thon 2013-08-26 12:48:11 UTC 
Still seeing this with EAP 6.1.1.ER7 (only in a single test case this time):

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-netdown-dist-async/30/
Comment 6 Paul Ferraro 2013-08-29 23:15:11 UTC 
This will be addressed by the new web session clustering implementation scheduled for 6.3.
Comment 7 Paul Ferraro 2013-12-09 14:02:36 UTC 
Please retest.  This may be fixed already.
Comment 8 Richard Janík 2013-12-10 07:02:39 UTC 
I'm still seeing this in 6.2.0 GA (CR3).
Link to job:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/67/

Link to server log with the exception:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/67/artifact/report/config/jboss-perf20/server.log

In addition to that there has been a similar "Failed to load session" warning with IllegalStateException as a cause:

14:37:30,453 WARN  [org.jboss.as.clustering.web.infinispan] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS010322: Failed to load session obRLuhOE6WPdrk5JB-CbkXDZ: java.lang.IllegalStateException: AtomicMap stored under key obRLuhOE6WPdrk5JB-CbkXDZ has been concurrently removed!
	at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:171)
	at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:109)
	at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:219)
	at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:212)
	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:85)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:550)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:238)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:196)
	at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1426) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
	at org.jboss.as.web.session.DistributableSessionManager.processExpirationPassivation(DistributableSessionManager.java:1217) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
	at org.jboss.as.web.session.AbstractSessionManager.processExpires(AbstractSessionManager.java:137) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
	at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:367) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1302) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1588) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1574) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

Link to server log (it is the same build, just different server):

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/67/artifact/report/config/jboss-perf19/server.log
Comment 9 Radoslav Husar 2014-02-10 16:06:04 UTC 
This bug should be revalidated after the Infinispan upgrade BZ#1036889.
Comment 10 Jitka Kozana 2014-03-20 14:39:14 UTC 
Still seeing this issue with EAP 6.3.0.DR4.

Link to server log:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-repl-sync/103/artifact/report/config/jboss-perf20/server.log
Comment 11 Scott Mumford 2014-05-13 23:00:23 UTC 
Added a generic release note text to this bug. If possible, more information about the circumstances under which this exception is encountered would be helpful.
Comment 12 Richard Janík 2014-05-15 08:36:27 UTC 
Hi Scott,

I haven't identified any special circumstances. Perhaps Paul can help?
Comment 13 Ladislav Thon 2014-07-08 13:08:21 UTC 
Still an issue, moving to 6.4.
Comment 14 Paul Ferraro 2014-10-17 12:53:16 UTC 
Setting this to post - to be retested against the Infinispan 5.2.11 release.
Comment 15 Ladislav Thon 2014-10-17 13:09:35 UTC 
We're still seeing this with EAP 6.4.0.DR5 [1]. Moving back to ASSIGNED.

The NPE comes from IncomingDistributableSessionDataImpl constructor, so I'd guess it's unlikely to be fixed by Infinispan upgrade.

[1] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-repl-sync/88/console-perf21/
Comment 16 Paul Ferraro 2014-10-17 13:36:34 UTC 
"The NPE comes from IncomingDistributableSessionDataImpl constructor, so I'd guess it's unlikely to be fixed by Infinispan upgrade."

The reason for the NPE is a lost atomic map delta write.  The Infinispan 5.2.11 upgrade brought in a fix for this - so, on the contrary, yes, I expected that this might be fully addressed by the upgrade.
Comment 17 MS 2016-07-13 20:42:20 UTC 
I'm seeing this issue with JBoss EAP 6.4.5.GA (AS 7.5.5.Final-redhat-3) whioch is using Infinispan 'Delirium' 5.2.15.Final

Any new updates regarding this issue?

JBAS010322: Failed to load session bGc4aKrcbvlkXpDlUUiebI6l: java.lang.IllegalStateException: AtomicMap stored under key bGc4aKrcbvlkXpDlUUiebI6l has been concurrently removed!
	at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:149)
	at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:92)
	at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:197)
	at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:212)
	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:85)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:550)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:238)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:196)
	at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:521) [jboss-as-web-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:488) [jboss-as-web-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.apache.catalina.connector.Request.doGetSession(Request.java:2638) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.Request.getSession(Request.java:2382) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:791) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:73) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:59) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:39) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:22) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:212) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:400) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]
Comment 26 Jiří Bílek 2017-01-26 09:40:16 UTC 
EAP QE status: 
Note: Error stacktraces mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=917635#c8 and https://bugzilla.redhat.com/show_bug.cgi?id=917635#c17 are duplicates of BZ922699.

The original issue "WARN: JBAS010322: Failed to load session X java.lang.NullPointerException" has not been hit since EAP 6.4.0 testing.
Comment 27 Radoslav Husar 2017-01-26 14:13:20 UTC 
Closing as per last comment from QE. Thanks guys!