917635 – [GSS](6.4.z) Failed to load session: NullPointerException

Bug 917635 - [GSS](6.4.z) Failed to load session: NullPointerException

Summary: [GSS](6.4.z) Failed to load session: NullPointerException

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	JBoss Enterprise Application Platform 6
Classification:	JBoss
Component:	Clustering
Sub Component:
Version:	6.1.1,6.2.0,6.4.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	GA
Target Release:	EAP 6.4.0
Assignee:	Radoslav Husar
QA Contact:	Jiří Bílek
Docs Contact:	Russell Dickenson
URL:
Whiteboard:
Depends On:	918562 920193 959495
Blocks:
TreeView+	depends on / blocked

Reported:	2013-03-04 12:38 UTC by Richard Janík
Modified:	2017-01-26 14:13 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Clone Of:
Environment:
Last Closed:	2017-01-26 14:13:20 UTC
Type:	Bug
Embargoed:

Attachments	(Terms of Use)

Description Richard Janík 2013-03-04 12:38:34 UTC

Description of problem:

After application deployment an exception gets thrown:

[JBossINF] [0m[33m07:20:02,192 WARN  [org.jboss.as.clustering.web.infinispan] (ServerService Thread Pool -- 58) JBAS010322: Failed to load session +bAA7z+EpDUUdd+WTTpBrps9: java.lang.NullPointerException
[JBossINF] 	at org.jboss.as.clustering.web.impl.IncomingDistributableSessionDataImpl.<init>(IncomingDistributableSessionDataImpl.java:44)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:221)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:210)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:542)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:236)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:206)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.initializeUnloadedSessions(DistributableSessionManager.java:287) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.start(DistributableSessionManager.java:176) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:452) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.JBossContextConfig.processWebMetaData(JBossContextConfig.java:282) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.JBossContextConfig.applicationWebConfig(JBossContextConfig.java:200) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.startup.ContextConfig.start(ContextConfig.java:359) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:139) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.JBossContextConfig.lifecycleEvent(JBossContextConfig.java:193) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardContext.start(StandardContext.java:3717) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.doStart(WebDeploymentService.java:156) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.access$000(WebDeploymentService.java:60) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService$1.run(WebDeploymentService.java:93) [jboss-as-web-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
[JBossINF] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
[JBossINF] 	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Seen here:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-granular-shutdown-repl-sync/17/artifact/report/config/jboss-perf18/server.log
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-jvmkill-repl-sync/47/console-perf18/

EAP version is EAP 6.1.0.ER1

Comment 1 Ladislav Thon 2013-03-26 14:03:30 UTC

I'm still seeing this in EAP 6.1.0.ER3: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-sync/18/console-perf19/

Comment 2 Paul Ferraro 2013-04-29 14:56:30 UTC

Please retest once BZ 900549 is resolved:
https://github.com/jbossas/jboss-eap/pull/122

BZ 900549 would have triggered symptoms like this.

Comment 3 Paul Ferraro 2013-04-30 15:48:16 UTC

This might also be due to ISPN-2974.

Comment 4 Jitka Kozana 2013-05-13 13:30:08 UTC

Still seeing in ER8. 

Link to job: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-undeploy-repl-sync/54/

Link to server.log:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-http-session-undeploy-repl-sync/54/artifact/report/config/jboss-perf18/server.log

Comment 5 Ladislav Thon 2013-08-26 12:48:11 UTC

Still seeing this with EAP 6.1.1.ER7 (only in a single test case this time):

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-netdown-dist-async/30/

Comment 6 Paul Ferraro 2013-08-29 23:15:11 UTC

This will be addressed by the new web session clustering implementation scheduled for 6.3.

Comment 7 Paul Ferraro 2013-12-09 14:02:36 UTC

Please retest.  This may be fixed already.

Comment 8 Richard Janík 2013-12-10 07:02:39 UTC

I'm still seeing this in 6.2.0 GA (CR3).
Link to job:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/67/

Link to server log with the exception:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/67/artifact/report/config/jboss-perf20/server.log

In addition to that there has been a similar "Failed to load session" warning with IllegalStateException as a cause:

14:37:30,453 WARN  [org.jboss.as.clustering.web.infinispan] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS010322: Failed to load session obRLuhOE6WPdrk5JB-CbkXDZ: java.lang.IllegalStateException: AtomicMap stored under key obRLuhOE6WPdrk5JB-CbkXDZ has been concurrently removed!
	at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:171)
	at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:109)
	at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:219)
	at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:212)
	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:85)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:550)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:238)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:196)
	at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1426) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
	at org.jboss.as.web.session.DistributableSessionManager.processExpirationPassivation(DistributableSessionManager.java:1217) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
	at org.jboss.as.web.session.AbstractSessionManager.processExpires(AbstractSessionManager.java:137) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
	at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:367) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1302) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1588) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1600) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1574) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

Link to server log (it is the same build, just different server):

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/67/artifact/report/config/jboss-perf19/server.log

Comment 9 Radoslav Husar 2014-02-10 16:06:04 UTC

This bug should be revalidated after the Infinispan upgrade BZ#1036889.

Comment 10 Jitka Kozana 2014-03-20 14:39:14 UTC

Still seeing this issue with EAP 6.3.0.DR4.

Link to server log:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-repl-sync/103/artifact/report/config/jboss-perf20/server.log

Comment 11 Scott Mumford 2014-05-13 23:00:23 UTC

Added a generic release note text to this bug. If possible, more information about the circumstances under which this exception is encountered would be helpful.

Comment 12 Richard Janík 2014-05-15 08:36:27 UTC

Hi Scott,

I haven't identified any special circumstances. Perhaps Paul can help?

Comment 13 Ladislav Thon 2014-07-08 13:08:21 UTC

Still an issue, moving to 6.4.

Comment 14 Paul Ferraro 2014-10-17 12:53:16 UTC

Setting this to post - to be retested against the Infinispan 5.2.11 release.

Comment 15 Ladislav Thon 2014-10-17 13:09:35 UTC

We're still seeing this with EAP 6.4.0.DR5 [1]. Moving back to ASSIGNED.

The NPE comes from IncomingDistributableSessionDataImpl constructor, so I'd guess it's unlikely to be fixed by Infinispan upgrade.

[1] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-repl-sync/88/console-perf21/

Comment 16 Paul Ferraro 2014-10-17 13:36:34 UTC

"The NPE comes from IncomingDistributableSessionDataImpl constructor, so I'd guess it's unlikely to be fixed by Infinispan upgrade."

The reason for the NPE is a lost atomic map delta write.  The Infinispan 5.2.11 upgrade brought in a fix for this - so, on the contrary, yes, I expected that this might be fully addressed by the upgrade.

Comment 17 MS 2016-07-13 20:42:20 UTC

I'm seeing this issue with JBoss EAP 6.4.5.GA (AS 7.5.5.Final-redhat-3) whioch is using Infinispan 'Delirium' 5.2.15.Final

Any new updates regarding this issue?

JBAS010322: Failed to load session bGc4aKrcbvlkXpDlUUiebI6l: java.lang.IllegalStateException: AtomicMap stored under key bGc4aKrcbvlkXpDlUUiebI6l has been concurrently removed!
	at org.infinispan.atomic.AtomicHashMapProxy.assertValid(AtomicHashMapProxy.java:149)
	at org.infinispan.atomic.AtomicHashMapProxy.getDeltaMapForRead(AtomicHashMapProxy.java:92)
	at org.infinispan.atomic.AtomicHashMapProxy.get(AtomicHashMapProxy.java:197)
	at org.jboss.as.clustering.web.infinispan.SessionMapEntry.get(SessionMapEntry.java:51)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:220)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:212)
	at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:85)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:550)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:238)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:196)
	at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:521) [jboss-as-web-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:488) [jboss-as-web-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.apache.catalina.connector.Request.doGetSession(Request.java:2638) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.Request.getSession(Request.java:2382) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:791) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:73) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:59) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:39) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:22) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:212) [weld-core-1.1.31.Final-redhat-1.jar:1.1.31.Final-redhat-1]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:400) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]

Comment 26 Jiří Bílek 2017-01-26 09:40:16 UTC

EAP QE status: 
Note: Error stacktraces mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=917635#c8 and https://bugzilla.redhat.com/show_bug.cgi?id=917635#c17 are duplicates of BZ922699.

The original issue "WARN: JBAS010322: Failed to load session X java.lang.NullPointerException" has not been hit since EAP 6.4.0 testing.

Comment 27 Radoslav Husar 2017-01-26 14:13:20 UTC

Closing as per last comment from QE. Thanks guys!

Note You need to log in before you can comment on or make changes to this bug.