| Summary: | Operation ("clean-obsolete-content") failed - address: ([]): java.lang.NullPointerException (high load, invalidation cache and shared cache store) | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Data Grid 6 | Reporter: | Michal Vinkler <mvinkler> |
| Component: | Server | Assignee: | Tristan Tarrant <ttarrant> |
| Status: | NEW --- | QA Contact: | Martin Gencur <mgencur> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.6.0 | CC: | jdg-bugs |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
If this is a bug, it is inherited from the EAP bits used by JDG server. |
We can see "Operation ("clean-obsolete-content") failed - address: ([]): java.lang.NullPointerException" error message in the JDG server logs in our failover tests with invalidation cache and shared cache store. Scenario description: HTTP traffic accessing clustered web application that has replicated sessions (uses a mod_cluster load balancer). Delay between sending a new request after receiving a response is 1000 ms (for each client). Session size is 34 KB. 4-node EAP cluster + 4-node JDG cluster, one EAP node at time is shut down and after some time started again, while 6000 standalone clients keep calling the application. Configuration: - 4-node EAP cluster with an invalidation cache + a shared cache store (remote JDG cluster) - 4-node JDG cluster with distributed cache - 4 nodes generating load (6000 clients in total) - cache mode: ASYNC or SYNC (for both invalidation and distributed caches, also "write-behind" element is set for "remote-store" element accordingly) - Links to configuration files: EAP http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stressfailover-remote-jdg-session-shutdown-invalidation-async-4nodes-perf17/4/artifact/report/config/jboss-perf18/standalone-ha.xml JDG http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stressfailover-remote-jdg-session-shutdown-invalidation-async-4nodes-perf17/4/artifact/report/config/jboss-perf22/clustered.xml One of the four JDG servers started logging this error message and continued logging it every 5 minutes until it was finally shut down in the end of the test. [JBossINF] [0m[31m23:51:55,948 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 18) JBAS014612: Operation ("clean-obsolete-content") failed - address: ([]): java.lang.NullPointerException [JBossINF] at org.jboss.as.repository.ContentRepository$Factory$ContentRepositoryImpl.listLocalContents(ContentRepository.java:436) [JBossINF] at org.jboss.as.repository.ContentRepository$Factory$ContentRepositoryImpl.cleanObsoleteContent(ContentRepository.java:398) [JBossINF] at org.jboss.as.server.operations.CleanObsoleteContentHandler.execute(CleanObsoleteContentHandler.java:75) [JBossINF] at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:563) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:553) [jboss-as-controller-7.5.2.Final-redhat-2.jar:7.5.2.Final-redhat-2] [JBossINF] at org.jboss.as.server.deployment.ContentRepositoryCleaner.cleanObsoleteContent(ContentRepositoryCleaner.java:132) [JBossINF] at org.jboss.as.server.deployment.ContentRepositoryCleaner$ContentRepositoryCleanerTask.run(ContentRepositoryCleaner.java:67) [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_91] [JBossINF] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_91] [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_91] [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [rt.jar:1.8.0_91] [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_91] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_91] [JBossINF] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91] [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1] Link to the JDG server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stressfailover-remote-jdg-session-shutdown-invalidation-async-4nodes-perf17/4/console-perf22/