Hide Forgot
Created attachment 1202456 [details] performance comparison Description of problem: We can see a performance regression in our stress tests, which test performance of the cluster under increasing load (number of concurrent clients). The regression can be seen when using an invalidation cache + a shared cache store compared to using distributed cache without any cache store. Scenario description: HTTP traffic accessing clustered web application that has replicated sessions (uses a mod_cluster load balancer). Delay between sending a new request after receiving a response is 100 ms (for each client). Session size is 34 KB. Configuration: - 4-node EAP cluster with an invalidation cache + a shared cache store (remote JDG cluster) - 4-node JDG cluster with distributed cache - 4 nodes generating load - cache mode: ASYNC or SYNC (for both invalidation and distributed caches, also "write-behind" element is set for "remote-store" element accordingly) Links to configuration files: SYNC scenario: EAP http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-remote-jdg-session-invalidation-sync-4nodes-perf17/3/artifact/report/config/jboss-perf18/standalone-ha.xml JDG http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-remote-jdg-session-invalidation-sync-4nodes-perf17/3/artifact/report/config/jboss-perf22/clustered.xml ASYNC scenario: EAP http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-remote-jdg-session-invalidation-async-4nodes-perf17/3/artifact/report/config/jboss-perf18/standalone-ha.xml JDG http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-remote-jdg-session-invalidation-async-4nodes-perf17/3/artifact/report/config/jboss-perf22/clustered.xml See the attachment "stress-test.png" for graphs with results. There are two scenarios (#1 SYNC, #2 ASYNC). Each scenario compares two runs. The graph on the left uses invalidation cache while the run on the right uses distributed cache. Link to full report with extended CPU and memory statistics: http://download.eng.brq.redhat.com/scratch/mvinkler/reports/2016-09-19_13-48-52/stress.html Horizontal axis represents number of concurrent clients generating load. Vertical axis represents throughput = TPS (number of requests processed per second). Red line represents total TPS. Blue line represents number of requests requests processed in less than 3 seconds. In short: SYNC scenario: run with distributed cahce was able to reach 23000 TPS compared to ~4000 TPS in run with invalidation cache. ASYNC scenario: run with distributed cahce was able to reach 31000 TPS compared to ~12000 TPS in run with invalidation cache.
"We can see a performance regression in our stress tests, which test performance of the cluster under increasing load (number of concurrent clients). The regression can be seen when using an invalidation cache + a shared cache store compared to using distributed cache without any cache store." This is not a regression. This is the 1st time performance testing has been done on this configuration and there is no reason to expect comparable performance with our default configuration. In fact, we expected performance of this configuration in EAP6 to be much worse than the default configuration.