Bug 1235332
| Summary: | JDG Performance degradation with replicated cache | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Data Grid 6 | Reporter: | Giuseppe Bonocore <gbonocor> | ||||||||||
| Component: | Performance | Assignee: | Nobody <nobody> | ||||||||||
| Status: | CLOSED UPSTREAM | QA Contact: | Martin Gencur <mgencur> | ||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | 6.4.1 | CC: | afield, mgencur, pzapataf, ttarrant, wfink | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | |||||||||||||
| : | 1272389 1272390 (view as bug list) | Environment: | |||||||||||
| Last Closed: | 2025-02-10 03:47:46 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 1272389, 1272390 | ||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 1058675 [details]
Java client
Adding 2 sample java clients, as there are some small differences between the client libraries in the 2 JDG versions. The clients are heavily based on the JDG quickstarts
Created attachment 1069916 [details]
Oracle JDK Java Flight Recorder dump
Attached an Oracle JDK Java Flight Recorder dump during a test. Test scenario:
- 2 nodes cluster
- test running on JDG 6.1 and then on JDG 6.4. for comparison
- 10000 random UUID inserted x 10 times
dump61_1 : dump of node 1 of JDG 6.1 cluster
dump61_2 : dump of node 2 of JDG 6.1 cluster
dump641_1 : dump of node 1 of JDG 6.4.1 cluster
dump641_2 : dump of node 2 of JDG 6.4.1 cluster
Test result: (min time and average in nanos for each put):
6.1:
48793 127222
33405 64405
26649 47684
26393 44935
26414 41612
27316 46360
26223 44363
26642 42076
26263 46148
26347 41687
6.4.1:
50456 137954
41507 81263
30952 58237
26699 51145
30142 48672
29518 50272
28450 45365
30065 49844
29805 49098
33054 49768
Difference:
1663 10732
8102 16858
4303 10553
306 6210
3728 7060
2202 3912
2227 1002
3423 7768
3542 2950
6707 8081
I looked at the attached flight recorder dumps, but the number of samples is too small to draw any conclusions (~ 20 samples in each 1-minute recording). Giuseppe, could you repeat the experiment with a lower sampling interval, e.g. by using the profiling template (-XX:FlightRecorderOptions=settings=profile)? Created attachment 1078669 [details]
New flight recorder dump
I've added a new flight recorder dump, as per comment 7 suggestion (using -XX:FlightRecorderOptions=settings=profile). Here the the test results of this testrun: (min time and average in nanos for each put): 6.1: 46938 133381 29934 63220 24596 43005 25305 42890 24579 44986 24515 43307 24013 41911 24572 43727 24846 45490 24978 42455 6.4.1: 47823 142502 40703 82950 30686 58270 27991 50919 29619 51945 34832 55043 27718 50488 28883 45690 28321 45638 28870 45570 Difference: 885 9121 10769 19730 6090 15265 2686 8029 5040 6959 10317 11736 3705 8577 4311 1963 3475 148 3892 3115 Created attachment 1078675 [details]
Flight Recorder dump mixed scenario
Just added a flight recorder dump in a mixed scenario (6.4.1 server. 6.1.0 client), as per Wolf request. Here the test results: Mix 49325 137189 35183 74782 27060 53027 26318 45940 27527 47494 27953 41728 25098 41966 27181 45743 26889 49537 29363 44990 at a first glance it seems a bit better than a pure 6.4.1 setup, but still worse than a pure 6.1 setup This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions This product has been discontinued or is no longer tracked in Red Hat Bugzilla. |
Description of problem: Put operation (using hotrod protocol) is on average 30% slower on JDG 6.4.1 compared to JDG 6.1.0 Version-Release number of selected component (if applicable): How reproducible: Create a 2 nodes JDG cluster with a replicated cache, on JDG 6.4.1 and 6.1.0 Create a simple hotrod client (in my case was the java version), that does a put of random values Compare the average results Steps to Reproduce: 1. Create a simple 2 nodes cluster on JDG 6.1.0 and JDG 6.4.1 with a replicated cache configured like: ... <replicated-cache name="default" mode="SYNC" remote-timeout="30000" start="EAGER"> <locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/> <transaction mode="NONE"/> </replicated-cache> ... 2. Create a simple hotrod client. In my case, a java one, similar to: ConfigurationBuilder builder = new ConfigurationBuilder(); builder.addServers(jdgProperty(JDG_SERVERS)); cacheManager = new RemoteCacheManager(builder.build()); cache = cacheManager.getCache(); for(Long i =1L; i< (numEntry+1);i++ ){ Long startCycle =System.nanoTime(); cache.put(UUID.randomUUID().toString(), UUID.randomUUID().toString()); Long elapsed = System.nanoTime() - startCycle; } 3. Execute the test on 6.1.0 and 6.4.1 (I did it on 1000000 put operations), and compare the results between the different JDG versions Actual results: 6.4.1 is 30% slower on average Expected results: at least same performance level Additional info: