Description of problem: Put operation (using hotrod protocol) is on average 30% slower on JDG 6.4.1 compared to JDG 6.1.0 Version-Release number of selected component (if applicable): How reproducible: Create a 2 nodes JDG cluster with a replicated cache, on JDG 6.4.1 and 6.1.0 Create a simple hotrod client (in my case was the java version), that does a put of random values Compare the average results Steps to Reproduce: 1. Create a simple 2 nodes cluster on JDG 6.1.0 and JDG 6.4.1 with a replicated cache configured like: ... <replicated-cache name="default" mode="SYNC" remote-timeout="30000" start="EAGER"> <locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/> <transaction mode="NONE"/> </replicated-cache> ... 2. Create a simple hotrod client. In my case, a java one, similar to: ConfigurationBuilder builder = new ConfigurationBuilder(); builder.addServers(jdgProperty(JDG_SERVERS)); cacheManager = new RemoteCacheManager(builder.build()); cache = cacheManager.getCache(); for(Long i =1L; i< (numEntry+1);i++ ){ Long startCycle =System.nanoTime(); cache.put(UUID.randomUUID().toString(), UUID.randomUUID().toString()); Long elapsed = System.nanoTime() - startCycle; } 3. Execute the test on 6.1.0 and 6.4.1 (I did it on 1000000 put operations), and compare the results between the different JDG versions Actual results: 6.4.1 is 30% slower on average Expected results: at least same performance level Additional info:
Created attachment 1058675 [details] Java client Adding 2 sample java clients, as there are some small differences between the client libraries in the 2 JDG versions. The clients are heavily based on the JDG quickstarts
Created attachment 1069916 [details] Oracle JDK Java Flight Recorder dump Attached an Oracle JDK Java Flight Recorder dump during a test. Test scenario: - 2 nodes cluster - test running on JDG 6.1 and then on JDG 6.4. for comparison - 10000 random UUID inserted x 10 times dump61_1 : dump of node 1 of JDG 6.1 cluster dump61_2 : dump of node 2 of JDG 6.1 cluster dump641_1 : dump of node 1 of JDG 6.4.1 cluster dump641_2 : dump of node 2 of JDG 6.4.1 cluster Test result: (min time and average in nanos for each put): 6.1: 48793 127222 33405 64405 26649 47684 26393 44935 26414 41612 27316 46360 26223 44363 26642 42076 26263 46148 26347 41687 6.4.1: 50456 137954 41507 81263 30952 58237 26699 51145 30142 48672 29518 50272 28450 45365 30065 49844 29805 49098 33054 49768 Difference: 1663 10732 8102 16858 4303 10553 306 6210 3728 7060 2202 3912 2227 1002 3423 7768 3542 2950 6707 8081
I looked at the attached flight recorder dumps, but the number of samples is too small to draw any conclusions (~ 20 samples in each 1-minute recording). Giuseppe, could you repeat the experiment with a lower sampling interval, e.g. by using the profiling template (-XX:FlightRecorderOptions=settings=profile)?
Created attachment 1078669 [details] New flight recorder dump
I've added a new flight recorder dump, as per comment 7 suggestion (using -XX:FlightRecorderOptions=settings=profile). Here the the test results of this testrun: (min time and average in nanos for each put): 6.1: 46938 133381 29934 63220 24596 43005 25305 42890 24579 44986 24515 43307 24013 41911 24572 43727 24846 45490 24978 42455 6.4.1: 47823 142502 40703 82950 30686 58270 27991 50919 29619 51945 34832 55043 27718 50488 28883 45690 28321 45638 28870 45570 Difference: 885 9121 10769 19730 6090 15265 2686 8029 5040 6959 10317 11736 3705 8577 4311 1963 3475 148 3892 3115
Created attachment 1078675 [details] Flight Recorder dump mixed scenario
Just added a flight recorder dump in a mixed scenario (6.4.1 server. 6.1.0 client), as per Wolf request. Here the test results: Mix 49325 137189 35183 74782 27060 53027 26318 45940 27527 47494 27953 41728 25098 41966 27181 45743 26889 49537 29363 44990 at a first glance it seems a bit better than a pure 6.4.1 setup, but still worse than a pure 6.1 setup
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions