Bug 1235332 - JDG Performance degradation with replicated cache
Summary: JDG Performance degradation with replicated cache
Keywords:
Status: POST
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Performance
Version: 6.4.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Nobody
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks: 1272389 1272390
TreeView+ depends on / blocked
 
Reported: 2015-06-24 14:26 UTC by Giuseppe Bonocore
Modified: 2023-09-30 08:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1272389 1272390 (view as bug list)
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)
Java client (43.41 KB, application/octet-stream)
2015-08-03 09:07 UTC, Giuseppe Bonocore
no flags Details
Oracle JDK Java Flight Recorder dump (567.95 KB, application/x-gzip)
2015-09-03 14:59 UTC, Giuseppe Bonocore
no flags Details
New flight recorder dump (600.50 KB, application/x-gzip)
2015-09-30 14:19 UTC, Giuseppe Bonocore
no flags Details
Flight Recorder dump mixed scenario (192.87 KB, application/x-gzip)
2015-09-30 14:35 UTC, Giuseppe Bonocore
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-5857 0 Critical Resolved HotRod client should use the consistent hash in replicated mode 2016-11-30 16:08:17 UTC
Red Hat Issue Tracker ISPN-5858 0 Major Resolved ValueMatcher and AvailabilityMode should have externalizers 2016-11-30 16:08:17 UTC
Red Hat Issue Tracker ISPN-5859 0 Major Resolved HotRod server should not use privileged actions for common stuff 2016-11-30 16:08:18 UTC
Red Hat Issue Tracker ISPN-5860 0 Major Resolved ComponentRegistry should cache the VersionGenerator in a field 2016-11-30 16:08:17 UTC

Description Giuseppe Bonocore 2015-06-24 14:26:56 UTC
Description of problem:

Put operation (using hotrod protocol) is on average 30% slower on JDG 6.4.1 compared to JDG 6.1.0

Version-Release number of selected component (if applicable):


How reproducible:

Create a 2 nodes JDG cluster with a replicated cache, on JDG 6.4.1 and 6.1.0

Create a simple hotrod client (in my case was the java version), that does a put of random values

Compare the average results


Steps to Reproduce:
1. Create a simple 2 nodes cluster on JDG 6.1.0 and JDG 6.4.1 with a replicated cache configured like:

...
<replicated-cache name="default" mode="SYNC" remote-timeout="30000" start="EAGER">
  <locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>
  <transaction mode="NONE"/> 
</replicated-cache>
...

2. Create a simple hotrod client. In my case, a java one, similar to:

         ConfigurationBuilder builder = new ConfigurationBuilder();
        builder.addServers(jdgProperty(JDG_SERVERS));
        cacheManager = new RemoteCacheManager(builder.build());
        cache = cacheManager.getCache();
        for(Long i =1L; i< (numEntry+1);i++ ){
    		Long startCycle =System.nanoTime();
		    cache.put(UUID.randomUUID().toString(), UUID.randomUUID().toString());
  	        Long elapsed = System.nanoTime() - startCycle;
}



3. Execute the test on 6.1.0 and 6.4.1 (I did it on 1000000 put operations), and compare the results between the different JDG versions

Actual results:

6.4.1 is 30% slower on average

Expected results:

at least same performance level


Additional info:

Comment 2 Giuseppe Bonocore 2015-08-03 09:07:00 UTC
Created attachment 1058675 [details]
Java client

Adding 2 sample java clients, as there are some small differences between the client libraries in the 2 JDG versions. The clients are heavily based on the JDG quickstarts

Comment 3 Giuseppe Bonocore 2015-09-03 14:59:31 UTC
Created attachment 1069916 [details]
Oracle JDK Java Flight Recorder dump

Attached an Oracle JDK Java Flight Recorder dump during a test. Test scenario: 

- 2 nodes cluster
- test running on JDG 6.1 and then on JDG 6.4. for comparison
- 10000 random UUID inserted x 10 times 


dump61_1 : dump of node 1 of JDG 6.1 cluster
dump61_2 : dump of node 2 of JDG 6.1 cluster

dump641_1 : dump of node 1 of JDG 6.4.1 cluster
dump641_2 : dump of node 2 of JDG 6.4.1 cluster

Test result: (min time and average in nanos for each put):

6.1:
48793	127222
33405	64405
26649	47684
26393	44935
26414	41612
27316	46360
26223	44363
26642	42076
26263	46148
26347	41687

6.4.1:
50456	137954
41507	81263
30952	58237
26699	51145
30142	48672
29518	50272
28450	45365
30065	49844
29805	49098
33054	49768


Difference:
1663	10732
8102	16858
4303	10553
306	6210
3728	7060
2202	3912
2227	1002
3423	7768
3542	2950
6707	8081

Comment 7 Dan Berindei 2015-09-29 07:18:54 UTC
I looked at the attached flight recorder dumps, but the number of samples is too small to draw any conclusions (~ 20 samples in each 1-minute recording).

Giuseppe, could you repeat the experiment with a lower sampling interval, e.g. by using the profiling template (-XX:FlightRecorderOptions=settings=profile)?

Comment 10 Giuseppe Bonocore 2015-09-30 14:19:38 UTC
Created attachment 1078669 [details]
New flight recorder dump

Comment 11 Giuseppe Bonocore 2015-09-30 14:22:15 UTC
I've added a new flight recorder dump, as per comment 7 suggestion (using -XX:FlightRecorderOptions=settings=profile).

Here the the test results of this testrun: (min time and average in nanos for each put):

6.1:		
	46938	133381
	29934	63220
	24596	43005
	25305	42890
	24579	44986
	24515	43307
	24013	41911
	24572	43727
	24846	45490
	24978	42455
		
6.4.1:		
	47823	142502
	40703	82950
	30686	58270
	27991	50919
	29619	51945
	34832	55043
	27718	50488
	28883	45690
	28321	45638
	28870	45570

Difference:

885	9121
10769	19730
6090	15265
2686	8029
5040	6959
10317	11736
3705	8577
4311	1963
3475	148
3892	3115

Comment 12 Giuseppe Bonocore 2015-09-30 14:35:40 UTC
Created attachment 1078675 [details]
Flight Recorder dump mixed scenario

Comment 13 Giuseppe Bonocore 2015-09-30 14:37:21 UTC
Just added a flight recorder dump in a mixed scenario (6.4.1 server. 6.1.0 client), as per Wolf request.

Here the test results:


Mix		
	49325	137189
	35183	74782
	27060	53027
	26318	45940
	27527	47494
	27953	41728
	25098	41966
	27181	45743
	26889	49537
	29363	44990


at a first glance it seems a bit better than a pure 6.4.1 setup, but still worse than a pure 6.1 setup

Comment 25 Mike McCune 2016-03-28 22:26:18 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions


Note You need to log in before you can comment on or make changes to this bug.