Bug 810155

Summary: Default configuration may not be optimal for a 2-node cluster
Product: [JBoss] JBoss Data Grid 6 Reporter: Michal Linhard <mlinhard>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: VERIFIED --- QA Contact: Nobody <nobody>
Severity: low Docs Contact:
Priority: low    
Version: 6.0.0CC: jdg-bugs, mgencur, nobody, pjha, ttarrant
Target Milestone: ER1   
Target Release: 6.0.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
A sample configuration is now included in the JBoss Data Grid Server distribution called docs/examples/configs/standalone-two-nodes.xml, which provides better performance when using clusters formed by only two nodes. This configuration changes the default JGroups configuration to use the UNICAST protocol instead of the UNICAST2 and RSVP combination of protocols. For larger clusters the UNICAST2 and RSVP combination is recommended.
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
JGroups UNICAST UDP config snippet none

Description Michal Linhard 2012-04-05 09:53:08 UTC
In performance tables of issue:
https://bugzilla.redhat.com/show_bug.cgi?id=758759

we have worse results with ER6 config compared to ER4 config.

Comment 1 Michal Linhard 2012-04-05 09:54:24 UTC
I'll try to obtain improved JProfiler snapshots (based on sampling for this)

Comment 2 Michal Linhard 2012-04-05 10:03:11 UTC
Workaround: for size2 installations use JGroups config with UNICAST if it performs better.

Comment 3 Michal Linhard 2012-04-05 10:03:11 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Hot rod client stress tests of cluster of size 2 perform better with ER4 JGroups config (UNICAST) vs ER6 (UNICAST2+RSVP)

Comment 4 prabhat jha 2012-05-11 15:08:52 UTC
Changed target release to 6.0.0.GA.

Comment 5 prabhat jha 2012-05-29 17:13:43 UTC
Ondra, pls update the status once you run ER10 with ER4 jgroups config for 2 node cluster.

Comment 6 Ondrej Nevelik 2012-05-31 07:15:54 UTC
[1] is a throughput chart of a client stress test with Hotrod, dist, cluster size2 and jgroups subsystem config used in ER4. It can handle 25% more clients then when we use our common config although the max throughput is approximately the same.

[1] https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg-60-perf-client-stress-test-hotrod/279/artifact/report/throughput.png

Comment 7 prabhat jha 2012-05-31 20:11:33 UTC
Ondra, we would need to run this again with ER11. If this is still the case then your observation needs to be documented in tuning guide.

Comment 8 Misha H. Ali 2012-06-04 00:58:28 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Hot rod client stress tests of cluster of size 2 perform better with ER4 JGroups config (UNICAST) vs ER6 (UNICAST2+RSVP)+Hot rod client stress tests with a two node cluster results in better performance with the ER4 JGroups configuration (using UNICAST) than the ER6 JGroups configuration (using UNICAST2 and RSVP).

Comment 9 Ondrej Nevelik 2012-06-06 08:13:31 UTC
For ER11 the conclusion is that for cluster size 2 the performance is better with jgroups configuration used with ER4 while it doesn't improve size 4 perf.

Comment 10 Misha H. Ali 2012-06-07 03:31:15 UTC
Prabhat's very valid point about external parties not knowing what we refer to by ERX:

"ER releases are internal and customers wont understand what we mean by these. I suggest this to convey "Out of box configuration is not optimal for a 2-node cluster." Ideally this should mention code snippet with right configuration or point to some URL which  has the configuration."

Flagging Michal as he provided the original information for this bug.

Comment 11 Michal Linhard 2012-06-08 13:10:51 UTC
Created attachment 590427 [details]
JGroups UNICAST UDP config snippet

Comment 12 Michal Linhard 2012-06-08 13:16:10 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Hot rod client stress tests with a two node cluster results in better performance with the ER4 JGroups configuration (using UNICAST) than the ER6 JGroups configuration (using UNICAST2 and RSVP).+Out-of-the-box JGroups configuration (using UNICAST2 and RSVP) might not lead to optimal performance in 2-node cluster scenario. We've found that 2-node clusters perform better with UNICAST and without RSVP. See attached JGroups subsystem configuration snippet "JGroups UNICAST UDP config snippet".

Comment 13 Misha H. Ali 2012-06-12 05:14:05 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Out-of-the-box JGroups configuration (using UNICAST2 and RSVP) might not lead to optimal performance in 2-node cluster scenario. We've found that 2-node clusters perform better with UNICAST and without RSVP. See attached JGroups subsystem configuration snippet "JGroups UNICAST UDP config snippet".+The default JBoss Data Grid JGroups configuration (using UNICAST2 and RSVP) does not lead to optimal performance in a two node cluster scenario. Two node clusters show improved performance with UNICAST and without RSVP.

Comment 15 Michal Linhard 2012-06-14 07:47:45 UTC
In hyperion size2 tests the performance difference isn't that significant
http://www.qa.jboss.com/~mlinhard/hyperion/run190-client-stress-02-CR2-unicast-no-rsvp/compare/merged-throughput.png

The two runs differ only in configuration of JGroups stack in the two protocols (UNICAST vs UNICAST2+RSVP), every other setting I've synced.
http://www.qa.jboss.com/~mlinhard/hyperion/run190-client-stress-02-CR2-unicast-no-rsvp/compare/configs/standalone_unicast.xml
http://www.qa.jboss.com/~mlinhard/hyperion/run190-client-stress-02-CR2-unicast-no-rsvp/compare/configs/standalone_unicast2+rsvp.xml

Perf difference (UNICAST vs UNICAST2+RSVP)
max number of clients: 26000 vs 23500 (10.6 %)
max throughput: 132609.20 vs 129351.05 (2.5 %)

Comment 16 Martin Gencur 2012-07-30 06:50:35 UTC
The JDG 6.0.1 ER1 server distribution now contains a sample configuration file ($JDG_HOME/docs/examples/configs/standalone-two-nodes.xml) with so far the most optimal configuration for 2-node cluster. I'm closing this for now as verified but in the future we should find out if there's not even a better configuration in terms of performance.