Bug 1108451

Summary:	Takes a long time to start up all clusters which has a lot of segments
Product:	[JBoss] JBoss Data Grid 6	Reporter:	ksuzumur
Component:	Infinispan	Assignee:	Tristan Tarrant <ttarrant>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Martin Gencur <mgencur>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.2.1	CC:	anistor, dereed, rvansa
Target Milestone:	ER7
Target Release:	6.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-01-26 14:05:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 2 Adrian Nistor 2014-06-13 11:38:49 UTC

I've made a simple test with one DIST cache, 10 nodes, no data at all.

Initial state transfer completes in 3 seconds when numSegments == 500 and when numSegments == 5000 it takes more than 50 seconds.

Item #2 seems the have most impact. By switching to a simple HashSet in InboundTransferTask the time drops significatly, about 5 times. Unfortiunatelly the trivial solution of just replacing CopyOnWriteArraySet with HashSet breaks some concurrency related concerns so it's not immediatelly applicable. Instead I will try to see if using syncronization can get us better performance than usign concurrent collections.

Item #1 does not seem to give us much impromvement, but it is indeed a potential optimization. Should be solved by extracting the wCh.getSegmentsForOwner(..) call outside the loop.

I'm still working one #2, not sure if I can have a quick solution today.

Comment 3 Adrian Nistor 2014-06-13 16:13:18 UTC

PR here: https://github.com/infinispan/jdg/pull/123

Comment 4 Radim Vansa 2014-06-18 15:23:28 UTC

I believe that the functional perspective of this PR can be verified within the usual elasticity/resilience tests (in fact, the changes seem so simple that Infinispan testsuite should catch any bug as well).

For performance check, I could set up the cluster with many segments and see whether the startup time has improved (from logs - I don't think any automatization would be beneficial).

Comment 5 Martin Gencur 2014-06-19 06:53:20 UTC

Please run the usual elasticity/resilience tests. Thanks.

Comment 6 Martin Gencur 2014-06-20 16:03:43 UTC

The fix has been verified. It does NOT introduce any regression. We did not measure the speed up, though.

Comment 7 Adrian Nistor 2014-06-27 13:32:03 UTC

After additional reviewing in community we found some small additional, rather cosmetic, fixes: https://github.com/infinispan/jdg/pull/146