Bug 1160635

Summary: Increasing default internal thread pool size
Product: [JBoss] JBoss Data Grid 6 Reporter: Dan Berindei <dberinde>
Component: InfinispanAssignee: Dan Berindei <dberinde>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.0CC: jdg-bugs, mcimbora, plohia, slaskawi
Target Milestone: ER3   
Target Release: 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-28 13:26:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Berindei 2014-11-05 09:49:30 UTC 
Using synchronous replication with high number of concurrent clients doing put() operations over a shared set of keys, lock-acquisition timeouts occur when various thread pools (internal, jgroups oob) do not have appropriate size.

org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [3 seconds] on key [key_00000000000003B4] for requestor [Thread[OOB-66,default,node03-12795,5,main]]! Lock held by [Thread[OOB-314,default,node03-12795,5,main]]
[org.infinispan.interceptors.InvocationContextInterceptor] (Stressor-1) ISPN000136: Execution error
org.infinispan.util.concurrent.TimeoutException: org.infinispan.util.concurrent.TimeoutException: Node node04-24454 timed out

This applies to both transactional and non-transactional configuration. The problem can be mitigated by increasing Infinispan's internal thread pool size (defined for remoteCommandsExecutor, blockingBoundedQueueThreadPool). In order to improve user experience either:

a) When needed, the size of the thread pool should be increased as the load increases
b) The default values should be high enough to handle even significant load (in terms of number of concurrent clients per node)
c) The documentation should describe how the end user should size the thread pools based on expected load on the system
Comment 2 Sebastian Łaskawiec 2014-11-05 20:02:56 UTC 
PR: https://github.com/infinispan/jdg/pull/328