Bug 1312186

Summary:	[GSS](JDG 6.x) Non-threadsafe use of HashSet in AdvancedAsyncCacheLoader
Product:	[JBoss] JBoss Data Grid 6	Reporter:	dereed
Component:	Infinispan	Assignee:	Tristan Tarrant <ttarrant>
Status:	VERIFIED ---	QA Contact:	Martin Gencur <mgencur>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	6.3.2	CC:	jdg-bugs, ssiddiqu, vjuranek, wfink
Target Milestone:	ER1
Target Release:	6.6.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1309749, 1314934

Description dereed 2016-02-26 03:06:26 UTC

org.infinispan.persistence.async.AdvancedAsyncCacheLoader$process creates a HashSet, and passes it to loadAllKeys().

loadAllKeys() creates a task to get each key and add it to the HashSet.
This task is run by org.infinispan.persistence.file.SingleFileStore#process, which runs it in multiple threads at once (one thread per key).

There is no synchronization on that HashSet that is shared by the multiple threads.

HashSet is not thread safe.  One known side effect of non-synchronized access by multiple threads is infinite loops, which has been witnessed here.

Comment 2 dereed 2016-02-26 03:16:24 UTC

Possible fixes:
- use Collections.synchronizedSet or another other thread-safe structure
- synchronize on the set around the only multi-threaded access to it in loadAllKeys$CacheLoaderTask#processEntry

Comment 3 Sebastian Łaskawiec 2016-02-26 14:03:10 UTC

Could you please tell me what is the configuration in this case (I'm especially interested which CacheLoader is used underneath) ?

Comment 4 dereed 2016-02-26 20:38:39 UTC

It's using the <singleFile> cache store.
The issue was seen during a cache.entrySet() call.

Comment 6 dereed 2016-02-26 20:46:12 UTC

Also relevant -- the loader is configured for async.

<singleFile ...>
    <async enabled="true" threadPoolSize="5" .../>
</singleFile>

Comment 7 Shay Matasaro 2016-02-29 20:06:24 UTC

The issue occurs with the local file store , but other stores should be examined for this possible race condition.

As the  # of entries in the cache increases, the odds of hitting it again increase also.

Comment 8 JBoss JIRA Server 2016-03-01 13:27:46 UTC

Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-6276 to Closed

Comment 13 Mike McCune 2016-03-28 23:35:19 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions