Bug 1212505

Summary: Persistence with stringKeyedJdbcStore throwing ConcurrentModificationException while adding data to cache
Product: [JBoss] JBoss Data Grid 6 Reporter: Shay Matasaro <smatasar>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: CLOSED UPSTREAM QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: high    
Version: 6.4.0CC: jdg-bugs, pzapataf
Target Milestone: ER1   
Target Release: 6.5.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-02-10 03:44:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shay Matasaro 2015-04-16 14:20:34 UTC
A  ConcurrentModificationException in the CacheWriter interceptor during a commit.

java.util.ConcurrentModificationException
    at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953)
    at java.util.LinkedList$ListItr.next(LinkedList.java:886)
    at org.infinispan.interceptors.CacheWriterInterceptor.store(CacheWriterInterceptor.java:210)
    at org.infinispan.interceptors.CacheWriterInterceptor.commitCommand(CacheWriterInterceptor.java:115)

The code is looping through the modifications associated with the transaction:
     List<WriteCommand> modifications = ctx.getCacheTransaction().getAllModifications();
     ...
210: for (WriteCommand cacheCommand : modifications) {

The transaction's modification list is stored as a synchronized list with a comment "we need to synchronize this collection to be able to get a valid snapshot from another thread during state transfer".

But this thread is not doing state transfer, and I'd assume "get a valid snapshot" wouldn't include modification?

Is it valid for this list to be modified from another thread (in which case all iterations should be synchronized), or is something modifying it incorrectly?

Comment 1 Shay Matasaro 2015-04-16 14:22:28 UTC
The stack trace includes LinkedList instead of SynchronizedList, so this is very likely a RemoteTransaction initialized from LockControlCommand.perform().

However, every code path that touches the modifications list in a RemoteTransaction seems to create a new list without modifying the old one, so I'm not sure how the CME could be triggered. I also ran the core test suite with a custom class replacing the LinkedList, and none of the mutating methods was ever called.

Can you post the cache configuration and perhaps some logs?

Cheers
Dan

Oops, you were right. LocalTransaction.addModification() also creates a LinkedList. It wraps it in a SynchronizedList, but SynchronizedList.iterator() doesn't wrap the iterator, so it doesn't show up in the stack trace.

That doesn't change the situation however, I don't see any way the modifications list could be modified during the commit. At this time, the only thing I can think of is to "freeze" the modifications list at the beginning of the commit and throw an exception if anyone tries to modify it afterwards, then give the customer the code and wait for them to give us back a stack trace of the modifying thread.

We could also synchronize the iteration over the modifications list. But adding a modification during commit seems like a bug anyway, and the added synchronization would just push the problem somewhere else.

In theory we could also get a CME if the collection was modified on one thread and then iterated over in a different thread, because TransactionSynchronizationInterceptor is not in the interceptor chain to ensure visibility. But CMTTxInterceptor.invokeInOurTx executes the entire transaction in a single thread, so that's clearly not the case here.

Cheers
Dan

Comment 3 Dan Berindei 2015-08-06 08:46:49 UTC
Pull request: https://github.com/infinispan/jdg/pull/705

Comment 4 JBoss JIRA Server 2015-08-06 14:35:55 UTC
William Burns <wburns> updated the status of jira ISPN-5379 to Resolved

Comment 8 Martin Gencur 2015-09-11 06:08:56 UTC
We're not able to reproduce this issue. Closing

Comment 11 Red Hat Bugzilla 2025-02-10 03:44:26 UTC
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.