Bug 1001634 - Write command may be ignored during state transfer
Write command may be ignored during state transfer
Status: VERIFIED
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high
: ER3
: 6.2.0
Assigned To: Tristan Tarrant
Martin Gencur
:
: 1024918 (view as bug list)
Depends On:
Blocks: 1017190 1010419
  Show dependency treegraph
 
Reported: 2013-08-27 08:34 EDT by Radim Vansa
Modified: 2014-04-28 11:39 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-3443 Blocker Resolved WriteCommand may be ignored during state transfer 2016-05-26 02:49 EDT

  None (edit)
Description Radim Vansa 2013-08-27 08:34:16 EDT
Distributed sync non-tx cache.
Situation:

1) A node is joining the cluster, requesting some segment
2) RemoveCommand is sent to backup owner with ignorePreviousValue=true
3) It looks up the entry and finds null
4) State transfer invokes the PutKeyValueCommand and sets the value for removed entry (updateKeys has not the key yet)
5) RemoveCommand adds its key to updateKeys set, but it does not remove the value as it is already null (in its context)

Result: the value is removed on primary but on backup this is still present
Comment 2 JBoss JIRA Server 2013-08-28 06:52:23 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443

Confirmed also for transactional cache.
Comment 3 JBoss JIRA Server 2013-08-28 06:55:06 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443

The issue may happen for PutKeyValueCommand as well - the command commits the entry, then the ST can commit the entry and only after the command records this key into the updateKeys set.
Comment 4 JBoss JIRA Server 2013-10-07 05:14:03 EDT
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-3443 to Reopened
Comment 5 JBoss JIRA Server 2013-10-07 05:14:03 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443

I still have ignored commands - see comment I've posted 28/Aug/13 12:55 PM. Experienced with ReplaceCommand on backup owner in non-tx cache.
I believe the cause is race condition between checking the update keys in EntryWrappingInterceptor.commitEntryIfNeeded and actually committing it - if the ReplaceCommand is executed just between the check and committing the value, the value is overwritten by ST.
Comment 6 JBoss JIRA Server 2013-10-09 04:52:54 EDT
Dan Berindei <dberinde@redhat.com> made a comment on jira ISPN-3443

Radim, I missed your comment about the race condition in EntryWrappingInterceptor.commitEntryIfNeeded. I don't think it's a case of the WriteCommand being ignored, though: the command is committed, it's just that state transfer than overwrites the value.
Comment 7 JBoss JIRA Server 2013-10-09 05:01:31 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443

You're right - should I rename this JIRA back to RemoveCommand may be ignored during state transfer and create different JIRA for overwriting by ST?
Comment 8 JBoss JIRA Server 2013-10-10 08:01:07 EDT
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-3443 to Reopened
Comment 9 JBoss JIRA Server 2013-10-10 08:01:07 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443

[~dan.berindei]: Regrettably, the fix is not correct. You have to atomically check and ADD the key to updated keys set. Otherwise, a situation when both transaction and state transfer commit the entry (in this order) is possible, resulting with outdated entry on the node.
Comment 10 JBoss JIRA Server 2013-10-11 10:28:14 EDT
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-3443 to Resolved
Comment 11 JBoss JIRA Server 2013-10-11 10:28:14 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443

My apologies, old version got into the test.
Comment 12 Radim Vansa 2013-12-10 05:10:15 EST
*** Bug 1024918 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.