Red Hat Bugzilla – Bug 1001634
Write command may be ignored during state transfer
Last modified: 2014-04-28 11:39:17 EDT
Distributed sync non-tx cache. Situation: 1) A node is joining the cluster, requesting some segment 2) RemoveCommand is sent to backup owner with ignorePreviousValue=true 3) It looks up the entry and finds null 4) State transfer invokes the PutKeyValueCommand and sets the value for removed entry (updateKeys has not the key yet) 5) RemoveCommand adds its key to updateKeys set, but it does not remove the value as it is already null (in its context) Result: the value is removed on primary but on backup this is still present
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443 Confirmed also for transactional cache.
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443 The issue may happen for PutKeyValueCommand as well - the command commits the entry, then the ST can commit the entry and only after the command records this key into the updateKeys set.
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-3443 to Reopened
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443 I still have ignored commands - see comment I've posted 28/Aug/13 12:55 PM. Experienced with ReplaceCommand on backup owner in non-tx cache. I believe the cause is race condition between checking the update keys in EntryWrappingInterceptor.commitEntryIfNeeded and actually committing it - if the ReplaceCommand is executed just between the check and committing the value, the value is overwritten by ST.
Dan Berindei <dberinde@redhat.com> made a comment on jira ISPN-3443 Radim, I missed your comment about the race condition in EntryWrappingInterceptor.commitEntryIfNeeded. I don't think it's a case of the WriteCommand being ignored, though: the command is committed, it's just that state transfer than overwrites the value.
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443 You're right - should I rename this JIRA back to RemoveCommand may be ignored during state transfer and create different JIRA for overwriting by ST?
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443 [~dan.berindei]: Regrettably, the fix is not correct. You have to atomically check and ADD the key to updated keys set. Otherwise, a situation when both transaction and state transfer commit the entry (in this order) is possible, resulting with outdated entry on the node.
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-3443 to Resolved
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-3443 My apologies, old version got into the test.
*** Bug 1024918 has been marked as a duplicate of this bug. ***