Bug 1006998 - Lost version info during state transfer causes overwtite of newer data that joining node has read from store
Lost version info during state transfer causes overwtite of newer data that j...
Status: CLOSED WONTFIX
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 6.2.0
Assigned To: Tristan Tarrant
Martin Gencur
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-11 13:00 EDT by Randy
Modified: 2013-09-23 09:27 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 09:27:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Sample app used to create the error (6.61 KB, application/zip)
2013-09-11 13:00 EDT, Randy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-3431 Critical Closed Lost version info during state transfer causes overwrite of newer data that joining node has read from store 2013-12-02 21:32:54 EST

  None (edit)
Description Randy 2013-09-11 13:00:20 EDT
Created attachment 796452 [details]
Sample app used to create the error

The problem starts with three remote nodes running in replicated mode. Each node is using a local cache store from which data is loaded upon startup.

1 Start node1, node2, node3
2 Put A⇒A1 from node1
3 Put B⇒B1 from node2
4 Put C⇒C1 from node3
5 Gracefully shutdown node1 saving the data
6 Gracefully shutdown node2 saving the data
7 Put C⇒C2 from node3
8 Gracefully shutdown node3 saving the data
9 Start node1 loading the data
10 Start node2 loading the data
11 Start node3 loading the data

12 Loaded from store on node3 - version information shows its newer than nodes 1 and 2 have: org.infinispan.interceptors.CallInterceptor Executing command: PutKeyValueCommandC, value=C2, flags=[CACHE_MODE_LOCAL, SKIP_LOCKING, SKIP_CACHE_STORE, SKIP_INDEXING, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES], putIfAbsent=false, metadata=EmbeddedMetadataSimpleClusteredVersion{topologyId=8, version=2}, successful=true, ignorePreviousValue=false.

13 During state transfer older value C1 is received with version information being null: org.infinispan.statetransfer.StateTransferInterceptor handleNonTxWriteCommand for command PutKeyValueCommandC, value=C1, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, metadata=EmbeddedMetadatanull, successful=true, ignorePreviousValue=false

14 As a result newer value is replaced with older: org.infinispan.interceptors.EntryWrappingInterceptor About to commit entry ClusteredRepeatableReadEntry(323f265b)C, value=C1, oldValue=C2, isCreated=false, isChanged=true, isRemoved=false, isValid=true, skipRemoteGet=true, metadata=EmbeddedMetadataSimpleClusteredVersion{topologyId=8, version=2}

Actual results
The old value for entry C replaces the newer value.

Expected results
The new value of C should be retained on node3 and overwrite the old value of C on nodes 1 and 2.

How to Reproduced with the Attached archive
Please find attached archive with the Maven project that contains code implementing the scenario and bundling all configuration files for Infinispan and JGroups.

To run the scenario:
1 Start node1: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Dinfinispan.store_name=A -jar infinispan-proto-5-1.0-SNAPSHOT.jar A

2 Start node2 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7801 -Dinfinispan.store_name=B -jar infinispan-proto-5-1.0-SNAPSHOT.jar B

3 Start node3 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7802 -Dinfinispan.store_name=C -jar infinispan-proto-5-1.0-SNAPSHOT.jar C

4 At this time all nodes will put their values to map and exit persisting data in standard Infinispan’s file store. Before node3 exists, it waits for node1 and node2 to leave and then changes its value C one again before persisting data and exiting.

5 Start node1: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Dinfinispan.store_name=A -jar infinispan-proto-5-1.0-SNAPSHOT.jar A

6 Start node2 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7801 -Dinfinispan.store_name=B -jar infinispan-proto-5-1.0-SNAPSHOT.jar B

7 Start node3 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7802 -Dinfinispan.store_name=C -jar infinispan-proto-5-1.0-SNAPSHOT.jar C

8 At this time newer version of configuration written by node3 gets reset to older value and last configuration update is lost. This can be seen by “C ⇒ C1 ←--------- Should be C2” in console for node3 (well, as in other consoles too) (Delete file store content to re-run scenario again.)
Comment 2 JBoss JIRA Server 2013-09-20 13:21:06 EDT
Mircea Markus <mmarkus@redhat.com> updated the status of jira ISPN-3431 to Closed
Comment 3 JBoss JIRA Server 2013-09-20 13:21:06 EDT
Mircea Markus <mmarkus@redhat.com> made a comment on jira ISPN-3431

This is not a bug, we don't support preloading from local cache stores ATM. 
The feature you're after is described here: https://community.jboss.org/wiki/ControlledClusterShutdownWithDataRestoreFromPersistentStorage 
And the corresponding JIRA is ISPN-3351.

Note You need to log in before you can comment on or make changes to this bug.