Still an issue, moving to 6.4.

I am reproducing the issue regulary with dist cache, 3+ nodes. The failure pattern is always:

1) update a SFSB
2) a node is shut down
3) update the SFSB again
4) the node is restarted
5) the SFSB is updated on an old node
6) the SFSB is updated on a new node but the previous update is lost

Notes:
* the failure never happen with repl cache
* there are no session updates while a node is stopping/rejoining the cluster


Test log fragment:
10:22:52,783 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-8) Counter 5 incremented to 38 on node node-1
10:22:52,989 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-8) Counter 5 incremented to 39 on node node-1
10:22:53,197 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-8) Counter 5 incremented to 40 on node node-1
10:22:53,381 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (main) Starting container 0.
[0m10:23:01,106 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (main) Container 0 started.
10:23:06,229 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-64) Counter 5 incremented to 41 on node node-1
10:23:06,811 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-64) Counter 5 incremented to 41 on node node-0
10:23:07,486 ERROR [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-64) Unexpected count from remote counter 5. Expected: 42 received: 41 from node: node-0.
10:23:07,560 INFO  [org.jboss.as.test.clustering.cluster.ejb3.stateful.remote.failover.RemoteEJBClientStatefulBeanFailoverStressTestCase] (pool-1-thread-64) Counter 5 incremented to 43 on node node-0


Testcase:
https://github.com/dpospisil/jboss-eap/blob/10192e7ef39c2bd1acea438a3073094b7ebc1830/testsuite/integration/clust/src/test/java/org/jboss/as/test/clustering/cluster/ejb3/stateful/remote/failover/RemoteEJBClientStatefulBeanFailoverStressTestCase.java

Seems that I have finally found the root cause of the issue. 

org.jboss.as.ejb3.cache.spi.BackingCache seems to rely on the fact that the underlying cache.get() returns the same object when called multiple times. Even BackingCache.release(K key) first has to a get first as it takes the key only.

However in the case of a distributed infinispan cache used as a store via InfinispanBackingCacheEntryStore when it is called on a node which does not hold specific data block this is not the case. The ISPN does remote get and recieves fresh data from the remote node. In suche case, BackingCache.release() fails as it first catch stale data from the cache and writes them back.

@dpospisi I don't think your assumption is correct.  In order to support REPEATABLE_READ semantics within a batch, Cache.get() *must* retain a reference to the data that was initially fetched remotely.  However, maybe the issue is with the transaction boundaries being in the wrong place, i.e. the initial Cache.get() must reside within the context of the same batch used in release().

After refreshing my memory with the old implementation, I recall that the initial cache.get() and the subsequent get()/put() in release() happen in separate batches.  These should really happen within the same batch - which is how the new implementation (in EAP7) works.
To workaround this in EAP6, it should be sufficient to enable the L1 cache (by setting a non-zero l1-lifespan).

Yes, that was what I tried as well. However, there seem to be another issue on top of this one which prevents enabling L1 cache. Setting l1-lifespan attribute to a non-zero value just set's the underlying cache config parameter, but does not explicitly enable l1 which is by default disabled. I'll doublecheck and create separate BZ in such case. Quick fix shows that enabling L1 does indeed help and could be used as a workaround for this issue.

I have created BZ1218148. If there are no objections, I will close this one with WONTFIX with known workaround once it is resolved.

pm_ack-
Triage 2017-Jan-30 -> close due to lack of support case.