Bug 1203121

The getBulk method doesn't declare such semantics.  In 6.3 it just returned all the entries or a subset in memory from the node it ran this on.  Interestingly you can receive different values based on which node it contacted each time it is invoked.

JDG 6.4/6.5 has the option of changing the behavior of the getBulk to return all entries in the cluster from a single invocation.  This internally does proper semantics regarding the consistent hash.  This is an optional feature though (not enabled by default) in JDG 6.4/6.5 and requires setting the JVM option "infinispan.accurate.bulk.ops".  Without this JVM property it will run the same as in JDG 6.3.

Also, even if the getBulk operates in a consistent hash friendly manner (like in JDG 6.5), the way that the user is "clearing" the cache will still not work.  This will leave data around still unless https://bugzilla.redhat.com/show_bug.cgi?id=1199210 is remedied.

> ... Interestingly you can receive different values based on which node it contacted each time it is invoked.

I forget to mention that the customer is using a custom FailoverRequestBalancingStrategy to target a specific server.  This way, repeating getBulk(size) works in their use case.


Thank you for the info about the change in new JDG.  By setting "infinispan.accurate.bulk.ops=true" and Flag.CACHE_MODE_LOCAL, it seems the same effect is achievable in JDG 6.4.0.  Are those setting valid for Hot Rod client as well?


> Also, even if the getBulk operates in a consistent hash friendly manner (like in JDG 6.5), the way that the user is "clearing" the cache will still not work.  This will leave data around still unless https://bugzilla.redhat.com/show_bug.cgi?id=1199210 is remedied.

I agree with the last statement that the root cause of leaving stale data is Bug 1199210.  But regarding the first statement, let me cofirm for sure.  If getBulk operates in a consistent hash friendly manner, by enabling infinispan.accurate.bulk.ops in new version of JDG, it retrieves only fresh data based on the current topology and ignores stale data.  So getBulk works correctly except the server side retains stale data.  Is this correct?  This should be acceptable to the customer for the moment.

(In reply to Osamu Nagano from comment #3)
> Thank you for the info about the change in new JDG.  By setting
> "infinispan.accurate.bulk.ops=true" and Flag.CACHE_MODE_LOCAL, it seems the
> same effect is achievable in JDG 6.4.0.  Are those setting valid for Hot Rod
> client as well?

The JVM property would be set on the HR server itself.  This is not set on the client.  There is no way to pass Flag.CACHE_MODE_LOCAL with HR client.

> I agree with the last statement that the root cause of leaving stale data is
> Bug 1199210.  But regarding the first statement, let me cofirm for sure.  If
> getBulk operates in a consistent hash friendly manner, by enabling
> infinispan.accurate.bulk.ops in new version of JDG, it retrieves only fresh
> data based on the current topology and ignores stale data.  So getBulk works
> correctly except the server side retains stale data.  Is this correct?  This
> should be acceptable to the customer for the moment.

The new JVM property guarantees by segment data yes, but it does this for all data in the cluster.  The entrySet method (what getBulk calls underneath) however ignores segments when Flag.CACHE_MODE_LOCAL is set just like the previous implementation and just returns all data in the local container/store.

There is no full proof way that I am aware of to only get keys local to a specific node that only are owned by that node.

You could install a listener on the remote cache to listen for topology changes and keep a reference to the consistent hash and figure out which keys retrieved locally are really mapped there or not.  However this method could be out of sync if there is a topology change that happens right after.

Thanks, William.  As in Bug 1199210, we expect that JDG 6.4.0 doesn't leave extra backups from the first and this getBulk issue will not happen.