1. A requests segment from B (there are multiple chunks) 2. B sends all chunks, but before A receives them, new topology arrives and A cancels the ST. 3. Another topology comes and A requests this segment again 4. A receives the old StateResponseCommand with lastChunk=true and thinks that it got all segments, therefore, it discards further chunks. Result is inconsistent cluster, and after further rebalances completely lost data. This ought to be rare, but was repeatedly observed when gracefully stopping coordinator on a 32-node cluster full of data.
I didn't follow on recent updates on ST, but we discussed lots of time in the past that an in-flight ST should never be cancelled when started, it should "move forward" the the next view if it changes again. Did the design change?
Yes, Sanne, there are two cases where ST is cancelled: when there is a merge, or when the coordinator leaves the cluster. Cancelling ST in the second case is not strictly necessary, it's done because it's a bit difficult to distinguish between the two. I've created https://issues.jboss.org/browse/ISPN-4404 to work on this.