Bug 1224770

Update:

The problem is not in Infinispan but in JGroups.

The JGroups's bundler puts address and site_address in the same batch. If the first message had a site_address, all the messages in the batch would be delivered with that address. When Infinispan sees a site_address, it thinks that it is from a remote site but that is not true and the IllegalStateException is thrown.

JGroups' JIRA: https://issues.jboss.org/browse/JGRP-1930

I also made a pull request with a fix (tested using the dockers images. thanks for that)

Thank you very much for your feedback. 

Don't know if relevant at this stage, but we noticed that apparently the problem does not occurs with Java Hotrod clients.

Sample Java Client Snippet:

...
        ConfigurationBuilder builder = new ConfigurationBuilder();
        builder.addServers( ... );
        cacheManager = new RemoteCacheManager(builder.build());
        cache = cacheManager.getCache();
       while (true) {
cache.put(UUID.randomUUID().toString(),UUID.randomUUID().toString());
	 }
...



Instead, it occurs systematically with simple C++ hotRod clients:

....
    infinispan::hotrod::ConfigurationBuilder builder;
    builder.addServers(address);
    infinispan::hotrod::RemoteCacheManager cacheManager(builder.build());
    infinispan::hotrod::RemoteCache<std::string, std::string>   cache(cacheManager.getCache<std::string, std::string>(false));
	boost::uuids::random_generator generator;
	while(true) {
    	std::string uuid_key = boost::lexical_cast<std::string>(generator());
	    std::string uuid_value = boost::lexical_cast<std::string>(generator());
	    cache.put(uuid_key, uuid_value);
	}
...

I'm not sure about the C++ client, but the Java client retries the operation in another server if it fails.

The issue only occurs if the operation is executed against node_2. I believe that the Java client will catch any exception from node_2 and retries in node_1.

Fix from https://issues.jboss.org/browse/JGRP-1930 incorporated into 3.6.3.Final-redhat-3 build (https://bugzilla.redhat.com/show_bug.cgi?id=1225028)

just FIY a short note about the reproducer - if you want to make sure the environment is reproducible (e.g. you include compiled C++ binary), please make sure you user version in your docker files, not just "FROM centos"

I updated reproducer for JDG 6.5.0.CR1, the stack trace is not logged any more and error is not propagated in the cluster or to client, but primary node still logs ERROR messages:

11:50:37,455 ERROR [org.jgroups.protocols.relay.RELAY2] (OOB-15,shared=udp) node-1/PRIMARY: no route to DR: dropping message
11:50:37,455 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-5) node-1/PRIMARY: no route to DR: dropping message
11:50:37,471 ERROR [org.jgroups.protocols.relay.RELAY2] (OOB-18,shared=udp) node-1/PRIMARY: no route to DR: dropping message
11:50:37,484 ERROR [org.jgroups.protocols.relay.RELAY2] (OOB-18,shared=udp) node-1/PRIMARY: no route to DR: dropping message
11:50:37,472 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-1) node-1/PRIMARY: no route to DR: dropping message
11:50:37,499 ERROR [org.jgroups.protocols.relay.RELAY2] (Incoming-5,shared=udp) node-1/PRIMARY: no route to DR: dropping message
11:50:37,502 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-2) node-1/PRIMARY: no route to DR: dropping message
11:50:37,509 ERROR [org.jgroups.protocols.relay.RELAY2] (Incoming-5,shared=udp) node-1/PRIMARY: no route to DR: dropping message

I can't do much about that errors in the logs. What you can do is turn off the logs for RELAY2. 

Since the backup is done asynchronously, Infinispan is never aware that the other site is down. So, it never stops trying to send data there.

Giuseppe, could you please comment on it, if you are fine with ERRORs in log? Otherwise I didn't find any other issue, so if you are ok with that, I'll mark it as verified.
Thanks

Hello Vojtech. If the error message is harmless, it's ok to us to keep it

Thanks Giuseppe, marking then as verified.

*** Bug 1231926 has been marked as a duplicate of this bug. ***