In Red Hat JBoss Data Grid, when roles based authentication was enabled, the Hot Rod client did not fail over; instead, it logged the exception when the JDG server to which the client was connected and went offline.
This issue is resolved as of Red Hat JBoss Data Grid 6.6.0. If the JDG server shuts down, then the Hot Rod client will fail over to another JDG server in the cluster silently, with no exception logged and the Hot Rod client will continue to work properly.
Description of problem:
A HotRod client fail over silent if one instance is shutting down if there is no Security enabled.
With role based authentication the client fail sporadically with a WARN message
WARN: ISPN004005: Error received from the server: java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
and the following Exception:
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[2825] returned server error (status=0x85): java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:57)
at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:36)
at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:23)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:257)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
at HotRodTestClient.updateCache(HotRodTestClient.java:69)
at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:88)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Created attachment 1084471[details]
reproducer.zip
Steps to reproduce:
- create two servers as copy
- copy jdg-test1/* to one and jdg-test2/* to the other
- start the nodes
bin/clustered.sh -b localhost -c=custom-clustered.xml -Djboss.node.name=server0
bin/clustered.sh -b localhost -c=custom-clustered.xml -Djboss.node.name=server1 -Djboss.socket.binding.port-offset=100
- run the client from the hotrodclient directory
java -cp /..../jboss-datagrid-6.5.1-remote-java-client/*:target/hotrodtestclient-1.0-SNAPSHOT.jar -Dinfinispan.client.hotrod.server_list="localhost:11222;localhost:11322" -Dinfinispan.client.hotrod.trust_store_file_name=localhost.jks -Dinfinispan.client.hotrod.trust_store_password=localhost.test HotRodTestClient testwriter Test1Writer!
- stop one of the servers (maybe retry)
Comment 3JBoss JIRA Server
2015-10-21 06:15:12 UTC
Galder Zamarreño <galder.zamarreno> updated the status of jira ISPN-5875 to Coding In Progress