Bug 1273137 - HotRod Client does not handle failover silent when nodes are stopped if using role based authentication
HotRod Client does not handle failover silent when nodes are stopped if using...
Status: VERIFIED
Product: JBoss Data Grid 6
Classification: JBoss
Component: Server (Show other bugs)
6.5.0
Unspecified Unspecified
high Severity high
: ER1
: 6.6.0
Assigned To: Galder Zamarreño
Martin Gencur
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-19 14:00 EDT by wfink
Modified: 2018-02-07 19:27 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
In Red Hat JBoss Data Grid, when roles based authentication was enabled, the Hot Rod client did not fail over; instead, it logged the exception when the JDG server to which the client was connected and went offline. This issue is resolved as of Red Hat JBoss Data Grid 6.6.0. If the JDG server shuts down, then the Hot Rod client will fail over to another JDG server in the cluster silently, with no exception logged and the Hot Rod client will continue to work properly.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
reproducer.zip (20.40 KB, application/zip)
2015-10-19 14:09 EDT, wfink
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-5875 Critical Resolved HotRod Client does not handle failover silent when nodes are stopped if using role based authentication 2018-02-20 23:29 EST

  None (edit)
Description wfink 2015-10-19 14:00:15 EDT
Description of problem:


A HotRod client fail over silent if one instance is shutting down if there is no Security enabled.
With role based authentication the client fail sporadically with a WARN message

WARN: ISPN004005: Error received from the server: java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.

and the following Exception:

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[2825] returned server error (status=0x85): java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:57)
at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:36)
at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:23)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:257)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
at HotRodTestClient.updateCache(HotRodTestClient.java:69)
at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:88)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Comment 2 wfink 2015-10-19 14:09 EDT
Created attachment 1084471 [details]
reproducer.zip

Steps to reproduce:
- create two servers as copy
- copy jdg-test1/* to one and jdg-test2/* to the other
- start the nodes
  bin/clustered.sh -b localhost -c=custom-clustered.xml -Djboss.node.name=server0
  bin/clustered.sh -b localhost -c=custom-clustered.xml -Djboss.node.name=server1 -Djboss.socket.binding.port-offset=100

- run the client from the hotrodclient directory
java -cp /..../jboss-datagrid-6.5.1-remote-java-client/*:target/hotrodtestclient-1.0-SNAPSHOT.jar -Dinfinispan.client.hotrod.server_list="localhost:11222;localhost:11322" -Dinfinispan.client.hotrod.trust_store_file_name=localhost.jks -Dinfinispan.client.hotrod.trust_store_password=localhost.test HotRodTestClient testwriter Test1Writer!

- stop one of the servers (maybe retry)
Comment 3 JBoss JIRA Server 2015-10-21 02:15:12 EDT
Galder Zamarreño <galder.zamarreno@redhat.com> updated the status of jira ISPN-5875 to Coding In Progress

Note You need to log in before you can comment on or make changes to this bug.