Bug 1273137 - HotRod Client does not handle failover silent when nodes are stopped if using role based authentication
Summary: HotRod Client does not handle failover silent when nodes are stopped if using...
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Server
Version: 6.5.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ER1
: 6.6.0
Assignee: Galder Zamarreño
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-19 18:00 UTC by wfink
Modified: 2023-05-31 23:39 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
In Red Hat JBoss Data Grid, when roles based authentication was enabled, the Hot Rod client did not fail over; instead, it logged the exception when the JDG server to which the client was connected and went offline. This issue is resolved as of Red Hat JBoss Data Grid 6.6.0. If the JDG server shuts down, then the Hot Rod client will fail over to another JDG server in the cluster silently, with no exception logged and the Hot Rod client will continue to work properly.
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)
reproducer.zip (20.40 KB, application/zip)
2015-10-19 18:09 UTC, wfink
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-5875 0 Critical Resolved HotRod Client does not handle failover silent when nodes are stopped if using role based authentication 2018-02-21 04:29:09 UTC

Description wfink 2015-10-19 18:00:15 UTC
Description of problem:


A HotRod client fail over silent if one instance is shutting down if there is no Security enabled.
With role based authentication the client fail sporadically with a WARN message

WARN: ISPN004005: Error received from the server: java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.

and the following Exception:

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[2825] returned server error (status=0x85): java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:57)
at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:36)
at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:23)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:257)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
at HotRodTestClient.updateCache(HotRodTestClient.java:69)
at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:88)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Comment 2 wfink 2015-10-19 18:09:46 UTC
Created attachment 1084471 [details]
reproducer.zip

Steps to reproduce:
- create two servers as copy
- copy jdg-test1/* to one and jdg-test2/* to the other
- start the nodes
  bin/clustered.sh -b localhost -c=custom-clustered.xml -Djboss.node.name=server0
  bin/clustered.sh -b localhost -c=custom-clustered.xml -Djboss.node.name=server1 -Djboss.socket.binding.port-offset=100

- run the client from the hotrodclient directory
java -cp /..../jboss-datagrid-6.5.1-remote-java-client/*:target/hotrodtestclient-1.0-SNAPSHOT.jar -Dinfinispan.client.hotrod.server_list="localhost:11222;localhost:11322" -Dinfinispan.client.hotrod.trust_store_file_name=localhost.jks -Dinfinispan.client.hotrod.trust_store_password=localhost.test HotRodTestClient testwriter Test1Writer!

- stop one of the servers (maybe retry)

Comment 3 JBoss JIRA Server 2015-10-21 06:15:12 UTC
Galder Zamarreño <galder.zamarreno> updated the status of jira ISPN-5875 to Coding In Progress


Note You need to log in before you can comment on or make changes to this bug.