Bug 1002957 - Hot Rod client doesn't retry operation on RemoteException
Hot Rod client doesn't retry operation on RemoteException
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
Unspecified Unspecified
high Severity high
: ER4
: 6.2.0
Assigned To: Tristan Tarrant
nobody nobody
Depends On:
Blocks: 1017190
  Show dependency treegraph
Reported: 2013-08-30 07:01 EDT by Michal Linhard
Modified: 2014-11-30 16:15 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
There are certain types of remote-side exceptions that the Java Hot Rod client will not recover from by retrying, even though it should be possible. This also happens when the org.infinispan.remoting.RemoteException is thrown in the server. The HotRodClientException is thrown on the client side, the operation does not retry and the issue needs to be handled by the user code.
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-3454 Critical Resolved Hot Rod client doesn't retry operation on RemoteException 2015-08-19 21:43:36 EDT

  None (edit)
Description Michal Linhard 2013-08-30 07:01:55 EDT
Found in JDG 6.2.0.DR3
Comment 2 Mircea Markus 2013-10-09 11:28:53 EDT
is this a GA blocker? 
there's no data loss here and there's a workaround: client can retry.
Comment 3 JBoss JIRA Server 2013-11-04 13:21:00 EST
Galder Zamarreño <galder.zamarreno@redhat.com> updated the status of jira ISPN-3454 to Coding In Progress
Comment 5 JBoss JIRA Server 2013-11-05 07:37:59 EST
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454

This is a side effect of ISPN-2737, where we introduced RemoteException in order to report application-level exceptions (i.e. lock timeout). What's happening here is that it seems that on occasion, [SuspectException is treated as a RemoteException|https://gist.github.com/galderz/bd1a71a06fa24efa7792], and others it just bubbles up as [SuspectException|https://gist.github.com/galderz/a121b1727a7f4ac1b78c]. The bug here is that SuspectExceptions should not be treated as RemoteException. RemoteException instances on the client should be reported by to the client, as it happens here, since they are application-level errors.
Comment 6 JBoss JIRA Server 2013-11-05 07:49:56 EST
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454

The reason this is happening is because node02 has been suspected. When SuspectException is returned, node01 has suspected node02 [directly|https://gist.github.com/galderz/a121b1727a7f4ac1b78c]. SuspectException is wrapped in a RemoteExcetion when node01 receives a response from node04 saying that node02 has been [suspected|https://gist.github.com/galderz/bd1a71a06fa24efa7792]. RemoteException handling logic should check the type of the exception returned in the response. Fixing this should be relatively straightforward. What's much harder to do is have a unit test to verify this. I'll see what can be done here.
Comment 7 JBoss JIRA Server 2013-11-05 09:19:18 EST
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454

Try [branch|https://github.com/galderz/infinispan/tree/t_3454] with possible fix in resilience jobs.
Comment 8 JBoss JIRA Server 2013-11-06 05:27:10 EST
Michal Linhard <mlinhard@redhat.com> made a comment on jira ISPN-3454

[~galder.zamarreno] I've got some compilation failure on both jenkins and my laptop, do you know what could be the problem ?

[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.679s
[INFO] Finished at: Wed Nov 06 11:17:09 CET 2013
[INFO] Final Memory: 45M/513M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project infinispan-core: Compilation failure
[ERROR] /home/mlinhard/dev/source/jdg-branch/t_3454/infinispan/core/src/main/java/org/infinispan/util/logging/Log.java:[1025,21]
[ERROR] -> [Help 1]

When I open the file in Eclipse I can't see any error, but the maven build fails..
Comment 9 JBoss JIRA Server 2013-11-06 08:01:46 EST
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454

There was a couple of errors in the message logging part, which was detected by the annotation processor. I've updated the branch.
Comment 11 JBoss JIRA Server 2013-11-08 03:48:17 EST
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454

Thanks [~mlinhard] for running these tests. Glad to hear that issue appears to be gone. I'll submit a PR asap.
Comment 12 JBoss JIRA Server 2013-11-08 04:14:44 EST
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454

ISPN-2737 caused this issue.
Comment 13 Michal Linhard 2013-11-20 07:35:21 EST
Verified for 6.2.0.ER4

Note You need to log in before you can comment on or make changes to this bug.