Red Hat Bugzilla – Bug 1002957
Hot Rod client doesn't retry operation on RemoteException
Last modified: 2014-11-30 16:15:57 EST
Found in JDG 6.2.0.DR3 see https://issues.jboss.org/browse/ISPN-3454
is this a GA blocker? there's no data loss here and there's a workaround: client can retry.
Galder Zamarreño <galder.zamarreno@redhat.com> updated the status of jira ISPN-3454 to Coding In Progress
Michal Linhard <mlinhard@redhat.com> made a comment on jira ISPN-3454 TRACE logs https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/RESILIENCE/job/jdg-func-resilience-dist-4-3/90/artifact/report/clientlogs.zip https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/RESILIENCE/job/jdg-func-resilience-dist-4-3/90/artifact/report/serverlogs.zip
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454 This is a side effect of ISPN-2737, where we introduced RemoteException in order to report application-level exceptions (i.e. lock timeout). What's happening here is that it seems that on occasion, [SuspectException is treated as a RemoteException|https://gist.github.com/galderz/bd1a71a06fa24efa7792], and others it just bubbles up as [SuspectException|https://gist.github.com/galderz/a121b1727a7f4ac1b78c]. The bug here is that SuspectExceptions should not be treated as RemoteException. RemoteException instances on the client should be reported by to the client, as it happens here, since they are application-level errors.
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454 The reason this is happening is because node02 has been suspected. When SuspectException is returned, node01 has suspected node02 [directly|https://gist.github.com/galderz/a121b1727a7f4ac1b78c]. SuspectException is wrapped in a RemoteExcetion when node01 receives a response from node04 saying that node02 has been [suspected|https://gist.github.com/galderz/bd1a71a06fa24efa7792]. RemoteException handling logic should check the type of the exception returned in the response. Fixing this should be relatively straightforward. What's much harder to do is have a unit test to verify this. I'll see what can be done here.
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454 Try [branch|https://github.com/galderz/infinispan/tree/t_3454] with possible fix in resilience jobs.
Michal Linhard <mlinhard@redhat.com> made a comment on jira ISPN-3454 [~galder.zamarreno] I've got some compilation failure on both jenkins and my laptop, do you know what could be the problem ? https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-qe-build-infinispan60-specific/6/console {code} [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 18.679s [INFO] Finished at: Wed Nov 06 11:17:09 CET 2013 [INFO] Final Memory: 45M/513M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project infinispan-core: Compilation failure [ERROR] /home/mlinhard/dev/source/jdg-branch/t_3454/infinispan/core/src/main/java/org/infinispan/util/logging/Log.java:[1025,21] [ERROR] -> [Help 1] [ERROR] {code} When I open the file in Eclipse I can't see any error, but the maven build fails..
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454 There was a couple of errors in the message logging part, which was detected by the annotation processor. I've updated the branch.
Michal Linhard <mlinhard@redhat.com> made a comment on jira ISPN-3454 I've ran the 4 node dist mode resilience test 2 times with your branch and didn't spot the RemoteException https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/RESILIENCE/job/jdg-func-resilience-dist-4-3/91/artifact/report/loganalysis/client/index.html https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/RESILIENCE/job/jdg-func-resilience-dist-4-3/97/artifact/report/loganalysis/client/index.html
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454 Thanks [~mlinhard] for running these tests. Glad to hear that issue appears to be gone. I'll submit a PR asap.
Galder Zamarreño <galder.zamarreno@redhat.com> made a comment on jira ISPN-3454 ISPN-2737 caused this issue.
Verified for 6.2.0.ER4