Bug 1342365
| Summary: | Client doesn't retry a non-tx write operation when one of backups is suspected | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Data Grid 6 | Reporter: | Osamu Nagano <onagano> | ||||
| Component: | Infinispan | Assignee: | Tristan Tarrant <ttarrant> | ||||
| Status: | CLOSED UPSTREAM | QA Contact: | Martin Gencur <mgencur> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.6.0 | CC: | afield, galder.zamarreno, jdg-bugs, wfink | ||||
| Target Milestone: | ER1 | ||||||
| Target Release: | 6.6.1 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1352446 (view as bug list) | Environment: | |||||
| Last Closed: | 2025-02-10 03:49:02 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1309749, 1352446 | ||||||
| Attachments: |
|
||||||
This is reproducible using the latest Byteman which is able to use a module class loader and 4 node cluster with a 3 owners dist-cache. Owners should be 3 to call "invokeRemoteCommands()", not "invokeRemoteCommand()". And nodes are 4 to not use broadcast. Then implement the following Byteman script and call "put()" or "remove()" from a Hot Rod client. Exactly the same stack trace will appear on the primary node and the client will not retry by throwing HotRodClientException.
clustered.conf:
~~~
BYTEMAN_HOME=/path/to/byteman-download-3.0.6
...
JAVA_OPTS="$JAVA_OPTS -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=sys:${BYTEMAN_HOME}/lib/byteman.jar,sys:${BYTEMAN_HOME}/contrib/jboss-modules-system/byteman-jboss-modules-plugin.jar,modules:org.jboss.byteman.modules.jbossmodules.JBossModulesSystem,script:/path/to/suspect-backups.btm"
~~~
suspect-backups.btm:
~~~
RULE unicast.getvalue fails
CLASS org.jgroups.blocks.UnicastRequest
IMPORT org.jgroups
METHOD getValue
AT ENTRY
IF java.lang.Thread.currentThread().getName().startsWith("HotRodServerWorker-")
DO throw new java.util.concurrent.ExecutionException(new org.jgroups.SuspectedException("dummy"));
ENDRULE
~~~
The customer is using 2 owners cache but the stack trace is showing "invokeRemoteCommands()", which means operating on multiple backups. I don't know why there are multiple backups with 2 owners cache. My guess is non-primary server node received the first request because the customer was testing a scenario of force shutting down of a machine.
Can you try with JDG 7 if the issue is present? This issue could have been solved indirectly by another fix. The reason I ask is because I've done some testing in community master and the method that generates the UnicastRequest.getValue() method does not get called under the circumstances specified above, and it's this method the one that can potentially generate the wrapped `ExecutionException(new SuspectedException(target))`. Created attachment 1174365 [details]
jdg700er9_server.log.zip
It seems reproducible even with JDG 7.0.0.ER9. I changed the Byteman script as follows.
~~~
RULE processCallsStaggered fails
CLASS org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher
IMPORT org.jgroups
METHOD processCallsStaggered
AT ENTRY
IF java.lang.Thread.currentThread().getName().startsWith("HotRodServerHandler-")
DO throw new java.util.concurrent.ExecutionException(new org.jgroups.SuspectedException("dummy"));
ENDRULE
RULE processCalls fails
CLASS org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher
IMPORT org.jgroups
METHOD processCalls
AT ENTRY
IF java.lang.Thread.currentThread().getName().startsWith("HotRodServerHandler-")
DO throw new java.util.concurrent.ExecutionException(new org.jgroups.SuspectedException("dummy"));
ENDRULE
~~~
Note that I instrumented CommandAwareRpcDispatcher this time. I believe it has the same effect with instrumenting UnicastRequest.getValue() on JDG 6.6.0. I'll attach the TRACE server.log as jdg700er9_server.log.zip. The server side stack trace is below.
~~~
2016-06-30 16:28:22,201 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (HotRodServerHandler-4-1) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: SuspectedException
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:150)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:130)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:610)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:126)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:167)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:366)
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:99)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:488)
at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:553)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:212)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:99)
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:37)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:65)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:358)
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:290)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:117)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:192)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:178)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:81)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:115)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1660)
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1194)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1184)
at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:526)
at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:230)
at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:194)
at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:73)
at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:56)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: SuspectedException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.jboss.byteman.rule.expression.ThrowExpression.interpret(ThrowExpression.java:230)
at org.jboss.byteman.rule.Action.interpret(Action.java:144)
at org.jboss.byteman.rule.helper.Helper_HelperAdapter_Interpreted_1.fire(suspect-backups.btm)
at org.jboss.byteman.rule.helper.Helper_HelperAdapter_Interpreted_1.execute0(suspect-backups.btm)
at org.jboss.byteman.rule.helper.Helper_HelperAdapter_Interpreted_1.execute(suspect-backups.btm)
at org.jboss.byteman.rule.Rule.execute(Rule.java:777)
at org.jboss.byteman.rule.Rule.execute(Rule.java:746)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:125)
... 47 more
Caused by: SuspectedException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.jboss.byteman.rule.expression.NewExpression.interpret(NewExpression.java:306)
at org.jboss.byteman.rule.expression.ThrowExpression.interpret(ThrowExpression.java:227)
... 55 more
~~~
I'm not sure if your Bytemnan rule is realistic in JDG 7.0 cos I don't see a method within processCallsStaggered that can throw an ExecutionException just like it did in 6.6. The code has changed considerably in this class. Although there's a relatively easy fix for JDG 6.6, I need to consult with other members of the team to see ExecutionException could end up being fired in JDG 7.0 given the updates. Galder is right, the Byteman rule for JDG 7.0.0.ER9 is not realistic. It's impossible to get an ExecutionException (or even a SuspectedException) in CommandAwareRpcDispatcher now. I have opened a PR to always unwrap the ExecutionException for JDG 6.6: https://github.com/infinispan/jdg/pull/1004 This product has been discontinued or is no longer tracked in Red Hat Bugzilla. |
Cache.remove() of a Hot Rod Java client failed with the following stack trace without trying to retry. ~~~ org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19706880] returned server error (status=0x85): org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: SuspectedException at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336) at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126) at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112) at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56) at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.sendKeyOperation(AbstractKeyOperation.java:53) at org.infinispan.client.hotrod.impl.operations.RemoveOperation.executeOperation(RemoveOperation.java:28) at org.infinispan.client.hotrod.impl.operations.RemoveOperation.executeOperation(RemoveOperation.java:18) at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52) at org.infinispan.client.hotrod.impl.RemoteCacheImpl.remove(RemoteCacheImpl.java:479) ... at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:745) ~~~ On the primary server side, the following ERROR log is recorded. ~~~ 14:11:09,105 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-114) ISPN000136: Error executing command RemoveCommand, writing keys [[B@48347a66]: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: SuspectedException at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:658) [infinispan-commons-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:140) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:575) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:292) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:299) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitRemoveCommand(NonTxDistributionInterceptor.java:186) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:366) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:454) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:245) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:93) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:340) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:272) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:138) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:200) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:105) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:74) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:48) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1610) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:528) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.CacheImpl.remove(CacheImpl.java:521) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.CacheImpl.remove(CacheImpl.java:516) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:295) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.CacheDecodeContext.remove(CacheDecodeContext.scala:231) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.handleModification(HotRodDecoder.scala:227) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:110) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:52) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:219) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:49) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:141) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32) [infinispan-server-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:341) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:327) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45] Caused by: java.util.concurrent.ExecutionException: SuspectedException at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:195) [jgroups-3.6.3.Final-redhat-4.jar:3.6.3.Final-redhat-4] at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:230) [jgroups-3.6.3.Final-redhat-4.jar:3.6.3.Final-redhat-4] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:478) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:131) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4] ... 53 more Caused by: SuspectedException ... 57 more ~~~ It seems an ExecutionException should be ignored if its root cause is a SuspectedException in CommandAwareRpcDispatcher.processCalls() or unwrap and re-throw the SuspectedException in invokeRemoteCommand[s](). Configuration: JDG 6.6.0 with BZ-1328307 and Java client SYNC mode distribution cache with 2 owners