1342365 – Client doesn't retry a non-tx write operation when one of backups is suspected

Bug 1342365 - Client doesn't retry a non-tx write operation when one of backups is suspected

Summary: Client doesn't retry a non-tx write operation when one of backups is suspected

Keywords:
Status:	VERIFIED
Alias:	None
Product:	JBoss Data Grid 6
Classification:	JBoss
Component:	Infinispan
Sub Component:
Version:	6.6.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	ER1
Target Release:	6.6.1
Assignee:	Tristan Tarrant
QA Contact:	Martin Gencur
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1309749 1352446
TreeView+	depends on / blocked

Reported:	2016-06-03 05:57 UTC by Osamu Nagano
Modified:	2022-03-03 09:11 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1352446 (view as bug list)
Environment:
Last Closed:
Type:	Bug
Embargoed:

Attachments	(Terms of Use)
jdg700er9_server.log.zip (194.32 KB, application/zip) 2016-06-30 07:44 UTC, Osamu Nagano	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	2443091	0	None	None	None	2016-07-14 02:07:48 UTC

Description Osamu Nagano 2016-06-03 05:57:29 UTC

Cache.remove() of a Hot Rod Java client failed with the following stack trace without trying to retry.

~~~
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19706880] returned server error (status=0x85): org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: SuspectedException
        at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
        at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
        at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
        at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
        at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.sendKeyOperation(AbstractKeyOperation.java:53)
        at org.infinispan.client.hotrod.impl.operations.RemoveOperation.executeOperation(RemoveOperation.java:28)
        at org.infinispan.client.hotrod.impl.operations.RemoveOperation.executeOperation(RemoveOperation.java:18)
        at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
        at org.infinispan.client.hotrod.impl.RemoteCacheImpl.remove(RemoteCacheImpl.java:479)
        ...
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:745)
~~~

On the primary server side, the following ERROR log is recorded.

~~~
14:11:09,105 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-114) ISPN000136: Error executing command RemoveCommand, writing keys [[B@48347a66]: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: SuspectedException
        at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:658) [infinispan-commons-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:140) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:575) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:292) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:299) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitRemoveCommand(NonTxDistributionInterceptor.java:186) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:366) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:454) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:245) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:93) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:340) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:272) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:138) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:200) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:105) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:74) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:48) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1610) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:528) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.CacheImpl.remove(CacheImpl.java:521) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.CacheImpl.remove(CacheImpl.java:516) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:295) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.CacheDecodeContext.remove(CacheDecodeContext.scala:231) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.handleModification(HotRodDecoder.scala:227) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:110) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:52) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:219) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:49) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:141) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32) [infinispan-server-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:341) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:327) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45]
Caused by: java.util.concurrent.ExecutionException: SuspectedException
        at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:195) [jgroups-3.6.3.Final-redhat-4.jar:3.6.3.Final-redhat-4]
        at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:230) [jgroups-3.6.3.Final-redhat-4.jar:3.6.3.Final-redhat-4]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:478) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:131) [infinispan-core-6.4.0.Final-redhat-4-bz-1328307.jar:6.4.0.Final-redhat-4]
        ... 53 more
Caused by: SuspectedException
        ... 57 more
~~~

It seems an ExecutionException should be ignored if its root cause is a SuspectedException in CommandAwareRpcDispatcher.processCalls() or unwrap and re-throw the SuspectedException in invokeRemoteCommand[s]().


Configuration:
  JDG 6.6.0 with BZ-1328307 and Java client
  SYNC mode distribution cache with 2 owners

Comment 2 Osamu Nagano 2016-06-07 02:49:20 UTC

This is reproducible using the latest Byteman which is able to use a module class loader and 4 node cluster with a 3 owners dist-cache.  Owners should be 3 to call "invokeRemoteCommands()", not "invokeRemoteCommand()".  And nodes are 4 to not use broadcast.  Then implement the following Byteman script and call "put()" or "remove()" from a Hot Rod client.  Exactly the same stack trace will appear on the primary node and the client will not retry by throwing HotRodClientException.

clustered.conf:
~~~
BYTEMAN_HOME=/path/to/byteman-download-3.0.6
...
   JAVA_OPTS="$JAVA_OPTS -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=sys:${BYTEMAN_HOME}/lib/byteman.jar,sys:${BYTEMAN_HOME}/contrib/jboss-modules-system/byteman-jboss-modules-plugin.jar,modules:org.jboss.byteman.modules.jbossmodules.JBossModulesSystem,script:/path/to/suspect-backups.btm"
~~~

suspect-backups.btm:
~~~
RULE unicast.getvalue fails
CLASS org.jgroups.blocks.UnicastRequest
IMPORT org.jgroups
METHOD getValue
AT ENTRY
IF java.lang.Thread.currentThread().getName().startsWith("HotRodServerWorker-")
DO throw new java.util.concurrent.ExecutionException(new org.jgroups.SuspectedException("dummy"));
ENDRULE
~~~

The customer is using 2 owners cache but the stack trace is showing "invokeRemoteCommands()", which means operating on multiple backups.  I don't know why there are multiple backups with 2 owners cache.  My guess is non-primary server node received the first request because the customer was testing a scenario of force shutting down of a machine.

Comment 3 Galder Zamarreño 2016-06-29 17:47:57 UTC

Can you try with JDG 7 if the issue is present? This issue could have been solved indirectly by another fix.

The reason I ask is because I've done some testing in community master and the method that generates the UnicastRequest.getValue() method does not get called under the circumstances specified above, and it's this method the one that can potentially generate the wrapped `ExecutionException(new SuspectedException(target))`.

Comment 4 Osamu Nagano 2016-06-30 07:44:00 UTC

Created attachment 1174365 [details]
jdg700er9_server.log.zip

It seems reproducible even with JDG 7.0.0.ER9. I changed the Byteman script as follows.
~~~
RULE processCallsStaggered fails
CLASS org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher
IMPORT org.jgroups
METHOD processCallsStaggered
AT ENTRY
IF java.lang.Thread.currentThread().getName().startsWith("HotRodServerHandler-")
DO throw new java.util.concurrent.ExecutionException(new org.jgroups.SuspectedException("dummy"));
ENDRULE

RULE processCalls fails
CLASS org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher
IMPORT org.jgroups
METHOD processCalls
AT ENTRY
IF java.lang.Thread.currentThread().getName().startsWith("HotRodServerHandler-")
DO throw new java.util.concurrent.ExecutionException(new org.jgroups.SuspectedException("dummy"));
ENDRULE
~~~

Note that I instrumented CommandAwareRpcDispatcher this time. I believe it has the same effect with instrumenting UnicastRequest.getValue() on JDG 6.6.0. I'll attach the TRACE server.log as jdg700er9_server.log.zip. The server side stack trace is below.

~~~
2016-06-30 16:28:22,201 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (HotRodServerHandler-4-1) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: SuspectedException
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:150)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:130)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:610)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:126)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:167)
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:366)
	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:99)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:488)
	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:553)
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:212)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:99)
	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:37)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:65)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:358)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:290)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:117)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:192)
	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:178)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:81)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:101)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:115)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1660)
	at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1194)
	at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1184)
	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:526)
	at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:230)
	at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:194)
	at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:73)
	at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:56)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: SuspectedException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.jboss.byteman.rule.expression.ThrowExpression.interpret(ThrowExpression.java:230)
	at org.jboss.byteman.rule.Action.interpret(Action.java:144)
	at org.jboss.byteman.rule.helper.Helper_HelperAdapter_Interpreted_1.fire(suspect-backups.btm)
	at org.jboss.byteman.rule.helper.Helper_HelperAdapter_Interpreted_1.execute0(suspect-backups.btm)
	at org.jboss.byteman.rule.helper.Helper_HelperAdapter_Interpreted_1.execute(suspect-backups.btm)
	at org.jboss.byteman.rule.Rule.execute(Rule.java:777)
	at org.jboss.byteman.rule.Rule.execute(Rule.java:746)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:125)
	... 47 more
Caused by: SuspectedException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.jboss.byteman.rule.expression.NewExpression.interpret(NewExpression.java:306)
	at org.jboss.byteman.rule.expression.ThrowExpression.interpret(ThrowExpression.java:227)
	... 55 more
~~~

Comment 6 Galder Zamarreño 2016-06-30 20:48:32 UTC

I'm not sure if your Bytemnan rule is realistic in JDG 7.0 cos I don't see a method within processCallsStaggered that can throw an ExecutionException just like it did in 6.6. The code has changed considerably in this class.

Although there's a relatively easy fix for JDG 6.6, I need to consult with other members of the team to see ExecutionException could end up being fired in JDG 7.0 given the updates.

Comment 7 Dan Berindei 2016-07-03 08:50:56 UTC

Galder is right, the Byteman rule for JDG 7.0.0.ER9 is not realistic. It's impossible to get an ExecutionException (or even a SuspectedException) in CommandAwareRpcDispatcher now.

I have opened a PR to always unwrap the ExecutionException for JDG 6.6:

https://github.com/infinispan/jdg/pull/1004

Note You need to log in before you can comment on or make changes to this bug.