Bug 1305407

Summary: ISPN000260: Exception executing command: java.lang.RuntimeException: Do we still need to deal with non-visitable commands? (org.infinispan.xsite.statetransfer.XSiteStateTransferControlCommand)
Product: [JBoss] JBoss Data Grid 6 Reporter: Osamu Nagano <onagano>
Component: InfinispanAssignee: Pedro Ruivo <pruivo>
Status: ASSIGNED --- QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.5.1CC: afield, jdg-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
xsiterepl.zip
none
eap640.zip none

Description Osamu Nagano 2016-02-08 04:24:14 UTC
With repl-async cache, XSiteAdmin.pushState() generates an error on the sending node.  Replication has succeeded but the PushStateStatus remains to SENDING.  As a workaround, you can clear the status by XSiteAdmin.clearPushStateStatus().

~~~
14:24:56,569 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (Incoming-9,master:server101-4765) ISPN000260: Exception executing command: java.lang.RuntimeException: Do we still need to deal with non-visitable commands? (org.infinispan.xsite.statetransfer.XSiteStateTransferControlCommand)
        at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:46) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.infinispan.commands.remote.MultipleRpcCommand.perform(MultipleRpcCommand.java:55) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:210) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:81) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:310) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:254) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.JChannel.up(JChannel.java:761) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1033) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.relay.RELAY2.up(RELAY2.java:446) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:447) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.stack.Protocol.up(Protocol.java:420) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1087) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:790) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.stack.Protocol.up(Protocol.java:374) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:299) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:286) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.Discovery.up(Discovery.java:291) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) [infinispan-embedded-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]
~~~

Comment 1 Osamu Nagano 2016-02-08 04:27:26 UTC
Created attachment 1122074 [details]
xsiterepl.zip

Sample library mode web app configured cross-site replication with 2 sites.

Comment 2 Osamu Nagano 2016-02-08 04:28:53 UTC
Created attachment 1122075 [details]
eap640.zip

Sample EAP 6.4.0 domain with 2 hosts configured (maser and slave).

Comment 4 Osamu Nagano 2016-02-08 04:38:29 UTC
Reproducing steps:

1. Build the sample app.
unzip  xsiterepl.zip
cd xsiterepl
mvn package

2. Start master host-controller (site "S1") on 127.0.0.1.
cd ..
unzip eap640.zip
cd eap640
$JBOSS_HOME/bin/domain.sh -Djboss.domain.base.dir=./master --host-config=host-master.xml

3. Start slave host-controller (site "S2") on 127.0.1.1.
sudo ip addr add 127.0.1.1/8 dev lo label lo:1
$JBOSS_HOME/bin/domain.sh -Djboss.domain.base.dir=./slave --host-config=host-slave.xml --master-address=127.0.0.1 -b 127.0.1.1 -bmanagement=127.0.1.1

4. Deploy the app and activate it to form clusters.
$JBOSS_HOME/bin/jboss-cli.sh -c 'deploy ../xsiterepl/target/xsiterepl.war --all-server-groups'
curl http://127.0.0.1:8080/xsiterepl/rest/repl-async-cache/get/key000001
curl http://127.0.0.1:8230/xsiterepl/rest/repl-async-cache/get/key000001
curl http://127.0.1.1:8080/xsiterepl/rest/repl-async-cache/get/key000001
curl http://127.0.1.1:8230/xsiterepl/rest/repl-async-cache/get/key000001

5. Put some test data
curl http://127.0.0.1:8080/xsiterepl/rest/repl-async-cache/filldata/1/20

6. Invoke XSiteAdmin.pushState() by JConsole on one node.
You'll see the error log message on the node.