Description of problem: See Description of ISPN-3376
Tristan Tarrant <ttarrant> made a comment on jira ISPN-3376 Can you please include source and target logs ?
Vitalii Chepeliuk <vchepeli> made a comment on jira ISPN-3376 Added server logs
Testes and looks like this bug is not fixed. The same error as before I do this In order to perform a rolling upgrade of a HotRod cluster, the following steps must be taken 1. Configure and start a new cluster with a RemoteCacheStore pointing to the old cluster and the hotRodWrapping flag enabled 2. Configure all clients so that they will connect to the new cluster 3. Invoke the upgrade --dumpkeys command on the old cluster for all of the caches that need to be migrated 4. Invoke the upgrade --synchronize=hotrod command on the new cluster to ensure that all data is migrated from the old cluster to the new one 5. Invoke the upgrade --disconnectsource=hotrod command on the new cluster to disable the RemoteCacheStore used to migrate the data 6. Switch off the old cluster And get exception And exception ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server
Vitalii: I have tried several times and cannot reproduce this issue.
Here what I was doing, 6.1.0.GA server [remoting://localhost:9999/local/default]> upgrade --dumpkeys default ISPN019502: Dumped keys for cache default [remoting://localhost:9999/local/default]> upgrade --dumpkeys --all ISPN019502: Dumped keys for cache default ISPN019502: Dumped keys for cache namedCache ISPN019502: Dumped keys for cache memcachedCache ISPN019502: Dumped keys for cache ___defaultcache [remoting://localhost:9999/local/default]> 6.2.0.ER4 server [remoting://localhost:10110/local/]> upgrade --synchronize=hotrod default ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [remoting://localhost:10110/local/]> upgrade --synchronize=hotrod ISPN019016: No such cache: 'null' [remoting://localhost:10110/local/]> upgrade --synchronize=hotrod default ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [remoting://localhost:10110/local/]> [remoting://localhost:10110/local/default]> upgrade --synchronize=hotrod --all ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server Maybe I am losing something but I do what is written what "help upgrade" says in USAGE
ah, it looks like it's confusing the cache name with the synchronizer name
(In reply to Tristan Tarrant from comment #7) > ah, it looks like it's confusing the cache name with the synchronizer name Yes, that's only problem remains I think
Mircea Markus <mmarkus> updated the status of jira ISPN-3376 to Resolved
CR1 client uses new protocol version of hotrod 13. When 1.3 client communicates with 6.1.0 GA server following exception is thrown 23:36:05,923 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-4) ISPN005003: Exception reported: org.infinispan.server.hotrod.RequestParsingException: Unable to parse header at org.infinispan.server.hotrod.HotRodDecoder.readHeader(HotRodDecoder.scala:94) [infinispan-server-hotrod-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] at org.infinispan.server.hotrod.HotRodDecoder.readHeader(HotRodDecoder.scala:45) [infinispan-server-hotrod-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] at org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:94) [infinispan-server-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:70) [infinispan-server-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45] Caused by: org.infinispan.server.hotrod.UnknownVersionException: Unknown version:13 at org.infinispan.server.hotrod.HotRodDecoder.readHeader(HotRodDecoder.scala:80) [infinispan-server-hotrod-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1] ... 22 more Threre is no option for upgrade command to specify protocol version of hotrod client to be used.
Works nicely in CR2. Verified. The incorrect warning "ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server" (incorrect just because the migrator and the cache name are swapped in it) might still happen but only if there's another problem, such as configuration issue. The rolling upgrade process is smooth when performed correctly.
So I think this command does not work now properly Steps to reproduce 1) Unzip ER4 build and copy to server0 folder 2) Unzip ER4 build and copy to server1 folder 3) copy sample configuration from examples/configs folder a)cp server1/docs/examples/configs/standalone-hotrod-rolling-upgrade.xml to server1/standalone/configuration/ b)cp server1/docs/examples/configs/standalone-rest-rolling-upgrade.xml to server1/standalone/configuration/ 4) change configuration for server1/stanalone/configuration/standalone-hotrod-rolling-upgrade.xml a) change default port-offset from 0 to 100 b) change remote-host to localhost in outbound socket binding configuration 5) start servers a) start server0, server0/bin/standalone.sh -c standalone.xml b) start server1, server1/bin/standalone.sh -c standalone-hotrod-rolling-upgrade.xml 6) connect to server via cli a) server0/bin/ispn-cli.ch connect localhost:9999 cache default put --codec=hotrod key1 val1 put --codec=hotrod key2 val2 put --codec=hotrod key3 val3 upgrade --dumpkeys --all Should see message ISPN019502: Dumped keys for cache namedCache ISPN019502: Dumped keys for cache default ISPN019502: Dumped keys for cache memcachedCache ISPN019502: Dumped keys for cache ___defaultcache b) server1/bin/ispn-cli.sh connect localhost:10099 upgrade --synchronize=hotrod Following message is shown ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server
Here is output from my console [standalone@localhost:10099 /] upgrade --synchronize=hotrod ISPN019016: No such cache: 'null' [standalone@localhost:10099 /] upgrade --synchronize=hotrod default ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] upgrade --synchronize=hotrod --all ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] upgrade --synchronize=hotrod --all ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] cache default [standalone@localhost:10099 /] upgrade --synchronize=hotrod ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /]