Red Hat Bugzilla – Bug 1101512
[CLI] UPGRADE command fails when testing data stored via CLI with REST encoding
Last modified: 2018-09-12 18:32:55 EDT
Description of problem: UPGRADE command in CLI does not synchronize data from OLD cluster(node) Version-Release number of selected component (if applicable): 6.3.0.ER4 Steps to reproduce 1) Unzip ER4 build and copy to server0 folder 2) Unzip ER4 build and copy to server1 folder 3) copy sample configuration from examples/configs folder a)cp server1/docs/examples/configs/standalone-hotrod-rolling-upgrade.xml to server1/standalone/configuration/ b)cp server1/docs/examples/configs/standalone-rest-rolling-upgrade.xml to server1/standalone/configuration/ 4) change configuration for server1/stanalone/configuration/standalone-hotrod-rolling-upgrade.xml a) change default port-offset from 0 to 100 b) change remote-host to localhost in outbound socket binding configuration 5) start servers a) start server0, server0/bin/standalone.sh -c standalone.xml b) start server1, server1/bin/standalone.sh -c standalone-hotrod-rolling-upgrade.xml 6) connect to server via cli a) server0/bin/ispn-cli.ch connect localhost:9999 cache default put --codec=hotrod key1 val1 put --codec=hotrod key2 val2 put --codec=hotrod key3 val3 upgrade --dumpkeys --all Should see message ISPN019502: Dumped keys for cache namedCache ISPN019502: Dumped keys for cache default ISPN019502: Dumped keys for cache memcachedCache ISPN019502: Dumped keys for cache ___defaultcache b) server1/bin/ispn-cli.sh connect localhost:10099 upgrade --synchronize=hotrod Following message is shown ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server Here is output from my console [standalone@localhost:10099 /] upgrade --synchronize=hotrod ISPN019016: No such cache: 'null' [standalone@localhost:10099 /] upgrade --synchronize=hotrod default ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] upgrade --synchronize=hotrod --all ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] upgrade --synchronize=hotrod --all ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] cache default [standalone@localhost:10099 /] upgrade --synchronize=hotrod ISPN019026: An error occurred while synchronizing data for cache 'hotrod' using migrator 'default' from the source server [standalone@localhost:10099 /] Actual results: Expected results: Data should be migrated to new node Additional info:
IF I use REST encoding it does not work too! Do not forget to remove BASIC security configuration from REST endpoint [disconnected /] connect localhost [standalone@localhost:9999 /] cache default [standalone@localhost:9999 /] put --codec=rest key1 val1 [standalone@localhost:9999 /] put --codec=rest key2 val2 [standalone@localhost:9999 /] put --codec=rest key3 val3 [standalone@localhost:9999 /] put --codec=rest key4 val4 [standalone@localhost:9999 /] put --codec=rest key5 val5 [standalone@localhost:9999 /] upgrade --dumpkeys --all ISPN019502: Dumped keys for cache namedCache ISPN019502: Dumped keys for cache default ISPN019502: Dumped keys for cache memcachedCache ISPN019502: Dumped keys for cache ___defaultcache [standalone@localhost:9999 /] Get data from other server [standalone@localhost:10099 /] cache default [standalone@localhost:10099 /] upgrade --synchronize=rest ISPN019500: Synchronized 2 entries using migrator 'rest' on cache 'default' [standalone@localhost:10099 /] get key1 "PG9yZy5pbmZpbmlzcGFuLnJlbW90aW5nLk1JTUVDYWNoZUVudHJ5PgogIDxjb250ZW50VHlwZT50ZXh0L3BsYWluPC9jb250ZW50VHlwZT4KICA8ZGF0YT5kbUZzTVE9PTwvZGF0YT4KPC9vcmcuaW5maW5pc3Bhbi5yZW1vdGluZy5NSU1FQ2FjaGVFbnRyeT4=" [standalone@localhost:10099 /] get key1 --codec=rest line 1:4 no viable alternative at input 'key1' [standalone@localhost:10099 /] get --codec=rest key1 [B cannot be cast to org.infinispan.remoting.MIMECacheEntry [standalone@localhost:10099 /] get --codec=rest key2 [B cannot be cast to org.infinispan.remoting.MIMECacheEntry [standalone@localhost:10099 /] get --codec=rest key3 [B cannot be cast to org.infinispan.remoting.MIMECacheEntry [standalone@localhost:10099 /] get --codec=rest key4 [B cannot be cast to org.infinispan.remoting.MIMECacheEntry [standalone@localhost:10099 /] get --codec=rest key5 [B cannot be cast to org.infinispan.remoting.MIMECacheEntry [standalone@localhost:10099 /] get --codec=rest key5
Hotrod rolling works as expected. REST rolling upgrades does not work. Steps to reproduce Do not forget to remove BASIC security configuration from REST endpoint on server we will migrated from [disconnected /] connect localhost [standalone@localhost:9999 /] cache default [standalone@localhost:9999 /] put --codec=rest key1 val1 [standalone@localhost:9999 /] put --codec=rest key2 val2 [standalone@localhost:9999 /] put --codec=rest key3 val3 [standalone@localhost:9999 /] put --codec=rest key4 val4 [standalone@localhost:9999 /] put --codec=rest key5 val5 [standalone@localhost:9999 /] upgrade --dumpkeys ISPN019502: Dumped keys for cache default Get data from other server [standalone@localhost:10099 /] cache default [standalone@localhost:10099 /] upgrade --synchronize=rest ISPN019500: Synchronized 2 entries using migrator 'rest' on cache 'default' And followinf exception is thrown 13:30:46,275 SEVERE [org.jboss.resteasy.core.SynchronousDispatcher] (http-/127.0.0.1:8080-1) Failed executing GET /rest/default/: org.jboss.resteasy.spi.WriterException: java.lang.ClassCastException: [B cannot be cast to java.lang.String at org.jboss.resteasy.core.ServerResponse.writeTo(ServerResponse.java:262) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.writeJaxrsResponse(SynchronousDispatcher.java:616) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:528) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.7.1.Final-redhat-1.jar:2.3.7.1.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.3.1.Final-redhat-1.jar:7.3.1.Final-redhat-1] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String at org.infinispan.rest.Server$$anonfun$getKeys$1$$anonfun$apply$6$$anonfun$apply$7.apply(Server.scala:102) [infinispan-classes.jar:] at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.infinispan.rest.Server$$anonfun$getKeys$1$$anonfun$apply$6.apply(Server.scala:102) [infinispan-classes.jar:] at org.infinispan.rest.Server$$anonfun$getKeys$1$$anonfun$apply$6.apply(Server.scala:102) [infinispan-classes.jar:] at org.infinispan.rest.Server$$anon$1.write(Server.scala:301) [infinispan-classes.jar:] at
Removing from blockers list and adding a not that this should be documented as a known issue if we don't fix it before CR2.
Thanks, Martin. Created a bug to track this for docs and set flag to + to indicate this. https://bugzilla.redhat.com/show_bug.cgi?id=1116246
Is this fixed for CR2?
No, this is not fixed for CR2
Thanks, Martin. Do we want to add some information in the docs for this issue, or just document it in the release notes as a known issue?
Please document is as a known issue in RN. That should be enough.
Thanks, setting the relevant flag for this bug and closing the docs bug.
Vitalii, you changed the title and added "with REST encoding". The doc text mentions HotRod encoding. Can you please clarify when exactly this problem occurs? Thanks
(In reply to Martin Gencur from comment #11) > Vitalii, > you changed the title and added "with REST encoding". The doc text mentions > HotRod encoding. Can you please clarify when exactly this problem occurs? > Thanks That occurs when you put data not with HTTP client but via CLI as I mentioned above put --codec=rest key1 val1, If --codec=hotrod then test passes