Bug 1031660

Tomas Sykora <tsykora> made a comment on jira ISPN-3724

might be connected to: https://issues.jboss.org/browse/ISPN-3723

Tomas Sykora <tsykora> made a comment on jira ISPN-3724

Attached server output when trying to fetch entry from "old" RCS

Tomas Sykora <tsykora> made a comment on jira ISPN-3724

Interesting thing is that the whole process seems to be ok. Entries are migrated. But process using rest is trying to do something with ___MigrationManager_HotRod_KnownKeys___ key, which is obviously the problem.

Created attachment 837257 [details]
IOException: Unsupported protocol version 60 in ER6 still appears

WARNING about mixed dumped keys during migration is fixed and the whole process of REST rolling upgrades seem to be working.

But there is still the other WARN present in the server output.

See this attachment. 
Using ER6 and testsuite from internal GIT repo.

mvn clean verify -Psuite.examples -Dzip.dist=/home/tsykora/Downloads/62_JDG/ER6/jboss-datagrid-6.2.0.ER6-server.zip -Dtest=ExampleConfigsTest#testRestRollingUpgrades -DfailIfNoTests=false

Making this a blocker is a bit excessive.

This is likely not a blocker. If the upgrade process, including correct migration of data, runs successfully, this might be removed from the list of blockers. Further, this feature is not really useful at this point. It will be useful since JDG 6.3. However, let me add that it's a bit alarming that we haven't got this straightforward rolling upgrade process run smoothly, without any exceptions along the way. This should be achieved in next JDG release.

I have problem with migration from 6.2.0->6.2.1 via CLI.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<Stack trace>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    10:43:39,479 WARN  [org.infinispan.persistence.rest.upgrade.RestTargetMigrator] (pool-5-thread-4) ISPN000277: Could not migrate key rkey152: org.infinispan.persistence.spi.PersistenceException: ISPN022003: HTTP error
            at org.infinispan.persistence.rest.RestStore.load(RestStore.java:229) [infinispan-cachestore-rest-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.persistence.manager.PersistenceManagerImpl.loadFromAllStores(PersistenceManagerImpl.java:432) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:263) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeededAndUpdateStats(CacheLoaderInterceptor.java:343) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:105) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:116) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetKeyValueCommand(NonTransactionalLockingInterceptor.java:32) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitGetKeyValueCommand(BaseTypeConverterInterceptor.java:75) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.CacheImpl.get(CacheImpl.java:381) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.CacheImpl.get(CacheImpl.java:373) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.persistence.rest.upgrade.RestTargetMigrator$1.run(RestTargetMigrator.java:55) [infinispan-cachestore-rest-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
            at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
            at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
    Caused by: java.io.IOException: Unsupported protocol version 60
            at org.jboss.marshalling.river.RiverUnmarshaller.start(RiverUnmarshaller.java:1237)
            at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.startObjectInput(AbstractJBossMarshaller.java:130) [infinispan-commons-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.marshall.core.VersionAwareMarshaller.startObjectInput(VersionAwareMarshaller.java:142) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:98) [infinispan-core-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commons.marshall.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:82) [infinispan-commons-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:75) [infinispan-commons-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.persistence.rest.RestStore.unmarshall(RestStore.java:150) [infinispan-cachestore-rest-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            at org.infinispan.persistence.rest.RestStore.load(RestStore.java:222) [infinispan-cachestore-rest-6.0.3.Final-redhat-1.jar:6.0.3.Final-redhat-1]
            ... 35 more


<<<<<<<<<<<<<<<<<<<<<<<<<<<<Last line shows me>>>>>>>>>>>>>>>>>>>>>>>>>>>
10:43:41,288 INFO  [org.infinispan.upgrade.RollingUpgradeManager] (pool-1-thread-1) ISPN000216: 0 entries migrated to cache default in 4.69 seconds
10:43:41,293 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-1-thread-1) ISPN000031: MBeans were successfully registered to the platform MBean server.
10:43:41,294 INFO  [org.jboss.as.clustering.infinispan] (pool-1-thread-1) JBAS010281: Started ___defaultcache cache from local container
 
So that means that no entries were migrated actually

This is caused by a bug in the Rest codec of the cli which doesn't understand how to create proper mimemetadata for entries (it still thinks it needs to deal with MIMECacheEntry). I think the severity of this can be lowered.

This product has been discontinued or is no longer tracked in Red Hat Bugzilla.