Please see this JIRA: https://issues.jboss.org/browse/ISPN-3183 Issue is the same for trying RollingUpgrades on 2 (JDG-6.2.0-DR1) servers. (One old - source node and the second new, target node) Trace logs are attached in the aforementioned JIRA.
Tomas Sykora <tsykora> made a comment on jira ISPN-3183 I've tried to remove checking for accessing old data from new node (old data = data stored into remote cache store even before starting new node which connects to it) to find out what will happen then. RecordKnownGlobalKeyset operation issued on source node *looks* ok but I'm really not sure. Then test will try to perform synchronizeData on target node with this result: testRollingUpgrades(org.infinispan.test.rollingupdates.IspnRollingUpdatesTest) Time elapsed: 36.699 sec <<< ERROR! javax.management.MBeanException at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527) at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263) at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058) at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269) ... 9 more Caused by: org.infinispan.commons.CacheException: ISPN020004: Could not find migration data in cache default at org.infinispan.upgrade.hotrod.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:94) at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59) ... 14 more Maybe this can help a little bit.
Tomas Sykora <tsykora> made a comment on jira ISPN-3183 I've tried to remove checking for accessing old data from new node (old data = data stored into remote cache store even before starting new node which connects to it) to find out what will happen then. RecordKnownGlobalKeyset operation issued on source node *looks* ok but I'm really not sure because of exception below. Then test will try to perform synchronizeData on target node with this result: testRollingUpgrades(org.infinispan.test.rollingupdates.IspnRollingUpdatesTest) Time elapsed: 36.699 sec <<< ERROR! javax.management.MBeanException at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527) at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263) at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058) at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269) ... 9 more Caused by: org.infinispan.commons.CacheException: ISPN020004: Could not find migration data in cache default at org.infinispan.upgrade.hotrod.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:94) at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59) ... 14 more Maybe this can help a little bit.
Tomas Sykora <tsykora> made a comment on jira ISPN-3183 Not absolutely sure whether this is the proper way of testing this... but I consider this scenario should work 2 JDG 6.1.GA as source cluster, 2 latest ispn-servers with this PR as a target cluster: 08:17:19,579 WARN [org.jgroups.protocols.UDP] (multicast receiver,shared=udp) [JGRP00010] packe08:17:19,579 WARN [org.jgroups.protocols.UDP] (multicast receiver,shared=udp) [JGRP00010] packet from 192.168.2.103:45688 has different version (3.4.0) than ours (3.3.4); packet is discarded (received 13 identical messages from 192.168.2.103:45688 in the last 66549 ms) t from 192.168.2.103:45688 has different version (3.4.0) than ours (3.3.4); packet is discarded (received 13 identical messages from 192.168.2.103:45688 in the last 66549 ms) Looks like we need to deal somehow with different version of jgroups as well. What do you thing? Any simple workaround for that? Thanks!
Tomas Sykora <tsykora> made a comment on jira ISPN-3183 Tristan's PR should fix this problem. I got it work and successfully migrated data using hotrod migrator from jdg 6.1 (ispn schema 5.2) to the latest ispn server (ispn schema 6.0). Nice work! Thank you
Shouldn't this be ON_QA?
I'm pretty sure that this should be ON_QA. Setting ON_QA + target milestone back to ER2, and immediately setting as VERIFIED (in for 6.2 ER2 build) because this seems to be no longer an issue since this build.