Bug 986307 - HotRod Rolling Upgrades - target can't obtain formerly stored data from RCS
Summary: HotRod Rolling Upgrades - target can't obtain formerly stored data from RCS
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ER2
: 6.2.0
Assignee: Tristan Tarrant
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-19 11:37 UTC by Tomas Sykora
Modified: 2013-10-08 08:55 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-3183 0 Critical Resolved HotRod RollUps from 5.2 to 5.3 -- target can't obtain formerly stored data from RCS 2017-04-11 04:49:26 UTC

Description Tomas Sykora 2013-07-19 11:37:35 UTC
Please see this JIRA: https://issues.jboss.org/browse/ISPN-3183

Issue is the same for trying RollingUpgrades on 2 (JDG-6.2.0-DR1) servers. (One old - source node and the second new, target node)

Trace logs are attached in the aforementioned JIRA.

Comment 2 JBoss JIRA Server 2013-08-01 07:24:10 UTC
Tomas Sykora <tsykora> made a comment on jira ISPN-3183

I've tried to remove checking for accessing old data from new node (old data = data stored into remote cache store even before starting new node which connects to it) to find out what will happen then.

RecordKnownGlobalKeyset operation issued on source node *looks* ok but I'm really not sure.

Then test will try to perform synchronizeData on target node with this result:

testRollingUpgrades(org.infinispan.test.rollingupdates.IspnRollingUpdatesTest)  Time elapsed: 36.699 sec  <<< ERROR!
javax.management.MBeanException
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
        at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269)
        ... 9 more
Caused by: org.infinispan.commons.CacheException: ISPN020004: Could not find migration data in cache default
        at org.infinispan.upgrade.hotrod.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:94)
        at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59)
        ... 14 more

Maybe this can help a little bit.

Comment 3 JBoss JIRA Server 2013-08-01 07:29:48 UTC
Tomas Sykora <tsykora> made a comment on jira ISPN-3183

I've tried to remove checking for accessing old data from new node (old data = data stored into remote cache store even before starting new node which connects to it) to find out what will happen then.

RecordKnownGlobalKeyset operation issued on source node *looks* ok but I'm really not sure because of exception below.

Then test will try to perform synchronizeData on target node with this result:

testRollingUpgrades(org.infinispan.test.rollingupdates.IspnRollingUpdatesTest)  Time elapsed: 36.699 sec  <<< ERROR!
javax.management.MBeanException
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
        at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269)
        ... 9 more
Caused by: org.infinispan.commons.CacheException: ISPN020004: Could not find migration data in cache default
        at org.infinispan.upgrade.hotrod.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:94)
        at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59)
        ... 14 more

Maybe this can help a little bit.

Comment 4 JBoss JIRA Server 2013-09-24 06:26:06 UTC
Tomas Sykora <tsykora> made a comment on jira ISPN-3183

Not absolutely sure whether this is the proper way of testing this... but I consider this scenario should work
2 JDG 6.1.GA as source cluster, 2 latest ispn-servers with this PR as a target cluster:

08:17:19,579 WARN  [org.jgroups.protocols.UDP] (multicast receiver,shared=udp) [JGRP00010] packe08:17:19,579 WARN  [org.jgroups.protocols.UDP] (multicast receiver,shared=udp) [JGRP00010] packet from 192.168.2.103:45688 has different version (3.4.0) than ours (3.3.4); packet is discarded (received 13 identical messages from 192.168.2.103:45688 in the last 66549 ms)
t from 192.168.2.103:45688 has different version (3.4.0) than ours (3.3.4); packet is discarded (received 13 identical messages from 192.168.2.103:45688 in the last 66549 ms)

Looks like we need to deal somehow with different version of jgroups as well. What do you thing? Any simple workaround for that?

Thanks!

Comment 5 JBoss JIRA Server 2013-09-24 09:35:25 UTC
Tomas Sykora <tsykora> made a comment on jira ISPN-3183

Tristan's PR should fix this problem.

I got it work and successfully migrated data using hotrod migrator from jdg 6.1 (ispn schema 5.2) to the latest ispn server (ispn schema 6.0).

Nice work!
Thank you

Comment 6 Martin Gencur 2013-10-07 15:46:27 UTC
Shouldn't this be ON_QA?

Comment 7 Tomas Sykora 2013-10-08 08:55:03 UTC
I'm pretty sure that this should be ON_QA.

Setting ON_QA + target milestone back to ER2, and immediately setting as VERIFIED (in for 6.2 ER2 build) because this seems to be no longer an issue since this build.


Note You need to log in before you can comment on or make changes to this bug.