Bug 986307 - HotRod Rolling Upgrades - target can't obtain formerly stored data from RCS
HotRod Rolling Upgrades - target can't obtain formerly stored data from RCS
Status: VERIFIED
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high
: ER2
: 6.2.0
Assigned To: Tristan Tarrant
Martin Gencur
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-19 07:37 EDT by Tomas Sykora
Modified: 2013-10-08 04:55 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-3183 Critical Resolved HotRod RollUps from 5.2 to 5.3 -- target can't obtain formerly stored data from RCS 2017-04-11 00:49 EDT

  None (edit)
Description Tomas Sykora 2013-07-19 07:37:35 EDT
Please see this JIRA: https://issues.jboss.org/browse/ISPN-3183

Issue is the same for trying RollingUpgrades on 2 (JDG-6.2.0-DR1) servers. (One old - source node and the second new, target node)

Trace logs are attached in the aforementioned JIRA.
Comment 2 JBoss JIRA Server 2013-08-01 03:24:10 EDT
Tomas Sykora <tsykora@redhat.com> made a comment on jira ISPN-3183

I've tried to remove checking for accessing old data from new node (old data = data stored into remote cache store even before starting new node which connects to it) to find out what will happen then.

RecordKnownGlobalKeyset operation issued on source node *looks* ok but I'm really not sure.

Then test will try to perform synchronizeData on target node with this result:

testRollingUpgrades(org.infinispan.test.rollingupdates.IspnRollingUpdatesTest)  Time elapsed: 36.699 sec  <<< ERROR!
javax.management.MBeanException
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
        at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269)
        ... 9 more
Caused by: org.infinispan.commons.CacheException: ISPN020004: Could not find migration data in cache default
        at org.infinispan.upgrade.hotrod.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:94)
        at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59)
        ... 14 more

Maybe this can help a little bit.
Comment 3 JBoss JIRA Server 2013-08-01 03:29:48 EDT
Tomas Sykora <tsykora@redhat.com> made a comment on jira ISPN-3183

I've tried to remove checking for accessing old data from new node (old data = data stored into remote cache store even before starting new node which connects to it) to find out what will happen then.

RecordKnownGlobalKeyset operation issued on source node *looks* ok but I'm really not sure because of exception below.

Then test will try to perform synchronizeData on target node with this result:

testRollingUpgrades(org.infinispan.test.rollingupdates.IspnRollingUpdatesTest)  Time elapsed: 36.699 sec  <<< ERROR!
javax.management.MBeanException
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
        at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058)
        at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269)
        ... 9 more
Caused by: org.infinispan.commons.CacheException: ISPN020004: Could not find migration data in cache default
        at org.infinispan.upgrade.hotrod.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:94)
        at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59)
        ... 14 more

Maybe this can help a little bit.
Comment 4 JBoss JIRA Server 2013-09-24 02:26:06 EDT
Tomas Sykora <tsykora@redhat.com> made a comment on jira ISPN-3183

Not absolutely sure whether this is the proper way of testing this... but I consider this scenario should work
2 JDG 6.1.GA as source cluster, 2 latest ispn-servers with this PR as a target cluster:

08:17:19,579 WARN  [org.jgroups.protocols.UDP] (multicast receiver,shared=udp) [JGRP00010] packe08:17:19,579 WARN  [org.jgroups.protocols.UDP] (multicast receiver,shared=udp) [JGRP00010] packet from 192.168.2.103:45688 has different version (3.4.0) than ours (3.3.4); packet is discarded (received 13 identical messages from 192.168.2.103:45688 in the last 66549 ms)
t from 192.168.2.103:45688 has different version (3.4.0) than ours (3.3.4); packet is discarded (received 13 identical messages from 192.168.2.103:45688 in the last 66549 ms)

Looks like we need to deal somehow with different version of jgroups as well. What do you thing? Any simple workaround for that?

Thanks!
Comment 5 JBoss JIRA Server 2013-09-24 05:35:25 EDT
Tomas Sykora <tsykora@redhat.com> made a comment on jira ISPN-3183

Tristan's PR should fix this problem.

I got it work and successfully migrated data using hotrod migrator from jdg 6.1 (ispn schema 5.2) to the latest ispn server (ispn schema 6.0).

Nice work!
Thank you
Comment 6 Martin Gencur 2013-10-07 11:46:27 EDT
Shouldn't this be ON_QA?
Comment 7 Tomas Sykora 2013-10-08 04:55:03 EDT
I'm pretty sure that this should be ON_QA.

Setting ON_QA + target milestone back to ER2, and immediately setting as VERIFIED (in for 6.2 ER2 build) because this seems to be no longer an issue since this build.

Note You need to log in before you can comment on or make changes to this bug.