Bug 1177228 - JDG 6.4.0.ER8: Topology hint breaks cross site replication in CacheTopologyControlCommand
Summary: JDG 6.4.0.ER8: Topology hint breaks cross site replication in CacheTopologyCo...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: CR1
: 6.4.0
Assignee: Pedro Ruivo
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks: jdg64-GA-Blockers
TreeView+ depends on / blocked
 
Reported: 2014-12-25 09:08 UTC by Osamu Nagano
Modified: 2018-12-09 19:22 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)
clustered.xml for site A (14.58 KB, application/xml)
2014-12-25 09:11 UTC, Osamu Nagano
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-5109 Major New Topology hint breaks cross site replication in CacheTopologyControlCommand 2018-03-16 07:24:52 UTC

Description Osamu Nagano 2014-12-25 09:08:19 UTC
Description of problem:
When topology hints like site, rack and machine are set under cross site replication configuration, server will keep generating the following WARN log messages.


Version-Release number of selected component (if applicable):
JDG 6.4.0.ER8 (Infinispan 6.2.0.ER8)


How reproducible:
Always


Steps to Reproduce:
1. Just start a server with the attached xml file.


Actual results:
~~~
17:51:03,037 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (MSC service thread 1-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=namedCache, type=JOIN, sender=svr01/xSite-A ( flags=2 (can_be_sm), site-id=s1, rack-id=r1, machine-id=m1), joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@f0d, hashFunction=MurmurHash3, numSegments=80, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=0}: java.lang.ClassCastException: org.infinispan.remoting.transport.jgroups.JGroupsAddress cannot be cast to org.infinispan.remoting.transport.TopologyAwareAddress
        at org.infinispan.distribution.topologyaware.TopologyInfo.addTopology(TopologyInfo.java:56) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.topologyaware.TopologyInfo.<init>(TopologyInfo.java:37) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:27) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:126) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:41) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:25) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.ClusterCacheStatus.createInitialCacheTopology(ClusterCacheStatus.java:567) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.ClusterCacheStatus.doJoin(ClusterCacheStatus.java:551) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:131) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:377) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:103) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:109) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) [infinispan-commons-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.CacheImpl.start(CacheImpl.java:756) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:581) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:536) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:414) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:428) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.security.Security.doPrivileged(Security.java:76) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71]
~~~


Expected results:
No WARN logs and stack traces.


Additional info:
If the topology hints are removed from the blow, no issues happen.
~~~
                <transport type="UDP" socket-binding="jgroups-udp"
                           site="${jboss.jgroups.transport.site:s1}"
                           rack="${jboss.jgroups.transport.rack:r1}"
                           machine="${jboss.jgroups.transport.machine:m1}"/>
~~~

Comment 3 Osamu Nagano 2014-12-25 09:11:37 UTC
Created attachment 972927 [details]
clustered.xml for site A

Comment 4 Matej Čimbora 2015-01-05 12:25:37 UTC
The issue seems to be JGroups-related and is caused by setting relay.enable_address_tagging property to true. In that case, RELAY2 protocol adds a new address generator, which influences resulting address (ExtendedUUID instead of TopologyUUID). JGroupsTransport.fromJGroupsAddress then returns wrong address wrapper class (JGroupsAddress instead of JGroupsTopologyAwareAddress), resulting in aforementioned ClassCastException.

Possible workaround (which however ignores can_become_site_master=false setting, if used):
Replace <property name="enable_address_tagging">true</property> with <property name="enable_address_tagging">false</property>.


Note You need to log in before you can comment on or make changes to this bug.