Bug 1177228

Summary: JDG 6.4.0.ER8: Topology hint breaks cross site replication in CacheTopologyControlCommand
Product: [JBoss] JBoss Data Grid 6 Reporter: Osamu Nagano <onagano>
Component: InfinispanAssignee: Pedro Ruivo <pruivo>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: bban, dmehra, dstahl, jdg-bugs, jpallich, ksuzumur, mcimbora, tkimura
Target Milestone: CR1   
Target Release: 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1153111    
Attachments:
Description Flags
clustered.xml for site A none

Description Osamu Nagano 2014-12-25 09:08:19 UTC
Description of problem:
When topology hints like site, rack and machine are set under cross site replication configuration, server will keep generating the following WARN log messages.


Version-Release number of selected component (if applicable):
JDG 6.4.0.ER8 (Infinispan 6.2.0.ER8)


How reproducible:
Always


Steps to Reproduce:
1. Just start a server with the attached xml file.


Actual results:
~~~
17:51:03,037 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (MSC service thread 1-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=namedCache, type=JOIN, sender=svr01/xSite-A ( flags=2 (can_be_sm), site-id=s1, rack-id=r1, machine-id=m1), joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@f0d, hashFunction=MurmurHash3, numSegments=80, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=0}: java.lang.ClassCastException: org.infinispan.remoting.transport.jgroups.JGroupsAddress cannot be cast to org.infinispan.remoting.transport.TopologyAwareAddress
        at org.infinispan.distribution.topologyaware.TopologyInfo.addTopology(TopologyInfo.java:56) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.topologyaware.TopologyInfo.<init>(TopologyInfo.java:37) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:27) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:126) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:41) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:25) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.ClusterCacheStatus.createInitialCacheTopology(ClusterCacheStatus.java:567) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.ClusterCacheStatus.doJoin(ClusterCacheStatus.java:551) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:131) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:377) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:103) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:109) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) [infinispan-commons-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.CacheImpl.start(CacheImpl.java:756) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:581) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:536) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:414) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:428) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.security.Security.doPrivileged(Security.java:76) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71]
~~~


Expected results:
No WARN logs and stack traces.


Additional info:
If the topology hints are removed from the blow, no issues happen.
~~~
                <transport type="UDP" socket-binding="jgroups-udp"
                           site="${jboss.jgroups.transport.site:s1}"
                           rack="${jboss.jgroups.transport.rack:r1}"
                           machine="${jboss.jgroups.transport.machine:m1}"/>
~~~

Comment 3 Osamu Nagano 2014-12-25 09:11:37 UTC
Created attachment 972927 [details]
clustered.xml for site A

Comment 4 Matej ─îimbora 2015-01-05 12:25:37 UTC
The issue seems to be JGroups-related and is caused by setting relay.enable_address_tagging property to true. In that case, RELAY2 protocol adds a new address generator, which influences resulting address (ExtendedUUID instead of TopologyUUID). JGroupsTransport.fromJGroupsAddress then returns wrong address wrapper class (JGroupsAddress instead of JGroupsTopologyAwareAddress), resulting in aforementioned ClassCastException.

Possible workaround (which however ignores can_become_site_master=false setting, if used):
Replace <property name="enable_address_tagging">true</property> with <property name="enable_address_tagging">false</property>.