Bug 1136864 - Segment owner list cannot be null or empty
Summary: Segment owner list cannot be null or empty
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering
Version: 6.1.0,6.2.0,6.1.1,6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Paul Ferraro
QA Contact: Jitka Kozana
Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-03 13:20 UTC by Kurt T Stam
Modified: 2016-01-25 17:19 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-06-19 08:52:57 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-3783 0 Major Closed JBOSS AS Infinispan nodes cannot interoperate with Plain JVM / Glassfish nodes 2016-11-02 15:00:34 UTC

Description Kurt T Stam 2014-09-03 13:20:17 UTC
Description of problem:

I see this issue in EAP6.1, 6.2 and 6.3 when trying to connect with a standalone cache to a EAP domain cluster. This is the logging on the client side. The client pom references:

<dependency>
      <groupId>org.infinispan</groupId>
      <artifactId>infinispan-core</artifactId>
      <version>5.2.10.Final-redhat-1</version>
    </dependency>

which is the same version as used in EAP6.3.

15:43:11,762  INFO ISPN000078: Starting JGroups Channel

-------------------------------------------------------------------
GMS: address=test-server-35385, cluster=jboss-as7-quickstart, physical address=127.0.0.1:8350
-------------------------------------------------------------------
15:43:13,387  INFO ISPN000094: Received new cluster view: [master:server-three/jboss-as7-quickstart|2] [master:server-three/jboss-as7-quickstart, master:server-four/jboss-as7-quickstart, test-server-35385]
15:43:13,471  INFO ISPN000079: Cache local address is test-server-35385, physical addresses are [127.0.0.1:8350]
15:43:13,477  INFO ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final
15:43:13,770  INFO ISPN000031: MBeans were successfully registered to the platform MBean server.
15:43:17,678  WARN Problems unmarshalling remote command from byte buffer
java.lang.IllegalArgumentException: Segment owner list cannot be null or empty
	at org.infinispan.distribution.ch.DefaultConsistentHash.<init>(DefaultConsistentHash.java:78)
	at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:299)
	at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:279)
	at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:406)
	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:226)
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:167)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:130)
	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:113)
	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:60)
	at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:406)
	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:226)
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:167)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
	at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:121)
	at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104)
	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:215)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
	at org.jgroups.JChannel.up(JChannel.java:707)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
	at org.jgroups.protocols.RSVP.up(RSVP.java:164)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453)
	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
	at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)



Version-Release number of selected component (if applicable):


How reproducible:

To reproduce follow the instructions for 

https://github.com/infinispan/infinispan-quickstart/tree/master/jboss-as7

Note that in my setup I'm using tcp; fragment from the domain.xml:

  <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcpping">
                 <stack name="tcpping">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="TCPPING">
                        <property name="initial_hosts">
                            127.0.0.1[7850],127.0.0.1[7950],127.0.0.1[8350]
                        </property>
                        <property name="num_initial_members">
                            2
                        </property>
                        <property name="port_range">
                            0
                        </property>
                        <property name="timeout">
                            2000
                        </property>
                    </protocol>
                    <protocol type="MERGE2"/>
                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                    <protocol type="FD"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK"/>
                    <protocol type="UNICAST2"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG2"/>
                    <protocol type="RSVP"/>
                </stack>

On the standalone side I'm using the following jgroups setup:

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:org:jgroups"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">

    <TCP bind_port="8350"
         bind_addr="localhost"
         />
         
    <TCPPING timeout="3000"
         initial_hosts="${jgroups.tcpping.initial_hosts:127.0.0.1[7850],127.0.0.1[7950],127.0.0.1[8050],127.0.0.1[8350]}"
         port_range="0"
         num_initial_members="2" /> 

    <MERGE2 />
    <FD_SOCK/>
    <FD_ALL/>
    <VERIFY_SUSPECT  />
    <pbcast.NAKACK2 />
    <UNICAST2 />
    <pbcast.STABLE/>
    <pbcast.GMS />
    <MFC/>
    <FRAG2 />
    <RSVP/>
</config>      

and infinispan_cache.xml

<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:5.2 http://www.infinispan.org/schemas/infinispan-config-5.2.xsd"
    xmlns="urn:infinispan:config:5.2">
 
    <global>
        <transport clusterName="jboss-as7-quickstart" distributedSyncTimeout="60000" nodeName="test-server" machineId="test1" >
            <properties>
                <property name="configurationFile" value="jgroups-tcpping.xml" />
            </properties>
        </transport>
        
        <globalJmxStatistics enabled="true" jmxDomain="standalone_domain"
                           
                           allowDuplicateDomains="true"/>
    </global>
   
    <namedCache name="jboss-as7-quickstart-cache"  >
        <clustering mode="distribution"  >
            <sync />
            
        </clustering>
        <jmxStatistics enabled="true"/>
        <transaction transactionMode="NON_TRANSACTIONAL"/>
        <storeAsBinary enabled="false"/>
    </namedCache>
 
</infinispan>
   


--Kurt


Actual results:


Expected results:

Note that this works ok using wildfly. Wildfly uses ISPN 6 while EAP is using ISPN 5.2.x

Additional info:

Just let me know f you need anything else to debug this. I can send you the unittest if needed.

Comment 1 Paul Ferraro 2014-09-04 01:24:13 UTC
Please post your infinispan subsystem configuration from domain.xml.  This is almost certainly an issue of mismatched infinispan configuration between your EAP instances and your standalone cache.

Comment 2 Paul Ferraro 2014-09-04 02:17:19 UTC
My guess is that your EAP cache is replicated while your standalone cache is distributed.  That would definitely account for the null/empty segment list.

Comment 3 Kurt T Stam 2014-09-05 18:30:01 UTC
Hi Paul, here you go (note that the same configuration works just fine on wildfly)

 <cache-container name="jboss-as7-quickstart" default-cache="jboss-as7-quickstart-cache">
   <transport lock-timeout="60000"/>
   <distributed-cache name="jboss-as7-quickstart-cache" owners="1" mode="SYNC" start="EAGER"/>
 </cache-container>

Comment 4 Paul Ferraro 2014-09-11 14:19:30 UTC
As I suspected, your configuration doesn't match.  You have owners="1" in your EAP configuration, but your Infinispan configuration uses the default numOwners value, which is 2.

Comment 5 Kurt T Stam 2014-09-17 14:42:52 UTC
I changed the owners to "2" in EAP-6.2 and am getting the same issue:

10:34:13,775  INFO TreeCRM Cache lookup in JNDI failed - falling back on standalone infinispan and jgroups config files
10:34:15,221  INFO ISPN000078: Starting JGroups Channel
10:34:15,796  WARN use_mcast_xmit should not be used because the transport (TCP) does not support IP multicasting; setting use_mcast_xmit to false

-------------------------------------------------------------------
GMS: address=ispn-standalone-server-63932, cluster=jboss-as7-quickstart, physical address=127.0.0.1:8350
-------------------------------------------------------------------
10:34:15,896  INFO ISPN000094: Received new cluster view: [master:server-four/jboss-as7-quickstart|4] [master:server-four/jboss-as7-quickstart, master:server-three/jboss-as7-quickstart, ispn-standalone-server-63932]
10:34:15,953  INFO ISPN000079: Cache local address is ispn-standalone-server-63932, physical addresses are [127.0.0.1:8350]
10:34:15,958  INFO ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final
10:34:16,155  INFO ISPN000031: MBeans were successfully registered to the platform MBean server.
10:34:16,237 ERROR failed unmarshalling buffer into return value
java.io.StreamCorruptedException: Unexpected lead byte 62
	at org.jboss.marshalling.river.BlockUnmarshaller.badLeadByte(BlockUnmarshaller.java:131)
	at org.jboss.marshalling.river.BlockUnmarshaller.readBlockHeader(BlockUnmarshaller.java:100)
	at org.jboss.marshalling.river.BlockUnmarshaller.readUnsignedByte(BlockUnmarshaller.java:262)
	at org.jboss.marshalling.river.BlockUnmarshaller.readInt(BlockUnmarshaller.java:300)
	at org.infinispan.util.ImmutableListCopy.readExternal(ImmutableListCopy.java:243)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:180)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObjectArray(RiverUnmarshaller.java:1503)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadArray(RiverUnmarshaller.java:1542)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:326)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
	at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:297)
	at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:279)
	at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404)
	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:225)
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
	at org.infinispan.topology.CacheTopology$Externalizer.readObject(CacheTopology.java:145)
	at org.infinispan.topology.CacheTopology$Externalizer.readObject(CacheTopology.java:134)
	at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404)
	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:225)
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
	at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:101)
	at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:86)
	at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404)
	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:225)
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
	at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:116)
	at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104)
	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:404)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
	at org.jgroups.JChannel.up(JChannel.java:707)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
	at org.jgroups.protocols.RSVP.up(RSVP.java:188)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765)
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420)
	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
	at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: an exception which occurred:
	in object of type org.infinispan.util.ImmutableListCopy
	in element at index [1] of size [80]

Comment 6 Kurt T Stam 2014-09-17 14:53:55 UTC
Actually the above error is different, and it goes away when using owners="1".

Comment 8 JBoss JIRA Server 2016-01-25 17:18:53 UTC
Sanne Grinovero <sanne+jbossorg> updated the status of jira ISPN-3783 to Resolved

Comment 9 JBoss JIRA Server 2016-01-25 17:19:01 UTC
Sanne Grinovero <sanne+jbossorg> updated the status of jira ISPN-3783 to Closed


Note You need to log in before you can comment on or make changes to this bug.