Description of problem: I see this issue in EAP6.1, 6.2 and 6.3 when trying to connect with a standalone cache to a EAP domain cluster. This is the logging on the client side. The client pom references: <dependency> <groupId>org.infinispan</groupId> <artifactId>infinispan-core</artifactId> <version>5.2.10.Final-redhat-1</version> </dependency> which is the same version as used in EAP6.3. 15:43:11,762 INFO ISPN000078: Starting JGroups Channel ------------------------------------------------------------------- GMS: address=test-server-35385, cluster=jboss-as7-quickstart, physical address=127.0.0.1:8350 ------------------------------------------------------------------- 15:43:13,387 INFO ISPN000094: Received new cluster view: [master:server-three/jboss-as7-quickstart|2] [master:server-three/jboss-as7-quickstart, master:server-four/jboss-as7-quickstart, test-server-35385] 15:43:13,471 INFO ISPN000079: Cache local address is test-server-35385, physical addresses are [127.0.0.1:8350] 15:43:13,477 INFO ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final 15:43:13,770 INFO ISPN000031: MBeans were successfully registered to the platform MBean server. 15:43:17,678 WARN Problems unmarshalling remote command from byte buffer java.lang.IllegalArgumentException: Segment owner list cannot be null or empty at org.infinispan.distribution.ch.DefaultConsistentHash.<init>(DefaultConsistentHash.java:78) at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:299) at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:279) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:406) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:226) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:167) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:130) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:113) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:60) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:406) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:226) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:167) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163) at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:121) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:215) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) at org.jgroups.JChannel.up(JChannel.java:707) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) at org.jgroups.protocols.RSVP.up(RSVP.java:164) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Version-Release number of selected component (if applicable): How reproducible: To reproduce follow the instructions for https://github.com/infinispan/infinispan-quickstart/tree/master/jboss-as7 Note that in my setup I'm using tcp; fragment from the domain.xml: <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcpping"> <stack name="tcpping"> <transport type="TCP" socket-binding="jgroups-tcp"/> <protocol type="TCPPING"> <property name="initial_hosts"> 127.0.0.1[7850],127.0.0.1[7950],127.0.0.1[8350] </property> <property name="num_initial_members"> 2 </property> <property name="port_range"> 0 </property> <property name="timeout"> 2000 </property> </protocol> <protocol type="MERGE2"/> <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/> <protocol type="FD"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK"/> <protocol type="UNICAST2"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="UFC"/> <protocol type="MFC"/> <protocol type="FRAG2"/> <protocol type="RSVP"/> </stack> On the standalone side I'm using the following jgroups setup: <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd"> <TCP bind_port="8350" bind_addr="localhost" /> <TCPPING timeout="3000" initial_hosts="${jgroups.tcpping.initial_hosts:127.0.0.1[7850],127.0.0.1[7950],127.0.0.1[8050],127.0.0.1[8350]}" port_range="0" num_initial_members="2" /> <MERGE2 /> <FD_SOCK/> <FD_ALL/> <VERIFY_SUSPECT /> <pbcast.NAKACK2 /> <UNICAST2 /> <pbcast.STABLE/> <pbcast.GMS /> <MFC/> <FRAG2 /> <RSVP/> </config> and infinispan_cache.xml <?xml version="1.0" encoding="UTF-8"?> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:5.2 http://www.infinispan.org/schemas/infinispan-config-5.2.xsd" xmlns="urn:infinispan:config:5.2"> <global> <transport clusterName="jboss-as7-quickstart" distributedSyncTimeout="60000" nodeName="test-server" machineId="test1" > <properties> <property name="configurationFile" value="jgroups-tcpping.xml" /> </properties> </transport> <globalJmxStatistics enabled="true" jmxDomain="standalone_domain" allowDuplicateDomains="true"/> </global> <namedCache name="jboss-as7-quickstart-cache" > <clustering mode="distribution" > <sync /> </clustering> <jmxStatistics enabled="true"/> <transaction transactionMode="NON_TRANSACTIONAL"/> <storeAsBinary enabled="false"/> </namedCache> </infinispan> --Kurt Actual results: Expected results: Note that this works ok using wildfly. Wildfly uses ISPN 6 while EAP is using ISPN 5.2.x Additional info: Just let me know f you need anything else to debug this. I can send you the unittest if needed.
Please post your infinispan subsystem configuration from domain.xml. This is almost certainly an issue of mismatched infinispan configuration between your EAP instances and your standalone cache.
My guess is that your EAP cache is replicated while your standalone cache is distributed. That would definitely account for the null/empty segment list.
Hi Paul, here you go (note that the same configuration works just fine on wildfly) <cache-container name="jboss-as7-quickstart" default-cache="jboss-as7-quickstart-cache"> <transport lock-timeout="60000"/> <distributed-cache name="jboss-as7-quickstart-cache" owners="1" mode="SYNC" start="EAGER"/> </cache-container>
As I suspected, your configuration doesn't match. You have owners="1" in your EAP configuration, but your Infinispan configuration uses the default numOwners value, which is 2.
I changed the owners to "2" in EAP-6.2 and am getting the same issue: 10:34:13,775 INFO TreeCRM Cache lookup in JNDI failed - falling back on standalone infinispan and jgroups config files 10:34:15,221 INFO ISPN000078: Starting JGroups Channel 10:34:15,796 WARN use_mcast_xmit should not be used because the transport (TCP) does not support IP multicasting; setting use_mcast_xmit to false ------------------------------------------------------------------- GMS: address=ispn-standalone-server-63932, cluster=jboss-as7-quickstart, physical address=127.0.0.1:8350 ------------------------------------------------------------------- 10:34:15,896 INFO ISPN000094: Received new cluster view: [master:server-four/jboss-as7-quickstart|4] [master:server-four/jboss-as7-quickstart, master:server-three/jboss-as7-quickstart, ispn-standalone-server-63932] 10:34:15,953 INFO ISPN000079: Cache local address is ispn-standalone-server-63932, physical addresses are [127.0.0.1:8350] 10:34:15,958 INFO ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final 10:34:16,155 INFO ISPN000031: MBeans were successfully registered to the platform MBean server. 10:34:16,237 ERROR failed unmarshalling buffer into return value java.io.StreamCorruptedException: Unexpected lead byte 62 at org.jboss.marshalling.river.BlockUnmarshaller.badLeadByte(BlockUnmarshaller.java:131) at org.jboss.marshalling.river.BlockUnmarshaller.readBlockHeader(BlockUnmarshaller.java:100) at org.jboss.marshalling.river.BlockUnmarshaller.readUnsignedByte(BlockUnmarshaller.java:262) at org.jboss.marshalling.river.BlockUnmarshaller.readInt(BlockUnmarshaller.java:300) at org.infinispan.util.ImmutableListCopy.readExternal(ImmutableListCopy.java:243) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:180) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObjectArray(RiverUnmarshaller.java:1503) at org.jboss.marshalling.river.RiverUnmarshaller.doReadArray(RiverUnmarshaller.java:1542) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:326) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:297) at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:279) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:225) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.topology.CacheTopology$Externalizer.readObject(CacheTopology.java:145) at org.infinispan.topology.CacheTopology$Externalizer.readObject(CacheTopology.java:134) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:225) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:101) at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:86) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:225) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163) at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:116) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:404) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) at org.jgroups.JChannel.up(JChannel.java:707) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) at org.jgroups.protocols.RSVP.up(RSVP.java:188) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: an exception which occurred: in object of type org.infinispan.util.ImmutableListCopy in element at index [1] of size [80]
Actually the above error is different, and it goes away when using owners="1".
Sanne Grinovero <sanne+jbossorg> updated the status of jira ISPN-3783 to Resolved
Sanne Grinovero <sanne+jbossorg> updated the status of jira ISPN-3783 to Closed