Description of problem: The current way of rolling upgrade is unable to handle a cache which has a key set larger than 2 GB. It seems the limit of byte array size of Integer.MAX_VALUE. The keys can't be dumped into a single byte array on the source cluster side. And also it can't be synced by a single Hot Rod payload on the target cluster side. What actually happen is a hang in recordKnownGlobalKeyset(). This size of keys is common for a large cluster these days. ~~~ "management-handler-thread - 2" prio=10 tid=0x00007f429c013800 nid=0x209cc9 runnable [0x00007f4398812000] java.lang.Thread.State: RUNNABLE at org.infinispan.commons.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:71) at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:108) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:295) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:342) at java.util.HashSet.writeObject(HashSet.java:284) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:73) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToBuffer(AbstractJBossMarshaller.java:81) at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:70) at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:60) at org.infinispan.server.hotrod.HotRodSourceMigrator.recordKnownGlobalKeyset(HotRodSourceMigrator.scala:38) at org.infinispan.upgrade.RollingUpgradeManager.recordKnownGlobalKeyset(RollingUpgradeManager.java:50) at org.infinispan.cli.interpreter.statement.UpgradeStatement.execute(UpgradeStatement.java:78) at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:153) at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) at org.infinispan.security.Security.doPrivileged(Security.java:89) at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:219) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:146) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:168) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:164) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) at org.jboss.threads.JBossThread.run(JBossThread.java:122) ~~~ Version-Release number of selected component (if applicable): JDG 6.5.1 client server mode. How reproducible: Always. Steps to Reproduce: 1. Configure RemoteCacheStore in the target cluster. 2. Fill the source cluster with keys larger than 2 GB in the total size. 3. Try to dump the keys in the source cluster. Actual results: recordKnownGlobalKeyset() hangs as in the above stack. Expected results: No hangs, no exceptions with keys larger than 2 GB. Additional info: http://infinispan.org/docs/7.2.x/user_guide/user_guide.html#_rolling_upgrades_for_infinispan_servers