Description of problem:
The current way of rolling upgrade is unable to handle a cache which has a key set larger than 2 GB.  It seems the limit of byte array size of Integer.MAX_VALUE.  The keys can't be dumped into a single byte array on the source cluster side.  And also it can't be synced by a single Hot Rod payload on the target cluster side.  What actually happen is a hang in recordKnownGlobalKeyset().  This size of keys is common for a large cluster these days.

~~~
"management-handler-thread - 2" prio=10 tid=0x00007f429c013800 nid=0x209cc9 runnable [0x00007f4398812000]
   java.lang.Thread.State: RUNNABLE
        at org.infinispan.commons.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:71)
        at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:108)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:295)
        at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
        at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
        at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
        at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:342)
        at java.util.HashSet.writeObject(HashSet.java:284)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:73)
        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToBuffer(AbstractJBossMarshaller.java:81)
        at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:70)
        at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:60)
        at org.infinispan.server.hotrod.HotRodSourceMigrator.recordKnownGlobalKeyset(HotRodSourceMigrator.scala:38)
        at org.infinispan.upgrade.RollingUpgradeManager.recordKnownGlobalKeyset(RollingUpgradeManager.java:50)
        at org.infinispan.cli.interpreter.statement.UpgradeStatement.execute(UpgradeStatement.java:78)
        at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:153)
        at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238)
        at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235)
        at org.infinispan.security.Security.doPrivileged(Security.java:89)
        at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56)
        at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241)
        at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702)
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537)
        at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:219)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:146)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:168)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:164)
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
~~~


Version-Release number of selected component (if applicable):
JDG 6.5.1 client server mode.


How reproducible:
Always.


Steps to Reproduce:
1. Configure RemoteCacheStore in the target cluster.
2. Fill the source cluster with keys larger than 2 GB in the total size.
3. Try to dump the keys in the source cluster.


Actual results:
recordKnownGlobalKeyset() hangs as in the above stack.


Expected results:
No hangs, no exceptions with keys larger than 2 GB.


Additional info:
http://infinispan.org/docs/7.2.x/user_guide/user_guide.html#_rolling_upgrades_for_infinispan_servers