Bug 1293575 - Rolling upgrade fails with keySet larger than 2 GB
Rolling upgrade fails with keySet larger than 2 GB
Status: VERIFIED
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.5.1
Unspecified Unspecified
unspecified Severity high
: ER1
: 6.6.1
Assigned To: Gustavo Fernandes
Martin Gencur
:
Depends On:
Blocks: 1309749
  Show dependency treegraph
 
Reported: 2015-12-22 03:59 EST by Osamu Nagano
Modified: 2016-08-18 05:58 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Rolling upgrades fail if the key set is larger than 2 GB of memory. The process fails when calling <methodname>recordKnownGlobalKeyset</methodname> because the keys cannot be dumped into a single byte array in the source cluster. This is a known issue in Red Hat JBoss Data Grid 6.6.0, and no workaround exists at this time.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-6046 Major Resolved Server Rolling Upgrade performance improvement 2017-04-12 02:58 EDT

  None (edit)
Description Osamu Nagano 2015-12-22 03:59:34 EST
Description of problem:
The current way of rolling upgrade is unable to handle a cache which has a key set larger than 2 GB.  It seems the limit of byte array size of Integer.MAX_VALUE.  The keys can't be dumped into a single byte array on the source cluster side.  And also it can't be synced by a single Hot Rod payload on the target cluster side.  What actually happen is a hang in recordKnownGlobalKeyset().  This size of keys is common for a large cluster these days.

~~~
"management-handler-thread - 2" prio=10 tid=0x00007f429c013800 nid=0x209cc9 runnable [0x00007f4398812000]
   java.lang.Thread.State: RUNNABLE
        at org.infinispan.commons.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:71)
        at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:108)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:295)
        at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
        at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
        at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
        at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:342)
        at java.util.HashSet.writeObject(HashSet.java:284)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:73)
        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToBuffer(AbstractJBossMarshaller.java:81)
        at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:70)
        at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:60)
        at org.infinispan.server.hotrod.HotRodSourceMigrator.recordKnownGlobalKeyset(HotRodSourceMigrator.scala:38)
        at org.infinispan.upgrade.RollingUpgradeManager.recordKnownGlobalKeyset(RollingUpgradeManager.java:50)
        at org.infinispan.cli.interpreter.statement.UpgradeStatement.execute(UpgradeStatement.java:78)
        at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:153)
        at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238)
        at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235)
        at org.infinispan.security.Security.doPrivileged(Security.java:89)
        at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56)
        at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241)
        at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702)
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537)
        at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:219)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:146)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:168)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:164)
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
~~~


Version-Release number of selected component (if applicable):
JDG 6.5.1 client server mode.


How reproducible:
Always.


Steps to Reproduce:
1. Configure RemoteCacheStore in the target cluster.
2. Fill the source cluster with keys larger than 2 GB in the total size.
3. Try to dump the keys in the source cluster.


Actual results:
recordKnownGlobalKeyset() hangs as in the above stack.


Expected results:
No hangs, no exceptions with keys larger than 2 GB.


Additional info:
http://infinispan.org/docs/7.2.x/user_guide/user_guide.html#_rolling_upgrades_for_infinispan_servers

Note You need to log in before you can comment on or make changes to this bug.