Bug 1293575
| Summary: | Rolling upgrade fails with keySet larger than 2 GB | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Data Grid 6 | Reporter: | Osamu Nagano <onagano> |
| Component: | Infinispan | Assignee: | Tristan Tarrant <ttarrant> |
| Status: | VERIFIED --- | QA Contact: | Martin Gencur <mgencur> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.5.1 | CC: | chuffman, jdg-bugs |
| Target Milestone: | ER1 | ||
| Target Release: | 6.6.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Rolling upgrades fail if the key set is larger than 2 GB of memory. The process fails when calling <methodname>recordKnownGlobalKeyset</methodname> because the keys cannot be dumped into a single byte array in the source cluster.
This is a known issue in Red Hat JBoss Data Grid 6.6.0, and no workaround exists at this time.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1309749 | ||
Description of problem: The current way of rolling upgrade is unable to handle a cache which has a key set larger than 2 GB. It seems the limit of byte array size of Integer.MAX_VALUE. The keys can't be dumped into a single byte array on the source cluster side. And also it can't be synced by a single Hot Rod payload on the target cluster side. What actually happen is a hang in recordKnownGlobalKeyset(). This size of keys is common for a large cluster these days. ~~~ "management-handler-thread - 2" prio=10 tid=0x00007f429c013800 nid=0x209cc9 runnable [0x00007f4398812000] java.lang.Thread.State: RUNNABLE at org.infinispan.commons.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:71) at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:108) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:295) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:342) at java.util.HashSet.writeObject(HashSet.java:284) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:73) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToBuffer(AbstractJBossMarshaller.java:81) at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:70) at org.infinispan.commons.marshall.AbstractMarshaller.objectToByteBuffer(AbstractMarshaller.java:60) at org.infinispan.server.hotrod.HotRodSourceMigrator.recordKnownGlobalKeyset(HotRodSourceMigrator.scala:38) at org.infinispan.upgrade.RollingUpgradeManager.recordKnownGlobalKeyset(RollingUpgradeManager.java:50) at org.infinispan.cli.interpreter.statement.UpgradeStatement.execute(UpgradeStatement.java:78) at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:153) at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) at org.infinispan.security.Security.doPrivileged(Security.java:89) at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:219) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:146) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:168) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:164) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) at org.jboss.threads.JBossThread.run(JBossThread.java:122) ~~~ Version-Release number of selected component (if applicable): JDG 6.5.1 client server mode. How reproducible: Always. Steps to Reproduce: 1. Configure RemoteCacheStore in the target cluster. 2. Fill the source cluster with keys larger than 2 GB in the total size. 3. Try to dump the keys in the source cluster. Actual results: recordKnownGlobalKeyset() hangs as in the above stack. Expected results: No hangs, no exceptions with keys larger than 2 GB. Additional info: http://infinispan.org/docs/7.2.x/user_guide/user_guide.html#_rolling_upgrades_for_infinispan_servers