Bug 1324030
| Summary: | ClassCastException at cache.getWithMetadata(key) in HotRod client during JDG rolling upgrade | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Data Grid 6 | Reporter: | Masafumi Miura <mmiura> |
| Component: | Server | Assignee: | Tristan Tarrant <ttarrant> |
| Status: | CLOSED UPSTREAM | QA Contact: | Martin Gencur <mgencur> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.6.0 | CC: | jdg-bugs, onagano, vjuranek, wfink |
| Target Milestone: | ER1 | ||
| Target Release: | 6.6.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2025-02-10 03:48:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1309749, 1328307 | ||
It seems that the ClassCastException happens for the very first getWithMetadata() for every key. ISPN-3330 and ISPN-4583 report similar error messages of "can not be cast to org.infinispan.container.entries.InternalCacheEntry" but their fixes are included in JDG 6.6.0 already. This product has been discontinued or is no longer tracked in Red Hat Bugzilla. |
### Description of problem: ClassCastException happened at cache.getWithMetadata(key) in HotRod client during JDG rolling upgrade ~~~ WARN: ISPN004005: Error received from the server: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[5] returned server error (status=0x85): java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336) at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126) at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112) at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56) at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.sendKeyOperation(AbstractKeyOperation.java:53) at org.infinispan.client.hotrod.impl.operations.GetWithMetadataOperation.executeOperation(GetWithMetadataOperation.java:35) at org.infinispan.client.hotrod.impl.operations.GetWithMetadataOperation.executeOperation(GetWithMetadataOperation.java:23) at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52) at org.infinispan.client.hotrod.impl.RemoteCacheImpl.getWithMetadata(RemoteCacheImpl.java:166) ...(snip)... ~~~ ### Version-Release number of selected component (if applicable): * Target cluster: JDG 6.6.0 * Source cluster: JDG 6.5.1 ### How reproducible: You will see ClassCastException when you follow steps to Reproduce. ### Steps to Reproduce: 1. Put some data on source cluster 2. Configure rolling upgrade (<remote-store> setting) and start target cluster 3. Call "cache.getWithMetadata(key)" against target cluster Note: - ClassCastException does not happen if "cache.get(key)" is called before "cache.getWithMetadata(key)". - ClassCastException does not happen after completing "upgrade --dumpkeys" (or recordKnownGlobalKeyset() in JMX）on source cluster and "upgrade --synchronize=hotrod" (or synchronizeData() in JMX) on target cluster. ### Actual results: HotRod client failed at "cache.getWithMetadata(key)" with ClassCastException ### Expected results: HotRod client successfully obtains the result from "cache.getWithMetadata(key)" without ClassCastException ### Additional info: ClassCastException happened at infinispan-6.4.0.Final-redhat-4-src/server/hotrod/src/main/scala/org/infinispan/server/hotrod/Decoder2x.scala: ~~~ 422 def getKeyMetadata(h: HotRodHeader, k: Array[Byte], cache: Cache): GetWithMetadataResponse = { 423 val ce = cache.getCacheEntry(k) 424 if (ce != null) { 425 val ice = ce.asInstanceOf[InternalCacheEntry] ~~~ You can also see ClassCastException on JDG Server when you enable DEBUG level of "org.infinispan.server.hotrod.CacheDecodeContext" on JDG target cluster with RemoteStore: ~~~ DEBUG [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-1) Exception caught: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org. infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:417) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:141) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32) [infinispan-server-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:341) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:327) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31] Caused by: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at org.infinispan.server.hotrod.CacheDecodeContext.createServerException(CacheDecodeContext.scala:96) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:65) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] ... 14 more Caused by: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at org.infinispan.server.hotrod.Decoder2x$.getKeyMetadata(Decoder2x.scala:425) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.Decoder2x$.customReadKey(Decoder2x.scala:353) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:207) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:114) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:52) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:219) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:49) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] ... 15 more ~~~