### Description of problem: ClassCastException happened at cache.getWithMetadata(key) in HotRod client during JDG rolling upgrade ~~~ WARN: ISPN004005: Error received from the server: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[5] returned server error (status=0x85): java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336) at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126) at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112) at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56) at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.sendKeyOperation(AbstractKeyOperation.java:53) at org.infinispan.client.hotrod.impl.operations.GetWithMetadataOperation.executeOperation(GetWithMetadataOperation.java:35) at org.infinispan.client.hotrod.impl.operations.GetWithMetadataOperation.executeOperation(GetWithMetadataOperation.java:23) at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52) at org.infinispan.client.hotrod.impl.RemoteCacheImpl.getWithMetadata(RemoteCacheImpl.java:166) ...(snip)... ~~~ ### Version-Release number of selected component (if applicable): * Target cluster: JDG 6.6.0 * Source cluster: JDG 6.5.1 ### How reproducible: You will see ClassCastException when you follow steps to Reproduce. ### Steps to Reproduce: 1. Put some data on source cluster 2. Configure rolling upgrade (<remote-store> setting) and start target cluster 3. Call "cache.getWithMetadata(key)" against target cluster Note: - ClassCastException does not happen if "cache.get(key)" is called before "cache.getWithMetadata(key)". - ClassCastException does not happen after completing "upgrade --dumpkeys" (or recordKnownGlobalKeyset() in JMX)on source cluster and "upgrade --synchronize=hotrod" (or synchronizeData() in JMX) on target cluster. ### Actual results: HotRod client failed at "cache.getWithMetadata(key)" with ClassCastException ### Expected results: HotRod client successfully obtains the result from "cache.getWithMetadata(key)" without ClassCastException ### Additional info: ClassCastException happened at infinispan-6.4.0.Final-redhat-4-src/server/hotrod/src/main/scala/org/infinispan/server/hotrod/Decoder2x.scala: ~~~ 422 def getKeyMetadata(h: HotRodHeader, k: Array[Byte], cache: Cache): GetWithMetadataResponse = { 423 val ce = cache.getCacheEntry(k) 424 if (ce != null) { 425 val ice = ce.asInstanceOf[InternalCacheEntry] ~~~ You can also see ClassCastException on JDG Server when you enable DEBUG level of "org.infinispan.server.hotrod.CacheDecodeContext" on JDG target cluster with RemoteStore: ~~~ DEBUG [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-1) Exception caught: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org. infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:417) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:141) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32) [infinispan-server-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:341) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:327) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31] Caused by: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at org.infinispan.server.hotrod.CacheDecodeContext.createServerException(CacheDecodeContext.scala:96) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:65) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1] ... 14 more Caused by: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry at org.infinispan.server.hotrod.Decoder2x$.getKeyMetadata(Decoder2x.scala:425) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.Decoder2x$.customReadKey(Decoder2x.scala:353) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:207) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:114) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:52) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:219) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:49) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4] ... 15 more ~~~
It seems that the ClassCastException happens for the very first getWithMetadata() for every key. ISPN-3330 and ISPN-4583 report similar error messages of "can not be cast to org.infinispan.container.entries.InternalCacheEntry" but their fixes are included in JDG 6.6.0 already.
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.