Bug 1324030 - ClassCastException at cache.getWithMetadata(key) in HotRod client during JDG rolling upgrade
Summary: ClassCastException at cache.getWithMetadata(key) in HotRod client during JDG ...
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Server
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ER1
: 6.6.1
Assignee: Tristan Tarrant
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks: 1309749 1328307
TreeView+ depends on / blocked
 
Reported: 2016-04-05 11:38 UTC by Masafumi Miura
Modified: 2022-01-27 04:15 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2302971 0 None None None 2016-05-09 10:25:54 UTC

Description Masafumi Miura 2016-04-05 11:38:45 UTC
### Description of problem:

ClassCastException happened at cache.getWithMetadata(key) in HotRod client during JDG rolling upgrade

~~~
WARN: ISPN004005: Error received from the server:
java.lang.ClassCastException:
org.infinispan.container.entries.ReadCommittedEntry cannot be cast to
org.infinispan.container.entries.InternalCacheEntry
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request
for message id[5] returned server error (status=0x85):
java.lang.ClassCastException:
org.infinispan.container.entries.ReadCommittedEntry cannot be cast to
org.infinispan.container.entries.InternalCacheEntry
    at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
    at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
    at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
    at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
    at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.sendKeyOperation(AbstractKeyOperation.java:53)
    at org.infinispan.client.hotrod.impl.operations.GetWithMetadataOperation.executeOperation(GetWithMetadataOperation.java:35)
    at org.infinispan.client.hotrod.impl.operations.GetWithMetadataOperation.executeOperation(GetWithMetadataOperation.java:23)
    at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
    at org.infinispan.client.hotrod.impl.RemoteCacheImpl.getWithMetadata(RemoteCacheImpl.java:166)
    ...(snip)...
~~~


### Version-Release number of selected component (if applicable):

  * Target cluster: JDG 6.6.0
  * Source cluster: JDG 6.5.1


### How reproducible:

You will see ClassCastException when you follow steps to Reproduce.


### Steps to Reproduce:

1. Put some data on source cluster
2. Configure rolling upgrade (<remote-store> setting) and start target cluster
3. Call "cache.getWithMetadata(key)" against target cluster

Note:
 - ClassCastException does not happen if "cache.get(key)" is called before "cache.getWithMetadata(key)".
 - ClassCastException does not happen after completing "upgrade --dumpkeys" (or recordKnownGlobalKeyset() in JMX)on source cluster and "upgrade --synchronize=hotrod" (or synchronizeData() in JMX) on target cluster.


### Actual results:

HotRod client failed at "cache.getWithMetadata(key)" with ClassCastException


### Expected results:

HotRod client successfully obtains the result from "cache.getWithMetadata(key)" without ClassCastException


### Additional info:

ClassCastException happened at infinispan-6.4.0.Final-redhat-4-src/server/hotrod/src/main/scala/org/infinispan/server/hotrod/Decoder2x.scala:

~~~
422    def getKeyMetadata(h: HotRodHeader, k: Array[Byte], cache: Cache): GetWithMetadataResponse = {
423       val ce = cache.getCacheEntry(k)
424       if (ce != null) {
425          val ice = ce.asInstanceOf[InternalCacheEntry]
~~~

You can also see ClassCastException on JDG Server when you enable DEBUG level of "org.infinispan.server.hotrod.CacheDecodeContext" on JDG target cluster with RemoteStore:

~~~
DEBUG [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-1) Exception caught: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org.
infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry
        at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:417) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:141) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32) [infinispan-server-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:32) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:341) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:327) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
Caused by: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry
        at org.infinispan.server.hotrod.CacheDecodeContext.createServerException(CacheDecodeContext.scala:96) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:65) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
        ... 14 more
Caused by: java.lang.ClassCastException: org.infinispan.container.entries.ReadCommittedEntry cannot be cast to org.infinispan.container.entries.InternalCacheEntry
        at org.infinispan.server.hotrod.Decoder2x$.getKeyMetadata(Decoder2x.scala:425) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.Decoder2x$.customReadKey(Decoder2x.scala:353) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:207) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:114) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:52) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:219) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:49) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
        ... 15 more
~~~

Comment 3 Osamu Nagano 2016-04-07 03:32:13 UTC
It seems that the ClassCastException happens for the very first getWithMetadata() for every key.  ISPN-3330 and ISPN-4583 report similar error messages of "can not be cast to org.infinispan.container.entries.InternalCacheEntry" but their fixes are included in JDG 6.6.0 already.


Note You need to log in before you can comment on or make changes to this bug.