Bug 745850 (EDG-55)

Summary: call to unsupported operation while using hotrod client/server access
Product: [JBoss] JBoss Data Grid 5 Reporter: Michal Linhard <mlinhard>
Component: InfinispanAssignee: Default User <jbpapp-maint>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: EAP 5.1.0 EDG TPCC: galder.zamarreno, mlinhard, nobody, rhusar
Target Milestone: ---   
Target Release: EAP 5.1.0 EDG TP   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/EDG-55
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-07 10:06:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 745860    

Description Michal Linhard 2011-02-28 09:35:13 UTC
project_key: EDG

infinispan version: 4.2.1.CR3

This appeared in the server log while running performance test (client stress test on 6 node cluster):

 java.lang.UnsupportedOperationException: Unsupported!
[JBoss] 	at org.infinispan.distribution.ch.UnionConsistentHash.getHashId(UnionConsistentHash.java:53)
[JBoss] 	at org.infinispan.server.hotrod.HotRodEncoder$$anonfun$writeHashTopologyHeader$4.apply(HotRodEncoder.scala:164)
[JBoss] 	at org.infinispan.server.hotrod.HotRodEncoder$$anonfun$writeHashTopologyHeader$4.apply(HotRodEncoder.scala:160)
[JBoss] 	at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
[JBoss] 	at scala.collection.immutable.List.foreach(List.scala:45)
[JBoss] 	at org.infinispan.server.hotrod.HotRodEncoder.writeHashTopologyHeader(HotRodEncoder.scala:160)
[JBoss] 	at org.infinispan.server.hotrod.HotRodEncoder.writeHeader(HotRodEncoder.scala:117)
[JBoss] 	at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:32)
[JBoss] 	at org.infinispan.server.core.transport.netty.EncoderAdapter.encode(EncoderAdapter.scala:17)
[JBoss] 	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66)
[JBoss] 	at org.jboss.netty.channel.Channels.write(Channels.java:611)
[JBoss] 	at org.jboss.netty.channel.Channels.write(Channels.java:578)
[JBoss] 	at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:258)
[JBoss] 	at org.infinispan.server.core.transport.netty.ChannelAdapter.write(ChannelAdapter.scala:21)
[JBoss] 	at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:87)
[JBoss] 	at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:51)
[JBoss] 	at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:18)
[JBoss] 	at org.infinispan.server.core.transport.netty.DecoderAdapter.decode(DecoderAdapter.scala:14)
[JBoss] 	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:461)
[JBoss] 	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:434)
[JBoss] 	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
[JBoss] 	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
[JBoss] 	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
[JBoss] 	at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
[JBoss] 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
[JBoss] 	at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
[JBoss] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[JBoss] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[JBoss] 	at java.lang.Thread.run(Thread.java:619)

full log:
http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-stress-client-size6-hotrod/1/console-perf01/consoleText

this will be probably hard to reproduce, but I guess is worth reporting and investigating.

Comment 1 Galder Zamarreño 2011-03-03 08:08:53 UTC
Hmmm, this is something that needs looking into. That UnionConsistentHash is used when rehashing is going on and a new view is being installed. It appears that the hash id for a given node cannot be calculated at that point. I'll probably need to check with Manik on this. Good catch Michal!

Comment 2 Galder Zamarreño 2011-03-03 08:15:58 UTC
Link: Added: This issue depends ISPN-963


Comment 3 Michal Linhard 2011-03-03 10:06:03 UTC
today Rado also ran into this problem when running a job:

http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-resilience-client-size4-hotrod/6/console-perf02/

Comment 4 Galder Zamarreño 2011-03-07 10:04:16 UTC
Michal/Rado, this has now been fixed and we'll release today a new CR (March 7). Can you make sure you run your tests to verify it works fine now? Marking it as resolved for the time being.

Comment 5 Galder Zamarreño 2011-03-08 12:27:39 UTC
Michal, CR4 is out now. Make sure you integrate and test it asap since we're targeting a Final at the end of this week.

Comment 6 Michal Linhard 2011-03-08 12:32:58 UTC
Sure, we're on it.

Comment 7 Michal Linhard 2011-03-08 16:11:32 UTC
Well, I've updated EDG to use infinispan 4.2.1.CR4 and Rado has rerun his job, we saw these exceptions in.
They are not occuring, but I don't know whether this was a valid run.
http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-resilience-client-size4-hotrod/13/

It's not currently possible to rerun my original job
http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-stress-client-size6-hotrod/
it was only a short time for which I had perf lab machines available for size6 cluster tests. It might take a little longer till we can run such jobs again. And even then It occured only once in my run. Rado had multiple occurences of them.

Since Rado is already gone, we'll try to rerun Rado's job tomorrow and verify again.


Comment 9 Michal Linhard 2011-03-09 16:34:29 UTC
Rado did another runs of his job and we didn't see the exception again
http://hudson.qa.jboss.com/hudson/view/EDG/job/edg-51x-resilience-client-size4-hotrod/

I don't know how to exactly replicate this so this so we didn't verify the issue for 100%, but it seems that it's ok now.
I'll run one more test with CR3 to see whether we'll see the exceptions there.

Comment 10 Michal Linhard 2011-03-10 07:39:09 UTC
unfortunately the same run with 4.2.1.CR3 didn't show the exceptions.

Comment 11 Galder Zamarreño 2011-03-10 11:17:45 UTC
Michal/Rado, just keep running your tests and if it appears again, we'll look into it.

Comment 12 Radoslav Husar 2011-03-11 10:31:50 UTC
Link: Added: This issue is a dependency of JBPAPP-6054