Bug 1567557 - [Infra][Scale] java.lang.IllegalStateException: Deserializer for key was not found [NEEDINFO]
Summary: [Infra][Scale] java.lang.IllegalStateException: Deserializer for key was not ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z5
: 13.0 (Queens)
Assignee: Josh Hershberg
QA Contact: Noam Manos
URL:
Whiteboard: Infra
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-14 23:45 UTC by Sai Sindhur Malleni
Modified: 2018-10-10 08:01 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-10 08:01:16 UTC
Target Upstream Version:
aadam: needinfo? (shague)


Attachments (Terms of Use)
Waldek's log (13.71 MB, text/plain)
2018-04-22 06:46 UTC, Mike Kolesnik
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenDaylight Bug OPNFLWPLUG-917 0 None None None 2018-06-11 20:49:48 UTC

Description Sai Sindhur Malleni 2018-04-14 23:45:26 UTC
Description of problem:
When running a Browbeat+Rally use case in my downstream scale CI which does:
Create a network
Create a sbunet
Create a router
Attach router to subnet and oublic network
Boot VM with floating IP
Ping VM

with concurrency 8 and times set to 50 we see that some VMs remain unpingable even after 300 seconds and we see a lot of the following exceptions in karaf logs
java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly
        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserializeEntry(MatchDeserializer.java:105) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:61) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:33) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:66) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:29) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:54) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:51) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:27) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [58:io.netty.codec:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [58:io.netty.codec:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [58:io.netty.codec:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [61:io.netty.handler:4.1.16.Final]
        at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:37) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) [63:io.netty.transport:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [60:io.netty.common:4.1.16.Final]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [60:io.netty.common:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:?]


Version-Release number of selected component (if applicable):
OSP 13
opendaylight-8.0.0-5.el7ost.noarch

How reproducible: 100%


Steps to Reproduce:
1. Install OSP13 + OpenDaylight
2. Run scale tests as mentioned above
3. 

Actual results:
Some Vms remain unpingable with exceptions in karaf logs

Expected results:
No exceptions and Vms should be pingable

Additional info:

Comment 4 Mike Kolesnik 2018-04-16 06:19:33 UTC
Can you provide logs with openflowplugin set to log in TRACE level?

karaf> log:set org.opendaylight.openflowplugin TRACE (or something like that)

Comment 5 Mike Kolesnik 2018-04-16 06:20:25 UTC
Waldek, did you see this error when the server failed to start, or something else?

Comment 6 Waldemar Znoinski 2018-04-16 08:26:03 UTC
Mike - yes, I do see the exception (slightly different one from Sai's) during server create and then the server goes into ERROR state. I do see similar exception (the exactly one Sai sees) in the logs for last couple of days even when the instances were comming up good. I'm not sure at this moment whether this exception affects server create then. I'll dig deeper + see the TRACE logs.

Comment 8 Mike Kolesnik 2018-04-22 06:46:06 UTC
Created attachment 1425250 [details]
Waldek's log

Comment 9 Mike Kolesnik 2018-04-22 06:52:54 UTC
Josh, I'm setting this to medium based on our previous conversation regarding this bug.
If you believe it shoudl be higher priority, please update accordingly.

Comment 11 Josh Hershberg 2018-10-10 08:01:16 UTC
Closing as this no longer happens


Note You need to log in before you can comment on or make changes to this bug.