Bug 1567557

Summary: [Infra][Scale] java.lang.IllegalStateException: Deserializer for key was not found
Product: Red Hat OpenStack Reporter: Sai Sindhur Malleni <smalleni>
Component: opendaylightAssignee: Josh Hershberg <jhershbe>
Status: CLOSED NOTABUG QA Contact: Noam Manos <nmanos>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: aadam, jhershbe, mkolesni, nyechiel, shague, smalleni, wznoinsk
Target Milestone: z5Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)Flags: aadam: needinfo? (shague)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Infra
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-10 08:01:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Waldek's log none

Description Sai Sindhur Malleni 2018-04-14 23:45:26 UTC
Description of problem:
When running a Browbeat+Rally use case in my downstream scale CI which does:
Create a network
Create a sbunet
Create a router
Attach router to subnet and oublic network
Boot VM with floating IP
Ping VM

with concurrency 8 and times set to 50 we see that some VMs remain unpingable even after 300 seconds and we see a lot of the following exceptions in karaf logs
java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly
        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserializeEntry(MatchDeserializer.java:105) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:61) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:33) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:66) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:29) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5]
        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:54) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:51) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:27) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [58:io.netty.codec:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [58:io.netty.codec:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [58:io.netty.codec:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [58:io.netty.codec:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [61:io.netty.handler:4.1.16.Final]
        at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:37) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [63:io.netty.transport:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) [63:io.netty.transport:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [60:io.netty.common:4.1.16.Final]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [60:io.netty.common:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:?]


Version-Release number of selected component (if applicable):
OSP 13
opendaylight-8.0.0-5.el7ost.noarch

How reproducible: 100%


Steps to Reproduce:
1. Install OSP13 + OpenDaylight
2. Run scale tests as mentioned above
3. 

Actual results:
Some Vms remain unpingable with exceptions in karaf logs

Expected results:
No exceptions and Vms should be pingable

Additional info:

Comment 4 Mike Kolesnik 2018-04-16 06:19:33 UTC
Can you provide logs with openflowplugin set to log in TRACE level?

karaf> log:set org.opendaylight.openflowplugin TRACE (or something like that)

Comment 5 Mike Kolesnik 2018-04-16 06:20:25 UTC
Waldek, did you see this error when the server failed to start, or something else?

Comment 6 Waldemar Znoinski 2018-04-16 08:26:03 UTC
Mike - yes, I do see the exception (slightly different one from Sai's) during server create and then the server goes into ERROR state. I do see similar exception (the exactly one Sai sees) in the logs for last couple of days even when the instances were comming up good. I'm not sure at this moment whether this exception affects server create then. I'll dig deeper + see the TRACE logs.

Comment 8 Mike Kolesnik 2018-04-22 06:46:06 UTC
Created attachment 1425250 [details]
Waldek's log

Comment 9 Mike Kolesnik 2018-04-22 06:52:54 UTC
Josh, I'm setting this to medium based on our previous conversation regarding this bug.
If you believe it shoudl be higher priority, please update accordingly.

Comment 11 Josh Hershberg 2018-10-10 08:01:16 UTC
Closing as this no longer happens