Hide Forgot
Description of problem: When running a Browbeat+Rally use case in my downstream scale CI which does: Create a network Create a sbunet Create a router Attach router to subnet and oublic network Boot VM with floating IP Ping VM with concurrency 8 and times set to 50 we see that some VMs remain unpingable even after 300 seconds and we see a lot of the following exceptions in karaf logs java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5] at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserializeEntry(MatchDeserializer.java:105) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5] at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:61) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5] at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:33) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5] at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:66) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5] at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:29) [389:org.opendaylight.openflowplugin.impl:0.6.0.redhat-5] at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:54) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5] at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:51) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5] at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:27) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [58:io.netty.codec:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [58:io.netty.codec:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [58:io.netty.codec:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [58:io.netty.codec:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [58:io.netty.codec:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [61:io.netty.handler:4.1.16.Final] at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:37) [398:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.redhat-5] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [63:io.netty.transport:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) [63:io.netty.transport:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [60:io.netty.common:4.1.16.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [60:io.netty.common:4.1.16.Final] at java.lang.Thread.run(Thread.java:748) [?:?] Version-Release number of selected component (if applicable): OSP 13 opendaylight-8.0.0-5.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Install OSP13 + OpenDaylight 2. Run scale tests as mentioned above 3. Actual results: Some Vms remain unpingable with exceptions in karaf logs Expected results: No exceptions and Vms should be pingable Additional info:
Can you provide logs with openflowplugin set to log in TRACE level? karaf> log:set org.opendaylight.openflowplugin TRACE (or something like that)
Waldek, did you see this error when the server failed to start, or something else?
Mike - yes, I do see the exception (slightly different one from Sai's) during server create and then the server goes into ERROR state. I do see similar exception (the exactly one Sai sees) in the logs for last couple of days even when the instances were comming up good. I'm not sure at this moment whether this exception affects server create then. I'll dig deeper + see the TRACE logs.
Created attachment 1425250 [details] Waldek's log
Josh, I'm setting this to medium based on our previous conversation regarding this bug. If you believe it shoudl be higher priority, please update accordingly.
Closing as this no longer happens