Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1590567

Summary:

Deployment with Opendaylight fails in step 5 with Failed to validate OVS OpenFlow pipeline at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/o vs/opendaylight.pp

Product:

Red Hat OpenStack

Reporter:

Sai Sindhur Malleni <smalleni>

Component:

opendaylight

Assignee:

lpeer <lpeer>

Status:

CLOSED NOTABUG

QA Contact:

Itzik Brown <itbrown>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

13.0 (Queens)

CC:

aadam, mkolesni, nyechiel, trozet, vorburger

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-06-13 20:57:22 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
/var/log/messages on failed controller node	none

Description Sai Sindhur Malleni 2018-06-12 22:48:22 UTC

Created attachment 1450692 [details]
/var/log/messages on failed controller node

Description of problem:
Deploying OSP13 with ODL fails at step 5 with the following error as seen using openstack stack failures list overcloud
            "Warning: Undefined variable 'deploy_config_name'; ", 
            "Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ip_address instead. They are described at https://docs.puppet.com/puppet/latest/reference/l
ang_data_type.html#match-expressions. at [\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp\", 56]:[\"/var/lib/tripleo-config/puppet_step_config.pp\", 35]", 
            "   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')", 
            "Warning: This method is deprecated, please use the stdlib validate_legacy function,", 
            "                    with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.pp\", 54]:
[\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]", 
            "                    with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.p
p\", 55]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]", 
            "                    with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.pp\", 56
]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]", 
            "                    with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.pp\", 66]
:[\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]", 
            "                    with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.pp\", 68]:[\"/etc/pup
pet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]", 
            "                    with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.pp\", 8
9]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]", 
            "Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules"
, 
            "Error: Evaluation Error: Error while evaluating a Function Call, Failed to validate OVS OpenFlow pipeline at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/o
vs/opendaylight.pp:138:7 on node overcloud-controller-1.localdomain"


Version-Release number of selected component (if applicable):
OSP13
Puddle: 2018-06-08.3

How reproducible:


Steps to Reproduce:
1. Deploy OSP with ODL
2.
3.

Actual results:
Deploy fails in step 5

Expected results:
Deploy should succeed

Additional info:

Comment 1 Sai Sindhur Malleni 2018-06-13 13:06:42 UTC

Seems to happen only when passing the env file to install odl-mdsal-trace
parameter_defaults:
  OpenDaylightFeatures: ["odl-mdsal-trace","odl-netvirt-openstack","odl-netvirt-ui","odl-jolokia"]

Comment 2 Michael Vorburger 2018-06-13 13:30:01 UTC

Sai, I'm confused - WHICH is the real error here?  What you show above is TripleO related.. perhaps someone (Tim?) can peek at it to see what could possibly cause this?  I don't see how  odl-mdsal-trace could.  I've also had a look at the attached var/log/messages - but that is partial - is there a way that you can attach more of where that came from, from the Karaf start?  It shows a bunch of these errors in middle where it's running, and I've no idea if those are serious problems or false flags - need to see more about what happened before that:

2018-06-12T22:40:19,047 | ERROR | nioEventLoopGroup-5-53 | ExceptionHandler                 | 403 - org.opendaylight.ovsdb.library - 1.6.0.redhat-10 | Exception occurred while processing connection pipeline
io.netty.handler.codec.DecoderException: org.opendaylight.ovsdb.lib.error.InvalidEncodingException: currently only UTF-8 is supported
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) [58:io.netty.codec:4.1.16.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [58:io.netty.codec:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [63:io.netty.transport:4.1.16.Final]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [63:io.netty.transport:4.1.16.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [60:io.netty.common:4.1.16.Final]
 at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [60:io.netty.common:4.1.16.Final]
 at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: org.opendaylight.ovsdb.lib.error.InvalidEncodingException: currently only UTF-8 is supported
 at org.opendaylight.ovsdb.lib.jsonrpc.JsonRpcDecoder.decode(JsonRpcDecoder.java:80) ~[403:org.opendaylight.ovsdb.library:1.6.0.redhat-10]
 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[58:io.netty.codec:4.1.16.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[58:io.netty.codec:4.1.16.Final]
 ... 16 more
2018-06-12T22:40:19,050 | INFO  | nioEventLoopGroup-5-53 | ExceptionHandler                 | 403 - org.opendaylight.ovsdb.library - 1.6.0.redhat-10 | Disconnecting channel to ovsdb [id: 0xbffe23cb, L:/172.16.0.17:6640 - R:/172.16.0.17:34200]
2018-06-12T22:40:19,050 | INFO  | nioEventLoopGroup-5-53 | OvsdbConnectionService           | 403 - org.opendaylight.ovsdb.library - 1.6.0.redhat-10 | Connection closed ConnectionInfo [Remote-address=172.16.0.17, Remote-port=34200, Local-address172.16.0.17, Local-port=6640, type=PASSIVE]
2018-06-12T22:40:19,050 | INFO  | nioEventLoopGroup-5-53 | HwvtepConnectionManager          | 402 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.6.0.redhat-10 | Library disconnected PASSIVE from /172.16.0.17:34200 to /172.16.0.17:6640. Cleaning up the operational data store
2018-06-12T22:40:19,051 | WARN  | nioEventLoopGroup-5-53 | HwvtepConnectionManager          | 402 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.6.0.redhat-10 | HWVTEP disconnected event did not find connection instance for ConnectionInfo [_localIp=IpAddress [_ipv4Address=Ipv4Address [_value=172.16.0.17]], _localPort=PortNumber [_value=6640], _remoteIp=IpAddress [_ipv4Address=Ipv4Address [_value=172.16.0.17]], _remotePort=PortNumber [_value=34200], augmentation=[]]
2018-06-12T22:40:19,051 | INFO  | nioEventLoopGroup-5-53 | OvsdbConnectionManager           | 407 - org.opendaylight.ovsdb.southbound-impl - 1.6.0.redhat-10 | Library disconnected PASSIVE from /172.16.0.17:34200 to /172.16.0.17:6640. Cleaning up the operational data store
2018-06-12T22:40:19,051 | WARN  | nioEventLoopGroup-5-53 | OvsdbConnectionManager           | 407 - org.opendaylight.ovsdb.southbound-impl - 1.6.0.redhat-10 | disconnected : Connection instance not found for OVSDB Node ConnectionInfo [_localIp=IpAddress [_ipv4Address=Ipv4Address [_value=172.16.0.17]], _localPort=PortNumber [_value=6640], _remoteIp=IpAddress [_ipv4Address=Ipv4Address [_value=172.16.0.17]], _remotePort=PortNumber [_value=34200], augmentation=[]]

Comment 3 Sai Sindhur Malleni 2018-06-13 20:44:22 UTC

Although I do not know the root cause of the bug, removing odl-netvirt-ui from the features list and moving odl-jolokia to the beginning resulted in consistently successful deploys.


parameter_defaults:
  OpenDaylightFeatures: ["odl-jolokia","odl-mdsal-trace","odl-netvirt-openstack"]

was the right order.

Comment 4 Sai Sindhur Malleni 2018-06-13 20:44:48 UTC

Based on above should we closed this as not a bug?

Comment 5 Tim Rozet 2018-06-13 20:57:22 UTC

Feature loading order is always fragile with ODL.  I think in this case your features were not loading all the way, so no flows were making it into OVS.  That is why you hit the error in OVS for an additional check we added to ensure the flow pipeline is correct in each OVS.