1665157 – Missing route when deploying DCN with OVS-DPDK

Bug 1665157 - Missing route when deploying DCN with OVS-DPDK

Summary: Missing route when deploying DCN with OVS-DPDK

Keywords:
Status:	CLOSED DUPLICATE of bug 1650298
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	os-net-config
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	nlevinki
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-01-10 15:22 UTC by riontel
Modified:	2019-01-18 13:02 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-01-18 13:02:11 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description riontel 2019-01-10 15:22:15 UTC

Description of problem:

When deploying Distributed Compute Nodes using leaf-and-spine networking, Compute nodes with OVS-DPDK enabled interfaces are missing a route to the other networking segments.

Version-Release number of selected component (if applicable):
RHOSP13

How reproducible:
Every time

Steps to Reproduce:
1. Configure route in compute nic file
              - type: ovs_user_bridge
                name: br-link0
                use_dhcp: false
                ovs_extra:
                  - str_replace:
                      template: set port br-link0 tag=_VLAN_TAG_
                      params:
                        _VLAN_TAG_:
                           get_param: Tenant2NetworkVlanID
                addresses:
                  - ip_netmask:
                      get_param: Tenant2IpSubnet
                routes:
                  - ip_netmask:
                      get_param: TenantNetCidr
                    next_hop:
                      get_param: Tenant2InterfaceDefaultRoute
                members:
                  - type: ovs_dpdk_port
                    name: dpdk0
                    mtu: 9216
                    members:
                      - type: interface
                        name: ens3f0

2. Define values in network-environment.yaml
  TenantNetworkVlanID: 987
  TenantNetCidr: 192.168.84.64/26
  TenantAllocationPools: [ { 'start': '192.168.84.74', 'end': '192.168.84.126' } ]
  TenantInterfaceDefaultRoute: 192.168.84.65

  Tenant2NetworkVlanID: 977
  Tenant2NetCidr: 192.168.84.0/26
  Tenant2AllocationPools: [ { 'start': '192.168.84.10','end': '192.168.84.62' } ]
  Tenant2InterfaceDefaultRoute: 192.168.84.1

3. Deploy overcloud

Actual results:

The route shows up in os-net-config:
[heat-admin@overcloud-remotesriov-0 ~]$ less /etc/os-net-config/config.json |python -m json.tool
{
    "network_config": [
        {
            "addresses": [
                {
                    "ip_netmask": "192.168.88.95/26"
                }
            ],
            "dns_servers": [
                "10.255.252.25"
            ],
            "name": "eno1",
            "routes": [
                {
                    "ip_netmask": "169.254.169.254/32",
                    "next_hop": "192.168.88.199"
                },
                {
                    "default": true,
                    "next_hop": "192.168.88.65"
                },
                {
                    "ip_netmask": "192.168.88.192/26",
                    "next_hop": "192.168.88.65"
                }
            ],
            "type": "interface",
            "use_dhcp": false
        },
        {
            "name": "eno5",
            "type": "interface",
            "use_dhcp": false
        },
        {
            "addresses": [
                {
                    "ip_netmask": "192.168.88.11/26"
                }
            ],
            "device": "eno5",
            "routes": [
                {
                    "ip_netmask": "192.168.88.128/26",
                    "next_hop": "192.168.88.1"
                }
            ],
            "type": "vlan",
            "vlan_id": 970
        },
        {
            "addresses": [
                {
                    "ip_netmask": "192.168.84.16/26"
                }
            ],
            "members": [
                {
                    "members": [
                        {
                            "name": "ens3f0",
                            "type": "interface"
                        }
                    ],
                    "mtu": 9216,
                    "name": "dpdk0",
                    "type": "ovs_dpdk_port"
                }
            ],
            "name": "br-link0",
            "ovs_extra": [
                "set port br-link0 tag=977"
            ],
            "routes": [
                {
                    "ip_netmask": "192.168.84.64/26",
                    "next_hop": "192.168.84.1"
                }
            ],
            "type": "ovs_user_bridge",
            "use_dhcp": false
        },
        {
            "defroute": false,
            "name": "eno2",
            "type": "interface",
            "use_dhcp": false
        },
        {
            "defroute": false,
            "name": "eno3",
            "type": "interface",
            "use_dhcp": false
        },
        {
            "defroute": false,
            "name": "eno4",
            "type": "interface",
            "use_dhcp": false
        },
        {
            "defroute": false,
            "name": "eno6",
            "type": "interface",
            "use_dhcp": false
        },
        {
            "defroute": false,
            "name": "ens2f1",
            "type": "interface",
            "use_dhcp": false
        },
        {
            "defroute": false,
            "name": "ens3f1",
            "type": "interface",
            "use_dhcp": false
        }
    ]
}

Route file gets created on the correct interface:
[heat-admin@overcloud-remotesriov-0 ~]$ cat /etc/sysconfig/network-scripts/route-br-link0
192.168.84.64/26 via 192.168.84.1 dev br-link0

The route doesn't get populated in the routing table:

[heat-admin@overcloud-remotesriov-0 ~]$ ip r
default via 192.168.88.65 dev eno1
169.254.0.0/16 dev eno1 scope link metric 1002
169.254.0.0/16 dev eno2 scope link metric 1003
169.254.0.0/16 dev eno3 scope link metric 1004
169.254.0.0/16 dev eno4 scope link metric 1005
169.254.0.0/16 dev eno5 scope link metric 1007
169.254.0.0/16 dev ens2f1 scope link metric 1009
169.254.0.0/16 dev ens3f1 scope link metric 1011
169.254.0.0/16 dev vlan970 scope link metric 1014
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1
192.168.84.0/26 dev br-link0 proto kernel scope link src 192.168.84.16
192.168.88.0/26 dev vlan970 proto kernel scope link src 192.168.88.14
192.168.88.64/26 dev eno1 proto kernel scope link src 192.168.88.99
192.168.88.128/26 via 192.168.88.1 dev vlan970
192.168.88.192/26 via 192.168.88.65 dev eno1


Expected results:

Route gets added to the routing table:

[heat-admin@overcloud-remotesriov-0 ~]$ ip r
default via 192.168.88.65 dev eno1
169.254.0.0/16 dev eno1 scope link metric 1002
169.254.0.0/16 dev eno2 scope link metric 1003
169.254.0.0/16 dev eno3 scope link metric 1004
169.254.0.0/16 dev eno4 scope link metric 1005
169.254.0.0/16 dev eno5 scope link metric 1007
169.254.0.0/16 dev ens2f1 scope link metric 1009
169.254.0.0/16 dev ens3f1 scope link metric 1011
169.254.0.0/16 dev vlan970 scope link metric 1014
169.254.0.0/16 dev br-link0 scope link metric 1022
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1
192.168.84.0/26 dev br-link0 proto kernel scope link src 192.168.84.16
192.168.84.64/26 via 192.168.84.1 dev br-link0
192.168.88.0/26 dev vlan970 proto kernel scope link src 192.168.88.14
192.168.88.64/26 dev eno1 proto kernel scope link src 192.168.88.99
192.168.88.128/26 via 192.168.88.1 dev vlan970
192.168.88.192/26 via 192.168.88.65 dev eno1


Additional info:

Troubleshooting steps:

1.  Verified there was a correct route-br-link0 file
$ cat /etc/sysconfig/network-scripts/route-br-link0
192.168.84.64/26 via 192.168.84.1 dev br-link0

2.  Deleted the file (to make sure the customer didn't manually add it in while I wasn't looking)

3.  Reran os-net-config and checked for br-link0:
[heat-admin@overcloud-remotesriov-0 ~]$ sudo os-net-config -c /etc/os-net-config/config.json -d
<skip>
[2019/01/10 02:34:02 PM] [INFO] adding ovs user bridge: br-link0
[2019/01/10 02:34:02 PM] [DEBUG] ovs user bridge data: # This file is autogenerated by os-net-config
DEVICE=br-link0
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSUserBridge
BOOTPROTO=static
IPADDR=192.168.84.16
NETMASK=255.255.255.192
OVS_EXTRA="set port br-link0 tag=977 -- set bridge br-link0 fail_mode=standalone -- del-controller br-link0 -- set bridge br-link0 fail_mode=standalone -- del-controller br-link0"

[2019/01/10 02:34:02 PM] [INFO] adding custom route for interface: br-link0
[2019/01/10 02:34:02 PM] [DEBUG] route data: 192.168.84.64/26 via 192.168.84.1 dev br-link0

4.  Checked that the route file was created:
[heat-admin@overcloud-remotesriov-0 ~]$ cat /etc/sysconfig/network-scripts/route-br-link0
192.168.84.64/26 via 192.168.84.1 dev br-link0

5.  Checked routing table:
[heat-admin@overcloud-remotesriov-0 ~]$ ip r
default via 192.168.88.65 dev eno1
169.254.0.0/16 dev eno1 scope link metric 1002
169.254.0.0/16 dev eno2 scope link metric 1003
169.254.0.0/16 dev eno3 scope link metric 1004
169.254.0.0/16 dev eno4 scope link metric 1005
169.254.0.0/16 dev eno5 scope link metric 1007
169.254.0.0/16 dev ens2f1 scope link metric 1009
169.254.0.0/16 dev ens3f1 scope link metric 1011
169.254.0.0/16 dev vlan970 scope link metric 1014
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1
192.168.84.0/26 dev br-link0 proto kernel scope link src 192.168.84.16
192.168.88.0/26 dev vlan970 proto kernel scope link src 192.168.88.14
192.168.88.64/26 dev eno1 proto kernel scope link src 192.168.88.99
192.168.88.128/26 via 192.168.88.1 dev vlan970
192.168.88.192/26 via 192.168.88.65 dev eno1

The route is not present.

6.  Bounced the interface and checked the routing table again:
[heat-admin@overcloud-remotesriov-0 ~]$ sudo ifdown br-link0
[heat-admin@overcloud-remotesriov-0 ~]$ sudo ifup br-link0
[heat-admin@overcloud-remotesriov-0 ~]$ ip r
default via 192.168.88.65 dev eno1
169.254.0.0/16 dev eno1 scope link metric 1002
169.254.0.0/16 dev eno2 scope link metric 1003
169.254.0.0/16 dev eno3 scope link metric 1004
169.254.0.0/16 dev eno4 scope link metric 1005
169.254.0.0/16 dev eno5 scope link metric 1007
169.254.0.0/16 dev ens2f1 scope link metric 1009
169.254.0.0/16 dev ens3f1 scope link metric 1011
169.254.0.0/16 dev vlan970 scope link metric 1014
169.254.0.0/16 dev br-link0 scope link metric 1022
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1
192.168.84.0/26 dev br-link0 proto kernel scope link src 192.168.84.16
192.168.84.64/26 via 192.168.84.1 dev br-link0
192.168.88.0/26 dev vlan970 proto kernel scope link src 192.168.88.14
192.168.88.64/26 dev eno1 proto kernel scope link src 192.168.88.99
192.168.88.128/26 via 192.168.88.1 dev vlan970
192.168.88.192/26 via 192.168.88.65 dev eno1

Route is present.

Comment 1 Bob Fournier 2019-01-14 20:44:58 UTC

I believe this should be fixed with patch for https://bugzilla.redhat.com/show_bug.cgi?id=1650298.  This fix will be in the next OSP-13 release - 13z5.

Comment 2 riontel 2019-01-14 21:28:05 UTC

(In reply to Bob Fournier from comment #1)
> I believe this should be fixed with patch for
> https://bugzilla.redhat.com/show_bug.cgi?id=1650298.  This fix will be in
> the next OSP-13 release - 13z5.

When will 13z5 be released, so I can update the customer?

Note You need to log in before you can comment on or make changes to this bug.