Created attachment 1969915 [details] networktopology Description of problem: We have a simple All in one standalone OSP(17.0) deployment. For the network topology please refer to the networktopoloy.png attached Networks are created as below. [stack@rhos-nfv-02 osp17.0-aio-compute04]$ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ openstack network list +--------------------------------------+----------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------------+--------------------------------------+ | 00b654f0-0dd8-452f-b094-cd9ffb5e5e1c | datacentre-112 | 9c9161c1-7dc0-4619-9640-7dc922e7d3e0 | | 1b8c6f1c-5a8f-4213-b7d4-498166efcdab | datacentre-114 | ed53eb7c-1981-4a01-96fe-416d6be14eff | | 8b0cc6d8-d88c-498e-be8e-4beb784daa5d | datacentre-111 | c052f816-41e7-4801-8f38-a94464c99841 | | a380bf21-c6fc-4f97-850c-256514784389 | datacentre-113 | 38e5e17d-5c6d-4b6a-b867-16346e26cba3 | +--------------------------------------+----------------+--------------------------------------+ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ openstack subnet list +--------------------------------------+-----------------------+--------------------------------------+------------------+ | ID | Name | Network | Subnet | +--------------------------------------+-----------------------+--------------------------------------+------------------+ | 38e5e17d-5c6d-4b6a-b867-16346e26cba3 | subnet-datacentre-113 | a380bf21-c6fc-4f97-850c-256514784389 | 192.168.113.0/24 | | 9c9161c1-7dc0-4619-9640-7dc922e7d3e0 | subnet-datacentre-112 | 00b654f0-0dd8-452f-b094-cd9ffb5e5e1c | 192.168.112.0/24 | | c052f816-41e7-4801-8f38-a94464c99841 | subnet-datacentre-111 | 8b0cc6d8-d88c-498e-be8e-4beb784daa5d | 192.168.111.0/24 | | ed53eb7c-1981-4a01-96fe-416d6be14eff | subnet-datacentre-114 | 1b8c6f1c-5a8f-4213-b7d4-498166efcdab | 192.168.114.0/24 | +--------------------------------------+-----------------------+--------------------------------------+------------------+ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 23eadebe-22fc-4ee0-83a8-f608ce81f316 | router2 | ACTIVE | UP | fbdcc48e18ff4cd084cc66011d68dcbb | | b82cb61e-340e-49ec-9ef7-7253f3886e0f | router1 | ACTIVE | UP | fbdcc48e18ff4cd084cc66011d68dcbb | +--------------------------------------+---------+--------+-------+----------------------------------+ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ openstack port list +--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+ | 0c3808a5-3f75-4ce3-b894-ca4f807534b5 | | fa:16:3e:b1:69:af | ip_address='192.168.113.2', subnet_id='38e5e17d-5c6d-4b6a-b867-16346e26cba3' | DOWN | | 2aaaa3ac-43f7-4909-a49c-5984030b5ce0 | | fa:16:3e:73:3d:ac | ip_address='192.168.113.30', subnet_id='38e5e17d-5c6d-4b6a-b867-16346e26cba3' | ACTIVE | | 479c002d-0d5a-4d2d-b49e-8e525609d239 | | fa:16:3e:15:12:7c | ip_address='192.168.112.2', subnet_id='9c9161c1-7dc0-4619-9640-7dc922e7d3e0' | DOWN | | 4f19a749-63ad-4ff7-8c20-9f8e0eb6a117 | | fa:16:3e:6e:24:99 | ip_address='192.168.112.8', subnet_id='9c9161c1-7dc0-4619-9640-7dc922e7d3e0' | ACTIVE | | 5fd65b76-02ef-433f-8a69-39c07ff28f0c | | fa:16:3e:ff:2f:81 | ip_address='192.168.114.1', subnet_id='ed53eb7c-1981-4a01-96fe-416d6be14eff' | DOWN | | a811fb48-e4c8-4542-ad02-30c8c7cc13e4 | | fa:16:3e:5a:90:99 | ip_address='192.168.111.1', subnet_id='c052f816-41e7-4801-8f38-a94464c99841' | DOWN | | ad754fbd-89a8-4515-aa9d-d47769599acf | | fa:16:3e:c7:13:26 | ip_address='192.168.113.1', subnet_id='38e5e17d-5c6d-4b6a-b867-16346e26cba3' | ACTIVE | | d5c10479-9858-4d60-86c1-de2dff33b5d5 | | fa:16:3e:39:cf:51 | ip_address='192.168.112.1', subnet_id='9c9161c1-7dc0-4619-9640-7dc922e7d3e0' | ACTIVE | | e6c58a01-64c2-45ed-926a-11174e8dae8c | | fa:16:3e:5e:f5:37 | ip_address='192.168.114.254', subnet_id='ed53eb7c-1981-4a01-96fe-416d6be14eff' | ACTIVE | | f87b32a7-192d-4901-aaea-dee0c432a445 | | fa:16:3e:a9:da:d2 | ip_address='192.168.111.254', subnet_id='c052f816-41e7-4801-8f38-a94464c99841' | ACTIVE | +--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+ [stack@rhos-nfv-02 osp17.0-aio-compute04]$ OVS SHOW command [stack@rhos-nfv-02 osp17.0-aio-compute04]$ sudo ovs-vsctl show 83cc51a6-7d0a-4670-acd1-1475a846e478 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-link0 fail_mode: standalone datapath_type: netdev Port patch-provnet-a1db0274-05c0-4c00-9394-4fa7db335840-to-br-int Interface patch-provnet-a1db0274-05c0-4c00-9394-4fa7db335840-to-br-int type: patch options: {peer=patch-br-int-to-provnet-a1db0274-05c0-4c00-9394-4fa7db335840} Port patch-provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1-to-br-int Interface patch-provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1-to-br-int type: patch options: {peer=patch-br-int-to-provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1} Port br-link0 Interface br-link0 type: internal Port dpdk-eno1 Interface dpdk-eno1 type: dpdk options: {dpdk-devargs="0000:01:00.0"} Port dpdk-eno2 Interface dpdk-eno2 type: dpdk options: {dpdk-devargs="0000:01:00.1"} Port patch-provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82-to-br-int Interface patch-provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82-to-br-int type: patch options: {peer=patch-br-int-to-provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82} Port patch-provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd-to-br-int Interface patch-provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd-to-br-int type: patch options: {peer=patch-br-int-to-provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd} Bridge br-int fail_mode: secure datapath_type: netdev Port patch-br-int-to-provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd Interface patch-br-int-to-provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd type: patch options: {peer=patch-provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd-to-br-int} Port patch-br-int-to-provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82 Interface patch-br-int-to-provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82 type: patch options: {peer=patch-provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82-to-br-int} Port vhu2aaaa3ac-43 Interface vhu2aaaa3ac-43 type: dpdkvhostuserclient options: {vhost-server-path="/var/lib/vhost_sockets/vhu2aaaa3ac-43"} Port tap00b654f0-00 Interface tap00b654f0-00 Port tapa380bf21-c0 Interface tapa380bf21-c0 Port br-int Interface br-int type: internal Port vhu4f19a749-63 Interface vhu4f19a749-63 type: dpdkvhostuserclient options: {vhost-server-path="/var/lib/vhost_sockets/vhu4f19a749-63"} Port patch-br-int-to-provnet-a1db0274-05c0-4c00-9394-4fa7db335840 Interface patch-br-int-to-provnet-a1db0274-05c0-4c00-9394-4fa7db335840 type: patch options: {peer=patch-provnet-a1db0274-05c0-4c00-9394-4fa7db335840-to-br-int} Port patch-br-int-to-provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1 Interface patch-br-int-to-provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1 type: patch options: {peer=patch-provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1-to-br-int} ovs_version: "2.17.6" [stack@rhos-nfv-02 osp17.0-aio-compute04]$ OVN SHOW command [root@rhos-nfv-02 /]# ovn-nbctl show switch a3322348-bc63-4982-ae76-02aa80ef83e1 (neutron-8b0cc6d8-d88c-498e-be8e-4beb784daa5d) (aka datacentre-111) port f87b32a7-192d-4901-aaea-dee0c432a445 type: router router-port: lrp-f87b32a7-192d-4901-aaea-dee0c432a445 port provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82 type: localnet tag: 111 addresses: ["unknown"] port a811fb48-e4c8-4542-ad02-30c8c7cc13e4 type: localport addresses: ["fa:16:3e:5a:90:99 192.168.111.1"] switch 9f943bb9-409f-41bd-b989-62ab33cd08d9 (neutron-1b8c6f1c-5a8f-4213-b7d4-498166efcdab) (aka datacentre-114) port provnet-a1db0274-05c0-4c00-9394-4fa7db335840 type: localnet tag: 114 addresses: ["unknown"] port 5fd65b76-02ef-433f-8a69-39c07ff28f0c type: localport addresses: ["fa:16:3e:ff:2f:81 192.168.114.1"] port e6c58a01-64c2-45ed-926a-11174e8dae8c type: router router-port: lrp-e6c58a01-64c2-45ed-926a-11174e8dae8c switch 6d91f3c2-9113-432a-aedf-132cbbe38ae3 (neutron-a380bf21-c6fc-4f97-850c-256514784389) (aka datacentre-113) port provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1 type: localnet tag: 113 addresses: ["unknown"] port ad754fbd-89a8-4515-aa9d-d47769599acf type: router router-port: lrp-ad754fbd-89a8-4515-aa9d-d47769599acf port 2aaaa3ac-43f7-4909-a49c-5984030b5ce0 addresses: ["fa:16:3e:73:3d:ac 192.168.113.30", "unknown"] port 0c3808a5-3f75-4ce3-b894-ca4f807534b5 type: localport addresses: ["fa:16:3e:b1:69:af 192.168.113.2"] switch 44607a9f-2f19-4fc3-bd02-0cb09381b3b5 (neutron-00b654f0-0dd8-452f-b094-cd9ffb5e5e1c) (aka datacentre-112) port 479c002d-0d5a-4d2d-b49e-8e525609d239 type: localport addresses: ["fa:16:3e:15:12:7c 192.168.112.2"] port d5c10479-9858-4d60-86c1-de2dff33b5d5 type: router router-port: lrp-d5c10479-9858-4d60-86c1-de2dff33b5d5 port provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd type: localnet tag: 112 addresses: ["unknown"] port 4f19a749-63ad-4ff7-8c20-9f8e0eb6a117 addresses: ["fa:16:3e:6e:24:99 192.168.112.8", "unknown"] router 158b7ff9-726e-4784-87c3-3f3543334e09 (neutron-23eadebe-22fc-4ee0-83a8-f608ce81f316) (aka router2) port lrp-ad754fbd-89a8-4515-aa9d-d47769599acf mac: "fa:16:3e:c7:13:26" networks: ["192.168.113.1/24"] port lrp-e6c58a01-64c2-45ed-926a-11174e8dae8c mac: "fa:16:3e:5e:f5:37" networks: ["192.168.114.254/24"] router 59a2aabb-26b3-40bf-943d-5591a8c4a2e0 (neutron-b82cb61e-340e-49ec-9ef7-7253f3886e0f) (aka router1) port lrp-f87b32a7-192d-4901-aaea-dee0c432a445 mac: "fa:16:3e:a9:da:d2" networks: ["192.168.111.254/24"] port lrp-d5c10479-9858-4d60-86c1-de2dff33b5d5 mac: "fa:16:3e:39:cf:51" networks: ["192.168.112.1/24"] [root@rhos-nfv-02 /]# [root@rhos-nfv-02 /]# ovn-nbctl show switch a3322348-bc63-4982-ae76-02aa80ef83e1 (neutron-8b0cc6d8-d88c-498e-be8e-4beb784daa5d) (aka datacentre-111) port f87b32a7-192d-4901-aaea-dee0c432a445 type: router router-port: lrp-f87b32a7-192d-4901-aaea-dee0c432a445 port provnet-2fc718a9-5e31-46e0-b3aa-7d6dbc058e82 type: localnet tag: 111 addresses: ["unknown"] port a811fb48-e4c8-4542-ad02-30c8c7cc13e4 type: localport addresses: ["fa:16:3e:5a:90:99 192.168.111.1"] switch 9f943bb9-409f-41bd-b989-62ab33cd08d9 (neutron-1b8c6f1c-5a8f-4213-b7d4-498166efcdab) (aka datacentre-114) port provnet-a1db0274-05c0-4c00-9394-4fa7db335840 type: localnet tag: 114 addresses: ["unknown"] port 5fd65b76-02ef-433f-8a69-39c07ff28f0c type: localport addresses: ["fa:16:3e:ff:2f:81 192.168.114.1"] port e6c58a01-64c2-45ed-926a-11174e8dae8c type: router router-port: lrp-e6c58a01-64c2-45ed-926a-11174e8dae8c switch 6d91f3c2-9113-432a-aedf-132cbbe38ae3 (neutron-a380bf21-c6fc-4f97-850c-256514784389) (aka datacentre-113) port provnet-55c2ecb3-39ed-4328-9459-5f5361cf96f1 type: localnet tag: 113 addresses: ["unknown"] port ad754fbd-89a8-4515-aa9d-d47769599acf type: router router-port: lrp-ad754fbd-89a8-4515-aa9d-d47769599acf port 2aaaa3ac-43f7-4909-a49c-5984030b5ce0 addresses: ["fa:16:3e:73:3d:ac 192.168.113.30", "unknown"] port 0c3808a5-3f75-4ce3-b894-ca4f807534b5 type: localport addresses: ["fa:16:3e:b1:69:af 192.168.113.2"] switch 44607a9f-2f19-4fc3-bd02-0cb09381b3b5 (neutron-00b654f0-0dd8-452f-b094-cd9ffb5e5e1c) (aka datacentre-112) port 479c002d-0d5a-4d2d-b49e-8e525609d239 type: localport addresses: ["fa:16:3e:15:12:7c 192.168.112.2"] port d5c10479-9858-4d60-86c1-de2dff33b5d5 type: router router-port: lrp-d5c10479-9858-4d60-86c1-de2dff33b5d5 port provnet-b7f9efd2-d0d6-47d9-bc6a-a9726952c5fd type: localnet tag: 112 addresses: ["unknown"] port 4f19a749-63ad-4ff7-8c20-9f8e0eb6a117 addresses: ["fa:16:3e:6e:24:99 192.168.112.8", "unknown"] router 158b7ff9-726e-4784-87c3-3f3543334e09 (neutron-23eadebe-22fc-4ee0-83a8-f608ce81f316) (aka router2) port lrp-ad754fbd-89a8-4515-aa9d-d47769599acf mac: "fa:16:3e:c7:13:26" networks: ["192.168.113.1/24"] port lrp-e6c58a01-64c2-45ed-926a-11174e8dae8c mac: "fa:16:3e:5e:f5:37" networks: ["192.168.114.254/24"] router 59a2aabb-26b3-40bf-943d-5591a8c4a2e0 (neutron-b82cb61e-340e-49ec-9ef7-7253f3886e0f) (aka router1) port lrp-f87b32a7-192d-4901-aaea-dee0c432a445 mac: "fa:16:3e:a9:da:d2" networks: ["192.168.111.254/24"] port lrp-d5c10479-9858-4d60-86c1-de2dff33b5d5 mac: "fa:16:3e:39:cf:51" networks: ["192.168.112.1/24"] [root@rhos-nfv-02 /]# NOTE that TRex is outside of NEUTRON network. TREX Port1 - IP and MAC - 192.168.111.111- e4:43:4b:49:a6:64 TREX Port2 - IP and MAC - 192.168.114.111- e4:43:4b:49:a6:66 When we send packets from TRex, although we are getting packets on the TRex Ports we do see "flooding" at the br-link0 for flows coming from both dpdk-eno1 and dpdk-eno2. [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-appctl dpctl/dump-flows -m flow-dump from pmd on cpu core: 6 ufid:ce3b079a-dc7a-48f2-a4cd-a18443bc0b67, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno2),packet_type(ns=0,id=0),eth(src=e4:43:4b:49:a6:66,dst=fa:16:3e:5e:f5:37),eth_type(0x8100),vlan(vid=114,pcp=0),encap(eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128,dst=192.168.111.111/255.255.255.128,proto=17,tos=0/0,ttl=64,frag=no),udp(src=1026/0,dst=1025/0)), packets:13190822, bytes:791449320, used:0.000s, dp:ovs, actions:br-link0,dpdk-eno1,set(eth(src=fa:16:3e:c7:13:26,dst=fa:16:3e:73:3d:ac)),set(ipv4(ttl=63)),pop_vlan,vhu2aaaa3ac-43, dp-extra-info:miniflow_bits(5,2) flow-dump from pmd on cpu core: 30 ufid:16ec2fb5-b481-4de8-8ce0-26c7b147b7ca, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vhu2aaaa3ac-43),packet_type(ns=0,id=0),eth(src=fa:16:3e:73:3d:ac,dst=fa:16:3e:c7:13:26),eth_type(0x0800),ipv4(src=192.168.111.111/255.255.255.128,dst=192.168.114.111,proto=17,tos=0/0,ttl=63,frag=no),udp(src=1025/0,dst=1026/0), packets:13128259, bytes:735182504, used:0.000s, dp:ovs, actions:set(eth(src=fa:16:3a:00:53:00,dst=e4:43:4b:49:a6:66)),set(ipv4(ttl=62)),push_vlan(vid=114,pcp=0),dpdk-eno2, dp-extra-info:miniflow_bits(5,2) flow-dump from pmd on cpu core: 4 ufid:35bd0d49-c298-4027-974f-820912b8ab8d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno1),packet_type(ns=0,id=0),eth(src=e4:43:4b:49:a6:64,dst=fa:16:3e:a9:da:d2),eth_type(0x8100),vlan(vid=111,pcp=0),encap(eth_type(0x0800),ipv4(src=192.168.111.111/255.255.255.128,dst=192.168.114.111/255.255.255.128,proto=17,tos=0/0,ttl=64,frag=no),udp(src=1025/0,dst=1026/0)), packets:13128323, bytes:787699380, used:0.000s, dp:ovs, actions:br-link0,dpdk-eno2,set(eth(src=fa:16:3e:39:cf:51,dst=fa:16:3e:6e:24:99)),set(ipv4(ttl=63)),pop_vlan,vhu4f19a749-63, dp-extra-info:miniflow_bits(5,2) flow-dump from pmd on cpu core: 28 ufid:73d78dba-7c73-43f9-8008-6552102a4081, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vhu4f19a749-63),packet_type(ns=0,id=0),eth(src=fa:16:3e:6e:24:99,dst=fa:16:3e:39:cf:51),eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128,dst=192.168.111.111,proto=17,tos=0/0,ttl=63,frag=no),udp(src=1026/0,dst=1025/0), packets:13190822, bytes:738686032, used:0.000s, dp:ovs, actions:set(eth(src=fa:16:3a:00:53:00,dst=e4:43:4b:49:a6:64)),set(ipv4(ttl=62)),push_vlan(vid=111,pcp=0),dpdk-eno1, dp-extra-info:miniflow_bits(5,2) [root@rhos-nfv-02 osp17.0-aio-compute04]# First lets consider the incoming flows.. Here if we see the flows are getting flooded to all the ports which is not expected. While fa:16:3e:a9:da:d2 and fa:16:3e:5e:f5:37 are neutron routers router1 and router2 gateway mac respectively ideally there should be no need of flooding because its knows it gateway mac!. Am I missing something? ufid:35bd0d49-c298-4027-974f-820912b8ab8d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno1),packet_type(ns=0,id=0),eth(src=e4:43:4b:49:a6:64,dst=fa:16:3e:a9:da:d2),eth_type(0x8100),vlan(vid=111,pcp=0),encap(eth_type(0x0800),ipv4(src=192.168.111.111/255.255.255.128,dst=192.168.114.111/255.255.255.128,proto=17,tos=0/0,ttl=64,frag=no),udp(src=1025/0,dst=1026/0)), packets:13128323, bytes:787699380, used:0.000s, dp:ovs, actions:br-link0,dpdk-eno2,set(eth(src=fa:16:3e:39:cf:51,dst=fa:16:3e:6e:24:99)),set(ipv4(ttl=63)),pop_vlan,vhu4f19a749-63, dp-extra-info:miniflow_bits(5,2) ufid:ce3b079a-dc7a-48f2-a4cd-a18443bc0b67, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno2),packet_type(ns=0,id=0),eth(src=e4:43:4b:49:a6:66,dst=fa:16:3e:5e:f5:37),eth_type(0x8100),vlan(vid=114,pcp=0),encap(eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128,dst=192.168.111.111/255.255.255.128,proto=17,tos=0/0,ttl=64,frag=no),udp(src=1026/0,dst=1025/0)), packets:13190822, bytes:791449320, used:0.000s, dp:ovs, actions:br-link0,dpdk-eno1,set(eth(src=fa:16:3e:c7:13:26,dst=fa:16:3e:73:3d:ac)),set(ipv4(ttl=63)),pop_vlan,vhu2aaaa3ac-43, dp-extra-info:miniflow_bits(5,2) Below are the return path flows. Here there is no flooding. But in the actions we see a strange MAC(src=fa:16:3a:00:53:00) being set which we dont get as its not of ANY of the PORTS created by NEUTRON. ufid:73d78dba-7c73-43f9-8008-6552102a4081, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vhu4f19a749-63),packet_type(ns=0,id=0),eth(src=fa:16:3e:6e:24:99,dst=fa:16:3e:39:cf:51),eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128,dst=192.168.111.111,proto=17,tos=0/0,ttl=63,frag=no),udp(src=1026/0,dst=1025/0), packets:13190822, bytes:738686032, used:0.000s, dp:ovs, actions:set(eth(src=fa:16:3a:00:53:00,dst=e4:43:4b:49:a6:64)),set(ipv4(ttl=62)),push_vlan(vid=111,pcp=0),dpdk-eno1, dp-extra-info:miniflow_bits(5,2) ufid:73d78dba-7c73-43f9-8008-6552102a4081, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vhu4f19a749-63),packet_type(ns=0,id=0),eth(src=fa:16:3e:6e:24:99,dst=fa:16:3e:39:cf:51),eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128,dst=192.168.111.111,proto=17,tos=0/0,ttl=63,frag=no),udp(src=1026/0,dst=1025/0), packets:13190822, bytes:738686032, used:0.000s, dp:ovs, actions:set(eth(src=fa:16:3a:00:53:00,dst=e4:43:4b:49:a6:64)),set(ipv4(ttl=62)),push_vlan(vid=111,pcp=0),dpdk-eno1, dp-extra-info:miniflow_bits(5,2) Version-Release number of selected component (if applicable): How reproducible: Already details about topology and issue given above. Steps to Reproduce: 1. 2. 3. Actual results: Seeing Flooding at external bridge (here - br-link0) Expected results: Expecting no flooding. Additional info:
Thanks Numan for quickly looking into it and help find the root cause.. Pasting the investigation details captured during interaction with Numan [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-ofctl dump-flows br-int | grep fa:16:3a:00:53:00 cookie=0xb5653cb2, duration=160123.138s, table=65, n_packets=8794604347, n_bytes=492497843592, idle_age=0, hard_age=65534, priority=150,reg15=0x1,metadata=0x1,dl_src=fa:16: 3e:a9:da:d2 actions=mod_dl_src:fa:16:3a:00:53:00,mod_vlan_vid:111,output:46,mod_dl_src:fa:16:3e:a9:da:d2,strip_vlan cookie=0xf6e03acb, duration=160123.138s, table=65, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=150,reg15=0x1,metadata=0x3,dl_src=fa:16:3e:39:cf:51 acti ons=mod_dl_src:fa:16:3a:00:53:00,mod_vlan_vid:112,output:47,mod_dl_src:fa:16:3e:39:cf:51,strip_vlan cookie=0xfdbb1983, duration=160123.107s, table=65, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=150,reg15=0x1,metadata=0x4,dl_src=fa:16:3e:c7:13:26 acti ons=mod_dl_src:fa:16:3a:00:53:00,mod_vlan_vid:113,output:50,mod_dl_src:fa:16:3e:c7:13:26,strip_vlan cookie=0xa0bb78a7, duration=160123.107s, table=65, n_packets=8758460231, n_bytes=490473773096, idle_age=0, hard_age=65534, priority=150,reg15=0x1,metadata=0x2,dl_src=fa:16: 3e:5e:f5:37 actions=mod_dl_src:fa:16:3a:00:53:00,mod_vlan_vid:114,output:49,mod_dl_src:fa:16:3e:5e:f5:37,strip_vlan [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-appctl fdb/show br-link0 port VLAN MAC Age 2 114 e4:43:4b:49:a6:66 0 27 114 fa:16:3a:00:53:00 0 1 111 e4:43:4b:49:a6:64 0 25 111 fa:16:3a:00:53:00 0 [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-vsctl list open _uuid : 83cc51a6-7d0a-4670-acd1-1475a846e478 bridges : [2fc30901-f680-4545-aa30-bb78f5513ad6, b6d53aff-384a-414f-851a-1c4b396dfc19] cur_cfg : 127 datapath_types : [netdev, system] datapaths : {netdev=92f3c847-f1c6-4e25-80dd-c7aa33646456} db_version : "8.3.0" dpdk_initialized : true dpdk_version : "DPDK 21.11.2" external_ids : {hostname=rhos-nfv-02.lab.eng.rdu2.redhat.com, ovn-bridge=br-int, ovn-bridge-datapath-type=netdev, ovn-bridge-mappings="datacentre:br-link0", ovn-chass is-mac-mappings="datacentre:fa:16:3a:00:53:00", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.8.2.153", ovn-encap-type=geneve, ovn-match-northd-version="true", ovn- monitor-all="true", ovn-openflow-probe-interval="60", ovn-remote="tcp:10.8.2.153:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="b23cb4b9 -9989-47b7-8886-447e14e1e4b6"} iface_types : [bareudp, dpdk, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan] manager_options : [8307b38a-2d33-4643-b369-030031641e6d] next_cfg : 127 other_config : {dpdk-extra=" -n 4", dpdk-init="true", dpdk-socket-mem="4096", pmd-cpu-mask=f00000f0, vhost-postcopy-support="true", vlan-limit="0"} ovs_version : "2.17.6" ssl : [] statistics : {} system_type : rhel system_version : "9.0" [root@rhos-nfv-02 osp17.0-aio-compute04]# So, the strange mac which we were seeing is the ovn-chassis-mac-mappings="datacentre:fa:16:3a:00:53:00" Since this is an AIO standalone deployment this is not needed so we tried clearing it and test as below. [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-vsctl remove open . external_ids ovn-chassis-mac-mappings [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-ofctl dump-flows br-int | grep fa:16:3a:00:53:00 [root@rhos-nfv-02 osp17.0-aio-compute04]# With that we dont see the flooding anymore [root@rhos-nfv-02 osp17.0-aio-compute04]# ovs-appctl dpctl/dump-flows -m flow-dump from pmd on cpu core: 6 ufid:ce3b079a-dc7a-48f2-a4cd-a18443bc0b67, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno2) ,packet_type(ns=0,id=0),eth(src=e4:43:4b:49:a6:66,dst=fa:16:3e:5e:f5:37),eth_type(0x8100),vlan(vid=114,pcp=0),encap(eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128 ,dst=192.168.111.111/255.255.255.128,proto=17,tos=0/0,ttl=64,frag=no),udp(src=1026/0,dst=1025/0)), packets:154299780, bytes:9257986800, used:0.000s, dp:ovs, actions:set(eth( src=fa:16:3e:c7:13:26,dst=fa:16:3e:73:3d:ac)),set(ipv4(ttl=63)),pop_vlan,vhu2aaaa3ac-43, dp-extra-info:miniflow_bits(5,2) flow-dump from pmd on cpu core: 30 ufid:16ec2fb5-b481-4de8-8ce0-26c7b147b7ca, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vhu2aaaa3a c-43),packet_type(ns=0,id=0),eth(src=fa:16:3e:73:3d:ac,dst=fa:16:3e:c7:13:26),eth_type(0x0800),ipv4(src=192.168.111.111/255.255.255.128,dst=192.168.114.111,proto=17,tos=0/0, ttl=63,frag=no),udp(src=1025/0,dst=1026/0), packets:155101283, bytes:8685671848, used:0.001s, dp:ovs, actions:set(eth(src=fa:16:3e:5e:f5:37,dst=e4:43:4b:49:a6:66)),set(ipv4( ttl=62)),push_vlan(vid=114,pcp=0),dpdk-eno2, dp-extra-info:miniflow_bits(5,2) flow-dump from pmd on cpu core: 4 ufid:baf5ba74-5de3-414c-955c-87ca52338f42, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno1) ,packet_type(ns=0,id=0),eth(src=c0:bf:a7:68:af:05,dst=01:80:c2:00:00:0e),eth_type(0x88cc), packets:0, bytes:0, used:never, dp:ovs, actions:drop, dp-extra-info:miniflow_bits( 5,0) ufid:35bd0d49-c298-4027-974f-820912b8ab8d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-eno1) ,packet_type(ns=0,id=0),eth(src=e4:43:4b:49:a6:64,dst=fa:16:3e:a9:da:d2),eth_type(0x8100),vlan(vid=111,pcp=0),encap(eth_type(0x0800),ipv4(src=192.168.111.111/255.255.255.128 ,dst=192.168.114.111/255.255.255.128,proto=17,tos=0/0,ttl=64,frag=no),udp(src=1025/0,dst=1026/0)), packets:155101283, bytes:9306076980, used:0.001s, dp:ovs, actions:set(eth( src=fa:16:3e:39:cf:51,dst=fa:16:3e:6e:24:99)),set(ipv4(ttl=63)),pop_vlan,vhu4f19a749-63, dp-extra-info:miniflow_bits(5,2) flow-dump from pmd on cpu core: 28 ufid:73d78dba-7c73-43f9-8008-6552102a4081, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vhu4f19a74 9-63),packet_type(ns=0,id=0),eth(src=fa:16:3e:6e:24:99,dst=fa:16:3e:39:cf:51),eth_type(0x0800),ipv4(src=192.168.114.111/255.255.255.128,dst=192.168.111.111,proto=17,tos=0/0, ttl=63,frag=no),udp(src=1026/0,dst=1025/0), packets:154299780, bytes:8640787680, used:0.001s, dp:ovs, actions:set(eth(src=fa:16:3e:a9:da:d2,dst=e4:43:4b:49:a6:64)),set(ipv4( ttl=62)),push_vlan(vid=111,pcp=0),dpdk-eno1, dp-extra-info:miniflow_bits(5,2) [root@rhos-nfv-02 osp17.0-aio-compute04]# As I understand from Numan its a known issue wrt the ovn-chassis-mac-mappings . Thanks & Regards, Abhiram R N
Hi @arn , if the problem goes away when ovn-chassis-mac-mappings are removed, does that mean that the issue is just a misconfiguration? Or is there something that needs to be fixed in OVN?
Hi @mmichels@mmichels , I think we should ask Numan on this. Since mine was an AIO standalone setup he said clearing should solve the issue we were seeing and hence we just cleared that flag and proceeded. He said its a known issue. Not sure if we see this issue in other setups OR if its applicable elsewhere. @nusiddiq , can you comment on this? Thanks & Regards, Abhiram R N
According to @nusiddiq , this patch may address the issue: http://patchwork.ozlabs.org/project/ovn/patch/b1f969b14815ab65134c079565c37357dc74c945.1685966596.git.lorenzo.bianconi@redhat.com/ I'm prioritizing this as "medium" for now since we already have a potential fix for this.