OVN-DVR on Red Hat OpenStack Platform release 16.0.1 (Train), DNAT not working for IPv4 with VLAN tenant network + VLAN provider network I already accounted for https://bugzilla.redhat.com/show_bug.cgi?id=1830734 and rebooted the instance after deployment. I tested 2 sequences: seq 1 where ARP was not yet resolved [1] and seq 2 where ARP was resolved already, about 30 seconds after resolving ARP for the first time [2] In [1]: ARP resolution: a. ARP request on VLAN 209 leaves undercloud-0 and hits all controllers and compute nodes (broadcast): 15:26:38.700599 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 46 b. overcloud-compute-0 answers across VLAN 209 and says that 172.31.0.227 is-at fa:16:3e:55:6a:38 (known unicast): 15:26:38.701064 fa:16:3e:55:6a:38 > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Reply 172.31.0.227 is-at fa:16:3e:55:6a:38, length 46 ICMP: c. ICMP echo request on VLAN 209 leaves undercloud-0 and hits overcloud-compute-0 only (known unicast): 15:26:38.701325 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 172.31.0.1 > 172.31.0.227: ICMP echo request, id 12680, seq 1, length 64 d. ICMP overcloud echo request leaves overcloud-novacompute-0 on VLAN 206 with destination MAC address fa:16:3e:2d:21:8c (unknown unicast, hits all other nodes on VLAN 206): 15:26:38.702135 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 Among others, this hits overcloud-controller-0: 15:26:38.701898 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 e. overcloud-controller-0 picks up the packet, routes it to VLAN 209 and sends it back to undercloud-0 (known unicast), but does *not* apply SNAT ; what's crazy though is that it rewrites the MAC address to the DNAT MAC fa:16:3e:55:6a:38 15:26:38.702650 fa:16:3e:55:6a:38 > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 I think what goes wrong here is step d), overcloud-compute-0 should answer directly. In [2]: In this case, the only host where I see any traffic is overcloud-controller-0 - direction INGRESS. Which is logical, the last host to send anything with MAC address fa:16:3e:55:6a:38 was overcloud-controller-0, thus manipulating the switch CAM table in such a way that frames are only steered to that controller. And overcloud-controller-0 should not answer to this ARP request, as the instance sits on overcloud-novacompute-0. MAC addresses & IPs & OVN configuration: [3] --------------------------- Further details --------------------------- [1] Sequence 1 - neighbor not known, ARP resolution gets triggered: Order of events: ARP resolution: a. ARP request on VLAN 209 leaves undercloud-0 and hits all controllers and compute nodes (broadcast): 15:26:38.700599 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 46 b. overcloud-compute-0 answers across VLAN 209 and says that 172.31.0.227 is-at fa:16:3e:55:6a:38 (known unicast): 15:26:38.701064 fa:16:3e:55:6a:38 > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Reply 172.31.0.227 is-at fa:16:3e:55:6a:38, length 46 ICMP: c. ICMP echo request on VLAN 209 leaves undercloud-0 and hits overcloud-compute-0 only (known unicast): 15:26:38.701325 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 172.31.0.1 > 172.31.0.227: ICMP echo request, id 12680, seq 1, length 64 d. ICMP overcloud echo request leaves overcloud-novacompute-0 on VLAN 206 with destination MAC address fa:16:3e:2d:21:8c (unknown unicast, hits all other nodes on VLAN 206): 15:26:38.702135 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 Among others, this hits overcloud-controller-0: 15:26:38.701898 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 e. overcloud-controller-0 picks up the packet, routes it to VLAN 209 and sends it back to undercloud-0 (known unicast), but does *not* apply SNAT 15:26:38.702650 fa:16:3e:55:6a:38 > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 I think what goes wrong here is step d), overcloud-compute-0 should answer directly. ~~~ (overcloud) [stack@undercloud-0 overcloud-test]$ sudo ip neigh del 172.31.0.227 dev eth2.209 (overcloud) [stack@undercloud-0 overcloud-test]$ ping 172.31.0.227 -c1 -W1 PING 172.31.0.227 (172.31.0.227) 56(84) bytes of data. --- 172.31.0.227 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms (overcloud) [stack@undercloud-0 overcloud-test]$ (... couple of minutes later ...) (overcloud) [stack@undercloud-0 overcloud-test]$ ip neigh | grep 172.31.0.227 172.31.0.227 dev eth2.209 lladdr fa:16:3e:55:6a:38 STALE ~~~ overcloud-novacompute-0 - IN: ~~~ [root@overcloud-novacompute-0 ~]# tcpdump -nne -i bond_api -Q in -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond_api, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.700599 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 46 15:26:38.701325 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 172.31.0.1 > 172.31.0.227: ICMP echo request, id 12680, seq 1, length 64 ~~~ overcloud-novacompute-0 - OUT: ~~~ [root@overcloud-novacompute-0 ~]# tcpdump -nne -i bond_api -Q out -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond_api, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.701064 fa:16:3e:55:6a:38 > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Reply 172.31.0.227 is-at fa:16:3e:55:6a:38, length 46 15:26:38.702135 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 ~~~ overcloud-novacompute-1 - IN: ~~~ [root@overcloud-novacompute-1 ~]# tcpdump -nne -i bond_api -Q in -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond_api, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.700614 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 46 15:26:38.702191 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 ~~~ overcloud-novacompute-1 - OUT: ~~~ [root@overcloud-novacompute-1 ~]# tcpdump -nne -i bond_api -Q out -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond_api, link-type EN10MB (Ethernet), capture size 262144 bytes ~~~ overcloud-controller-0 - IN: ~~~ [root@overcloud-controller-0 ~]# tcpdump -nne -i ens5 -Q in -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.700239 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 28 15:26:38.701898 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 ~~~ overcloud-controller-0 - OUT: ~~~ [root@overcloud-controller-0 ~]# tcpdump -nne -i ens5 -Q out -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.702650 fa:16:3e:55:6a:38 > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 ~~~ overcloud-controller-1 - IN: ~~~ [root@overcloud-controller-1 ~]# tcpdump -nne -i ens5 -Q in -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.699806 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 28 15:26:38.701520 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 ~~~ overcloud-controller-1 - OUT: ~~~ [root@overcloud-controller-1 ~]# tcpdump -nne -i ens5 -Q out -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes ~~~ overcloud-controller-2 - IN: ~~~ [root@overcloud-controller-2 ~]# tcpdump -nne -i ens5 -Q in -l | egrep -i '172.31.0.227|172.31.0.1' vtcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 15:26:38.700121 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 28 15:26:38.701770 fa:16:3e:0c:91:5f > fa:16:3e:2d:21:8c, ethertype 802.1Q (0x8100), length 102: vlan 206, p 0, ethertype IPv4, 192.168.0.120 > 172.31.0.1: ICMP echo reply, id 12680, seq 1, length 64 ~~~ overcloud-controller-2 - OUT: ~~~ [heat-admin@overcloud-controller-2 ~]$ sudo -i [root@overcloud-controller-2 ~]# tcpdump -nne -i ens5 -Q out -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes ~~~ ------------------------------------------------------------------------------ [2] Sequence 2 - ARP already resolved: ~~~ (overcloud) [stack@undercloud-0 overcloud-test]$ ip neigh | grep 172.31.0.227 172.31.0.227 dev eth2.209 lladdr fa:16:3e:55:6a:38 STALE (overcloud) [stack@undercloud-0 overcloud-test]$ ping 172.31.0.227 -c1 -W1 PING 172.31.0.227 (172.31.0.227) 56(84) bytes of data. --- 172.31.0.227 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms (overcloud) [stack@undercloud-0 overcloud-test]$ ip neigh | grep 172.31.0.227 172.31.0.227 dev eth2.209 FAILED ~~~ In this case, the only host where I see any traffic is: overcloud-controller-0 - IN: ~~~ [root@overcloud-controller-0 ~]# tcpdump -nne -i ens5 -Q in -l | egrep -i '172.31.0.227|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 15:36:59.865081 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 172.31.0.1 > 172.31.0.227: ICMP echo request, id 27837, seq 1, length 64 15:37:05.114855 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 28 15:37:06.138855 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 28 15:37:07.162952 52:54:00:2e:2e:2e > fa:16:3e:55:6a:38, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.227 tell 172.31.0.1, length 28 ^C281 packets captured 475 packets received by filter 0 packets dropped by kernel [root@overcloud-controller-0 ~]# ~~~ Which is logical, the last host to send anything with MAC address fa:16:3e:55:6a:38 was overcloud-controller-0, thus manipulating the switch CAM table in such a way that frames are only steered to that controller. [3] MAC addresses and IP addresses and OVN configuration: undercloud: ~~~ (overcloud) [stack@undercloud-0 overcloud-test]$ ip a ls dev eth2.209 13: eth2.209@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:2e:2e:2e brd ff:ff:ff:ff:ff:ff inet 172.31.0.1/24 scope global eth2.209 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe2e:2e2e/64 scope link valid_lft forever preferred_lft forever ~~~ ~~~ [root@overcloud-controller-0 ~]# export SB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g') [root@overcloud-controller-0 ~]# export NB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g') [root@overcloud-controller-0 ~]# alias ovn-sbctl='sudo docker exec ovn_controller ovn-sbctl --db=$SB' [root@overcloud-controller-0 ~]# alias ovn-nbctl='sudo docker exec ovn_controller ovn-nbctl --db=$NB' [root@overcloud-controller-0 ~]# alias ovn-trace='sudo docker exec ovn_controller ovn-trace --db=$SB' [root@overcloud-controller-0 ~]# ovn-nbctl find NAT type=dnat_and_snat Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. _uuid : 57cc5931-4bf6-4920-8fc5-004d466bad32 external_ids : {"neutron:fip_external_mac"="fa:16:3e:55:6a:38", "neutron:fip_id"="f4257837-4dd1-492a-8ab7-d64e8544565b", "neutron:fip_port_id"="4c922bf3-6964-425c-9908-0fff5ba58b13", "neutron:revision_number"="2", "neutron:router_name"="neutron-5437c444-865f-450e-8746-a71620dffc89"} external_ip : "172.31.0.227" external_mac : "fa:16:3e:55:6a:38" logical_ip : "192.168.0.120" logical_port : "4c922bf3-6964-425c-9908-0fff5ba58b13" options : {} type : dnat_and_snat _uuid : 1ad174e7-25c1-41a6-9047-442bb5ca4679 external_ids : {"neutron:fip_external_mac"="fa:16:3e:0c:68:55", "neutron:fip_id"="2fdcd9a2-a167-4b40-81f3-aa9b40854cbd", "neutron:fip_port_id"="b6f83df7-5f85-4635-8b14-dda3bb6302ce", "neutron:revision_number"="2", "neutron:router_name"="neutron-5437c444-865f-450e-8746-a71620dffc89"} external_ip : "172.31.0.207" external_mac : [] logical_ip : "192.168.0.121" logical_port : "b6f83df7-5f85-4635-8b14-dda3bb6302ce" options : {} type : dnat_and_snat [root@overcloud-controller-0 ~]# ovn-nbctl show neutron-5437c444-865f-450e-8746-a71620dffc89 Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. router 51a83530-80f2-47af-95d5-c9283e1368b5 (neutron-5437c444-865f-450e-8746-a71620dffc89) (aka router) port lrp-76e1604c-8aa9-465d-a3b1-479a3c9e718d mac: "fa:16:3e:3a:09:ee" networks: ["172.31.0.249/24", "2000:10::f816:3eff:fe3a:9ee/64"] gateway chassis: [5ad93788-6411-45e2-b5b1-4ca9e82afdf4 887e3897-8ee3-4266-964a-13477020b6bd 1838f9b8-3326-42ab-9b6b-a5807eb6be74 7c9032c5-8a08-49cf-a0f0-a09e8bbc9b72 3db1b6e4-a681-4a6b-b43d-ca8cd3a3d112] port lrp-ecb2caa2-4966-4ea7-8ba0-930c93a7be78 mac: "fa:16:3e:2d:21:8c" networks: ["192.168.0.1/24"] port lrp-361bba21-614b-456b-a896-fc0bd861f280 mac: "fa:16:3e:28:6e:df" networks: ["192.168.1.1/24"] nat 1ad174e7-25c1-41a6-9047-442bb5ca4679 external ip: "172.31.0.207" logical ip: "192.168.0.121" type: "dnat_and_snat" nat 57cc5931-4bf6-4920-8fc5-004d466bad32 external ip: "172.31.0.227" logical ip: "192.168.0.120" type: "dnat_and_snat" nat 8aa2d070-eb71-4941-bee2-c154283974cd external ip: "172.31.0.249" logical ip: "192.168.1.0/24" type: "snat" nat cd54b836-70ce-4cb7-93fb-067f1bab2a0c external ip: "172.31.0.249" logical ip: "192.168.0.0/24" type: "snat" [root@overcloud-controller-0 ~]# ovn-nbctl lr-nat-list neutron-5437c444-865f-450e-8746-a71620dffc89 Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. TYPE EXTERNAL_IP LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat 172.31.0.207 192.168.0.121 dnat_and_snat 172.31.0.227 192.168.0.120 fa:16:3e:55:6a:38 4c922bf3-6964-425c-9908-0fff5ba58b13 snat 172.31.0.249 192.168.0.0/24 snat 172.31.0.249 192.168.1.0/24 ~~~
I think you're hitting bug 1704596 - the DVR with VLAN doesn't work well and we're working on it. If you think it's really a dup, we can close this one.
Hi, Thanks, I'm currently looking into this. Just for completeness, here's the overcloud setup (networks) that did not work for me. ~~~ (overcloud) [stack@undercloud-0 ~]$ cat ovn_dvr/common/overcloud-test/overcloud-test-ovn-dvr.sh #!/bin/bash PROVIDER_SEGMENTATION_ID_PRIVATE=206 PROVIDER_SEGMENTATION_ID_PRIVATE2=207 PROVIDER_SEGMENTATION_ID_EXTERNAL=209 PROVIDER_PHYSICAL_NETWORK="tenant" PROVIDER_PHYSICAL_NETWORK_EXTERNAL="tenant" (...) DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" source /home/stack/overcloudrc if ! `neutron net-list | grep -q private1`;then neutron net-create private1 --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK --provider:segmentation_id $PROVIDER_SEGMENTATION_ID_PRIVATE --shared --router:external fi if ! `neutron net-list | grep -q private2`;then neutron net-create private2 --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK --provider:segmentation_id $PROVIDER_SEGMENTATION_ID_PRIVATE2 --shared --router:external fi if ! `neutron net-list | grep -q provider1-ovn`;then neutron net-create provider1-ovn --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK_EXTERNAL --provider:segmentation_id $PROVIDER_SEGMENTATION_ID_EXTERNAL --shared --router:external fi if ! `neutron subnet-list | grep -q provider1-ovn-subnet`;then neutron subnet-create --gateway 172.31.0.1 --allocation-pool start=172.31.0.200,end=172.31.0.250 --dns-nameserver 10.11.5.4 --name provider1-ovn-subnet provider1-ovn 172.31.0.0/24 fi if ! `neutron subnet-list | grep -q provider1-ovn-ipv6-subnet`;then neutron subnet-create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --gateway 2000:10::250 --name provider1-ovn-ipv6-subnet provider1-ovn 2000:10::/64 fi if ! `neutron router-list | grep -q router`;then neutron router-create router neutron router-gateway-set router provider1-ovn fi if ! `neutron router-list | grep -q router-ipv6`;then neutron router-create router-ipv6 neutron router-gateway-set router-ipv6 provider1-ovn fi if ! `neutron subnet-list | grep -q private1-subnet`;then neutron subnet-create --name private1-subnet private1 192.168.0.0/24 --allocation-pool start=192.168.0.100,end=192.168.0.150 neutron router-interface-add router private1-subnet fi if ! `neutron subnet-list | grep -q private1-ipv6-subnet`;then neutron subnet-create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --name private1-ipv6-subnet private1 2000:192:168:0::/64 neutron router-interface-add router-ipv6 private1-ipv6-subnet fi if ! `neutron subnet-list | grep -q private2-subnet`;then neutron subnet-create --name private2-subnet private2 192.168.1.0/24 --allocation-pool start=192.168.1.100,end=192.168.1.150 neutron router-interface-add router private2-subnet fi if ! `neutron subnet-list | grep -q private2-ipv6-subnet`;then neutron subnet-create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --name private2-ipv6-subnet private2 2000:192:168:1::/64 neutron router-interface-add router-ipv6 private1-ipv6-subnet fi /bin/bash ${DIR}/cirros-image-create.sh /bin/bash ${DIR}/rhel-image-create.sh project_id=`openstack project list | awk '/admin/ {print $2}'` security_group_id=`openstack security group list | grep $project_id | awk '{print $2}'` openstack security group rule create ${security_group_id} --protocol icmp --remote-ip 0.0.0.0/0 openstack security group rule create ${security_group_id} --protocol tcp --dst-port 1:65535 --remote-ip 0.0.0.0/0 openstack security group rule create ${security_group_id} --protocol udp --dst-port 1:65535 --remote-ip 0.0.0.0/0 openstack security group rule create ${security_group_id} --ethertype IPv6 --protocol icmp --remote-ip ::/0 openstack security group rule create ${security_group_id} --ethertype IPv6 --protocol tcp --dst-port 1:65535 --remote-ip ::/0 openstack security group rule create ${security_group_id} --ethertype IPv6 --protocol udp --dst-port 1:65535 --remote-ip ::/0 if ! `nova keypair-list | grep -q id_rsa`;then nova keypair-add --pub-key ~/.ssh/id_rsa.pub id_rsa fi if ! `nova flavor-list | grep -q m1.tiny`;then nova flavor-create m1.tiny auto 512 8 1 fi if ! `nova flavor-list | grep -q m1.small`;then nova flavor-create m1.small auto 1024 16 1 fi /bin/bash ${DIR}/overcloud-test-ovn-spawn-rhel.sh /bin/bash ${DIR}/overcloud-test-ovn-spawn-cirros.sh ~~~ ------------------------ ~~~ (overcloud) [stack@undercloud-0 ~]$ cat ovn_dvr/common/overcloud-test/overcloud-test-ovn-spawn-rhel.sh #!/bin/bash PROVIDER_SEGMENTATION_ID_PRIVATE=206 PROVIDER_SEGMENTATION_ID_PRIVATE2=207 PROVIDER_SEGMENTATION_ID_EXTERNAL=209 PROVIDER_PHYSICAL_NETWORK="tenant" PROVIDER_PHYSICAL_NETWORK_EXTERNAL="tenant" PROVIDER_NETWORK="provider1-ovn" CIRROS_INSTANCE_COUNT=1 RHEL_INSTANCE_COUNT=1 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" source /home/stack/overcloudrc PROVIDERNETID=$(openstack network show $PROVIDER_NETWORK -c id -f value) NETID=$(neutron net-list | grep private1 | awk '{print $2}') for i in `seq 1 $RHEL_INSTANCE_COUNT`;do uuid=$(uuidgen | cut -b 1-8) openstack floating ip create $PROVIDERNETID nova boot --nic net-id=$NETID --image rhel --flavor m1.small --key-name id_rsa rhel-test-${uuid} FLOATINGIP=$(openstack floating ip list --network $PROVIDERNETID --long | grep DOWN | awk '{print $2}' | head -1) sleep 10 openstack server add floating ip rhel-test-${uuid} $FLOATINGIP done ~~~
And updated version: (undercloud) [stack@undercloud-0 ~]$ cat ovn_dvr/common/overcloud-test/overcloud-test-ovn-dvr.sh #!/bin/bash PROVIDER_SEGMENTATION_ID_PRIVATE=206 PROVIDER_SEGMENTATION_ID_PRIVATE2=207 PROVIDER_SEGMENTATION_ID_EXTERNAL=209 PROVIDER_PHYSICAL_NETWORK="tenant" PROVIDER_PHYSICAL_NETWORK_EXTERNAL="tenant" (...) DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" source /home/stack/overcloudrc if ! `neutron net-list | grep -q private1`;then neutron net-create private1 --provider:network_type geneve # neutron net-create private1 --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK --provider:segmentation_id $PROVIDER_SEGMENTATION_ID_PRIVATE --shared --router:external fi if ! `neutron net-list | grep -q private2`;then neutron net-create private2 --provider:network_type geneve # neutron net-create private2 --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK --provider:segmentation_id $PROVIDER_SEGMENTATION_ID_PRIVATE2 --shared --router:external fi if ! `neutron net-list | grep -q provider1-ovn`;then neutron net-create provider1-ovn --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK_EXTERNAL --provider:segmentation_id $PROVIDER_SEGMENTATION_ID_EXTERNAL --shared --router:external fi if ! `neutron subnet-list | grep -q provider1-ovn-subnet`;then neutron subnet-create --gateway 172.31.0.1 --allocation-pool start=172.31.0.200,end=172.31.0.250 --dns-nameserver 10.11.5.4 --name provider1-ovn-subnet provider1-ovn 172.31.0.0/24 fi if ! `neutron subnet-list | grep -q provider1-ovn-ipv6-subnet`;then neutron subnet-create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --gateway 2000:10::250 --name provider1-ovn-ipv6-subnet provider1-ovn 2000:10::/64 fi if ! `neutron router-list | grep -q router`;then neutron router-create router neutron router-gateway-set router provider1-ovn fi if ! `neutron router-list | grep -q router-ipv6`;then neutron router-create router-ipv6 neutron router-gateway-set router-ipv6 provider1-ovn fi if ! `neutron subnet-list | grep -q private1-subnet`;then neutron subnet-create --name private1-subnet private1 192.168.0.0/24 --allocation-pool start=192.168.0.100,end=192.168.0.150 neutron router-interface-add router private1-subnet fi if ! `neutron subnet-list | grep -q private1-ipv6-subnet`;then neutron subnet-create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --name private1-ipv6-subnet private1 2000:192:168:0::/64 neutron router-interface-add router-ipv6 private1-ipv6-subnet fi if ! `neutron subnet-list | grep -q private2-subnet`;then neutron subnet-create --name private2-subnet private2 192.168.1.0/24 --allocation-pool start=192.168.1.100,end=192.168.1.150 neutron router-interface-add router private2-subnet fi if ! `neutron subnet-list | grep -q private2-ipv6-subnet`;then neutron subnet-create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --name private2-ipv6-subnet private2 2000:192:168:1::/64 neutron router-interface-add router-ipv6 private1-ipv6-subnet fi /bin/bash ${DIR}/cirros-image-create.sh /bin/bash ${DIR}/rhel-image-create.sh project_id=`openstack project list | awk '/admin/ {print $2}'` security_group_id=`openstack security group list | grep $project_id | awk '{print $2}'` openstack security group rule create ${security_group_id} --protocol icmp --remote-ip 0.0.0.0/0 openstack security group rule create ${security_group_id} --protocol tcp --dst-port 1:65535 --remote-ip 0.0.0.0/0 openstack security group rule create ${security_group_id} --protocol udp --dst-port 1:65535 --remote-ip 0.0.0.0/0 openstack security group rule create ${security_group_id} --ethertype IPv6 --protocol icmp --remote-ip ::/0 openstack security group rule create ${security_group_id} --ethertype IPv6 --protocol tcp --dst-port 1:65535 --remote-ip ::/0 openstack security group rule create ${security_group_id} --ethertype IPv6 --protocol udp --dst-port 1:65535 --remote-ip ::/0 if ! `nova keypair-list | grep -q id_rsa`;then nova keypair-add --pub-key ~/.ssh/id_rsa.pub id_rsa fi if ! `nova flavor-list | grep -q m1.tiny`;then nova flavor-create m1.tiny auto 512 8 1 fi if ! `nova flavor-list | grep -q m1.small`;then nova flavor-create m1.small auto 1024 16 1 fi /bin/bash ${DIR}/overcloud-test-ovn-spawn-rhel.sh /bin/bash ${DIR}/overcloud-test-ovn-spawn-cirros.sh
Making the comments public as this is my lab environment and doesn't contain any customer data. After the change in comment 5 (and after migrating or restarting the instance, see https://bugzilla.redhat.com/show_bug.cgi?id=1830734 ) (overcloud) [stack@undercloud-0 ~]$ sudo ip neigh del 172.31.0.201 dev eth2.209 (overcloud) [stack@undercloud-0 ~]$ ping 172.31.0.201 -c1 -W1 PING 172.31.0.201 (172.31.0.201) 56(84) bytes of data. 64 bytes from 172.31.0.201: icmp_seq=1 ttl=63 time=1.96 ms --- 172.31.0.201 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.961/1.961/1.961/0.000 ms (overcloud) [stack@undercloud-0 ~]$ [root@overcloud-controller-0 ~]# tcpdump -nne -i ens5 -l | egrep -i '172.31.0.201|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 14:14:29.332200 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.201 tell 172.31.0.1, length 28 [root@overcloud-controller-1 ~]# tcpdump -nne -i ens5 -l | egrep -i '172.31.0.201|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 14:14:29.332322 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.201 tell 172.31.0.1, length 28 [root@overcloud-controller-2 ~]# tcpdump -nne -i ens5 -l | egrep -i '172.31.0.201|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes 14:14:29.332151 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.201 tell 172.31.0.1, length 28 [root@overcloud-novacompute-0 ~]# tcpdump -nne -i bond_api -l | egrep -i '172.31.0.201|172.31.0.1' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond_api, link-type EN10MB (Ethernet), capture size 262144 bytes 14:14:29.332654 52:54:00:2e:2e:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Request who-has 172.31.0.201 tell 172.31.0.1, length 46 14:14:29.333065 fa:16:3e:9b:7b:fb > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 64: vlan 209, p 0, ethertype ARP, Reply 172.31.0.201 is-at fa:16:3e:9b:7b:fb, length 46 14:14:29.333326 52:54:00:2e:2e:2e > fa:16:3e:9b:7b:fb, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 172.31.0.1 > 172.31.0.201: ICMP echo request, id 21173, seq 1, length 64 14:14:29.334233 fa:16:3e:9b:7b:fb > 52:54:00:2e:2e:2e, ethertype 802.1Q (0x8100), length 102: vlan 209, p 0, ethertype IPv4, 172.31.0.201 > 172.31.0.1: ICMP echo reply, id 21173, seq 1, length 64 *** This bug has been marked as a duplicate of bug 1704596 ***
So thanks. Indeed, OVN-DVR currently does not seem to work at all if the tenant network is of type VLAN: https://bugzilla.redhat.com/show_bug.cgi?id=1704596 On the other hand, if the provider network is of type VLAN, then we hit: https://bugzilla.redhat.com/show_bug.cgi?id=1830734