Bug 1834417

Summary:	OSP10 - Tenant VM is able to arping tenant's network interface even though they are not using same VLANID
Product:	Red Hat OpenStack	Reporter:	David Vallee Delisle <dvd>
Component:	python-os-vif	Assignee:	Slawek Kaplonski <skaplons>
Status:	CLOSED EOL	QA Contact:	Eran Kuris <ekuris>
Severity:	high	Docs Contact:
Priority:	high
Version:	10.0 (Newton)	CC:	aconole, apevec, chrisw, coldford, elicohen, jjoyce, jlibosva, jschluet, jzaher, njohnston, nlevinki, ralonsoh, rhos-maint, scohen, skaplons, slinaber, tvignaud, vcojot
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-os-vif-1.2.1-5.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-07 09:45:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Vallee Delisle 2020-05-11 16:46:34 UTC

Description of problem:
From a VM, if we arping -D -I eth0 172.18.0.21 where 172.18.0.21 is the compute's TenantNetwork interface IP, we get a reply.

This behavior is not present in OSP13.

We reproduced this issue in an internal lab with OSP10, latest version.

Version-Release number of selected component (if applicable):
openvswitch-2.9.0-101.el7fdp.x86_64

How reproducible:
All the time

Steps to Reproduce:
1. Spawn a VM with a tenant network, with a different VLANID than the TenantNetwork on the overcloud
2. arping -D -I eth0 172.18.0.xx (compute's TenantNetwork IP)


Actual results:
We get a reply

Expected results:
We shouldn't get a reply

Additional info:

Customer is reporting that their tenants are using the same address space as their overcloud's TenantNetwork. When the Guest OS is bringing the interfaces up, and it's using the same IP as the host, the interface is not getting up because of IP conflict (even though they are not on the same VLAN).

When we look at the OSP10[1] flows, compare them to OSP13[2] flows [3], we see that they are not built the same way.

We would like to understand what changed this behavior and possibly, have a workaround for OSP10.



[1]
~~~
[root@ess10z9-scpu-0 ~]# ovs-ofctl dump-flows br-int
 cookie=0x8beacdb4759d5d1d, duration=176518.876s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port="qvoae54803e-b0",icmp_type=136 actions=resubmit(,24)
 cookie=0x8beacdb4759d5d1d, duration=176518.857s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port="qvo36beade1-d9",icmp_type=136 actions=resubmit(,24)
 cookie=0x8beacdb4759d5d1d, duration=176517.103s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port="qvo7394136d-b4",icmp_type=136 actions=resubmit(,24)
 cookie=0x8beacdb4759d5d1d, duration=176518.873s, table=0, n_packets=192, n_bytes=8064, priority=10,arp,in_port="qvoae54803e-b0" actions=resubmit(,24)
 cookie=0x8beacdb4759d5d1d, duration=176518.854s, table=0, n_packets=4, n_bytes=168, priority=10,arp,in_port="qvo36beade1-d9" actions=resubmit(,24)
 cookie=0x8beacdb4759d5d1d, duration=176517.100s, table=0, n_packets=189, n_bytes=7938, priority=10,arp,in_port="qvo7394136d-b4" actions=resubmit(,24)
 cookie=0x8beacdb4759d5d1d, duration=176525.102s, table=0, n_packets=0, n_bytes=0, priority=2,in_port="int-br-external" actions=drop
 cookie=0x8beacdb4759d5d1d, duration=176525.054s, table=0, n_packets=0, n_bytes=0, priority=2,in_port="int-br-tenant" actions=drop
 cookie=0x8beacdb4759d5d1d, duration=176518.880s, table=0, n_packets=5461, n_bytes=473978, priority=9,in_port="qvoae54803e-b0" actions=resubmit(,25)
 cookie=0x8beacdb4759d5d1d, duration=176518.861s, table=0, n_packets=9, n_bytes=746, priority=9,in_port="qvo36beade1-d9" actions=resubmit(,25)
 cookie=0x8beacdb4759d5d1d, duration=176517.108s, table=0, n_packets=5558, n_bytes=482624, priority=9,in_port="qvo7394136d-b4" actions=resubmit(,25)
 cookie=0x8beacdb4759d5d1d, duration=176519.333s, table=0, n_packets=24580, n_bytes=3988356, priority=3,in_port="int-br-tenant",dl_vlan=1170 actions=mod_vlan_vid:1,NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176517.315s, table=0, n_packets=5746, n_bytes=517873, priority=3,in_port="int-br-tenant",dl_vlan=1140 actions=mod_vlan_vid:2,NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176525.153s, table=0, n_packets=10, n_bytes=876, priority=0 actions=NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176525.157s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x8beacdb4759d5d1d, duration=176518.877s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port="qvoae54803e-b0",icmp_type=136,nd_target=fe80::f816:3eff:fed1:bbb0 actions=NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176518.858s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port="qvo36beade1-d9",icmp_type=136,nd_target=fe80::f816:3eff:fe3f:f58c actions=NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176517.106s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port="qvo7394136d-b4",icmp_type=136,nd_target=fe80::f816:3eff:fe38:f16f actions=NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176518.874s, table=24, n_packets=192, n_bytes=8064, priority=2,arp,in_port="qvoae54803e-b0",arp_spa=172.18.0.24 actions=resubmit(,25)
 cookie=0x8beacdb4759d5d1d, duration=176518.855s, table=24, n_packets=2, n_bytes=84, priority=2,arp,in_port="qvo36beade1-d9",arp_spa=172.18.0.22 actions=resubmit(,25)
 cookie=0x8beacdb4759d5d1d, duration=176517.102s, table=24, n_packets=189, n_bytes=7938, priority=2,arp,in_port="qvo7394136d-b4",arp_spa=192.168.88.157 actions=resubmit(,25)
 cookie=0x8beacdb4759d5d1d, duration=176525.150s, table=24, n_packets=2, n_bytes=84, priority=0 actions=drop
 cookie=0x8beacdb4759d5d1d, duration=176518.883s, table=25, n_packets=5653, n_bytes=482042, priority=2,in_port="qvoae54803e-b0",dl_src=fa:16:3e:d1:bb:b0 actions=NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176518.865s, table=25, n_packets=8, n_bytes=656, priority=2,in_port="qvo36beade1-d9",dl_src=fa:16:3e:3f:f5:8c actions=NORMAL
 cookie=0x8beacdb4759d5d1d, duration=176517.113s, table=25, n_packets=5747, n_bytes=490562, priority=2,in_port="qvo7394136d-b4",dl_src=fa:16:3e:38:f1:6f actions=NORMAL

~~~

[2]
~~~
[root@ess13latest-scpu-0 network-scripts]# ovs-ofctl dump-flows br-int
 cookie=0xa3ed5c3f07f4f47c, duration=575529.523s, table=0, n_packets=0, n_bytes=0, priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
 cookie=0xa3ed5c3f07f4f47c, duration=316755.551s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port="qvo7054f143-33",icmp_type=136 actions=resubmit(,24)
 cookie=0xa3ed5c3f07f4f47c, duration=316389.489s, table=0, n_packets=1, n_bytes=86, priority=10,icmp6,in_port="qvo88365e20-85",icmp_type=136 actions=resubmit(,24)
 cookie=0xa3ed5c3f07f4f47c, duration=316755.547s, table=0, n_packets=128, n_bytes=5376, priority=10,arp,in_port="qvo7054f143-33" actions=resubmit(,24)
 cookie=0xa3ed5c3f07f4f47c, duration=316389.485s, table=0, n_packets=731, n_bytes=30702, priority=10,arp,in_port="qvo88365e20-85" actions=resubmit(,24)
 cookie=0xa3ed5c3f07f4f47c, duration=575529.493s, table=0, n_packets=0, n_bytes=0, priority=2,in_port="int-br-external" actions=drop
 cookie=0xa3ed5c3f07f4f47c, duration=575529.460s, table=0, n_packets=63, n_bytes=8994, priority=2,in_port="int-br-tenant" actions=drop
 cookie=0xa3ed5c3f07f4f47c, duration=316755.556s, table=0, n_packets=197, n_bytes=29578, priority=9,in_port="qvo7054f143-33" actions=resubmit(,25)
 cookie=0xa3ed5c3f07f4f47c, duration=316389.493s, table=0, n_packets=197, n_bytes=31510, priority=9,in_port="qvo88365e20-85" actions=resubmit(,25)
 cookie=0xa3ed5c3f07f4f47c, duration=316757.633s, table=0, n_packets=0, n_bytes=0, priority=3,in_port="int-br-tenant",dl_vlan=1234 actions=mod_vlan_vid:2,resubmit(,60)
 cookie=0xa3ed5c3f07f4f47c, duration=575529.529s, table=0, n_packets=174, n_bytes=12976, priority=0 actions=resubmit(,60)
 cookie=0xa3ed5c3f07f4f47c, duration=575529.531s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0xa3ed5c3f07f4f47c, duration=316755.554s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port="qvo7054f143-33",icmp_type=136,nd_target=fe80::f816:3eff:fe69:ecb4 actions=resubmit(,60)
 cookie=0xa3ed5c3f07f4f47c, duration=316389.491s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port="qvo88365e20-85",icmp_type=136,nd_target=fe80::f816:3eff:fee0:db8f actions=resubmit(,60)
 cookie=0xa3ed5c3f07f4f47c, duration=316755.549s, table=24, n_packets=29, n_bytes=1218, priority=2,arp,in_port="qvo7054f143-33",arp_spa=172.18.0.13 actions=resubmit(,25)
 cookie=0xa3ed5c3f07f4f47c, duration=316389.487s, table=24, n_packets=16, n_bytes=672, priority=2,arp,in_port="qvo88365e20-85",arp_spa=172.18.0.14 actions=resubmit(,25)
 cookie=0xa3ed5c3f07f4f47c, duration=575529.525s, table=24, n_packets=815, n_bytes=34274, priority=0 actions=drop
 cookie=0xa3ed5c3f07f4f47c, duration=316755.561s, table=25, n_packets=226, n_bytes=30796, priority=2,in_port="qvo7054f143-33",dl_src=fa:16:3e:69:ec:b4 actions=resubmit(,60)
 cookie=0xa3ed5c3f07f4f47c, duration=316389.498s, table=25, n_packets=211, n_bytes=32002, priority=2,in_port="qvo88365e20-85",dl_src=fa:16:3e:e0:db:8f actions=resubmit(,60)
 cookie=0xa3ed5c3f07f4f47c, duration=575529.527s, table=60, n_packets=882, n_bytes=90412, priority=3 actions=NORMAL
~~~

[3]
~~~
[root@ess10z9-scpu-0 ~]# diff -W 220 -y <(ovs-ofctl dump-flows br-int | cut -f 4,9- -d " ") <(cut -f 4,7- -d " " osp13.flows)
                                                                                                             |  table=0, priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
table=0, priority=10,icmp6,in_port=5,icmp_type=136 actions=resubmit(,24)                                     |  table=0, priority=10,icmp6,in_port="qvo7054f143-33",icmp_type=136 actions=resubmit(,24)
table=0, priority=10,icmp6,in_port=7,icmp_type=136 actions=resubmit(,24)                                     |  table=0, priority=10,icmp6,in_port="qvo88365e20-85",icmp_type=136 actions=resubmit(,24)
table=0, priority=10,icmp6,in_port=9,icmp_type=136 actions=resubmit(,24)                                     |  table=0, priority=10,arp,in_port="qvo7054f143-33" actions=resubmit(,24)
table=0, priority=10,arp,in_port=5 actions=resubmit(,24)                                                     |  table=0, priority=10,arp,in_port="qvo88365e20-85" actions=resubmit(,24)
table=0, priority=10,arp,in_port=7 actions=resubmit(,24)                                                     |  table=0, priority=2,in_port="int-br-external" actions=drop
table=0, priority=10,arp,in_port=9 actions=resubmit(,24)                                                     |  table=0, priority=2,in_port="int-br-tenant" actions=drop
table=0, priority=2,in_port=1 actions=drop                                                                   |  table=0, priority=9,in_port="qvo7054f143-33" actions=resubmit(,25)
table=0, priority=2,in_port=2 actions=drop                                                                   |  table=0, priority=9,in_port="qvo88365e20-85" actions=resubmit(,25)
table=0, priority=9,in_port=5 actions=resubmit(,25)                                                          |  table=0, priority=3,in_port="int-br-tenant",dl_vlan=1234 actions=mod_vlan_vid:2,resubmit(,60)
table=0, priority=9,in_port=7 actions=resubmit(,25)                                                          |  table=0, priority=0 actions=resubmit(,60)
table=0, priority=9,in_port=9 actions=resubmit(,25)                                                          <
table=0, priority=3,in_port=2,dl_vlan=1170 actions=mod_vlan_vid:1,NORMAL                                     <
table=0, priority=3,in_port=2,dl_vlan=1140 actions=mod_vlan_vid:2,NORMAL                                     <
table=0, priority=0 actions=NORMAL                                                                           <
table=23, priority=0 actions=drop                                                                               table=23, priority=0 actions=drop
table=24, priority=2,icmp6,in_port=5,icmp_type=136,nd_target=fe80::f816:3eff:fed1:bbb0 actions=NORMAL        |  table=24, priority=2,icmp6,in_port="qvo7054f143-33",icmp_type=136,nd_target=fe80::f816:3eff:fe69:ecb4 action
table=24, priority=2,icmp6,in_port=7,icmp_type=136,nd_target=fe80::f816:3eff:fe3f:f58c actions=NORMAL        |  table=24, priority=2,icmp6,in_port="qvo88365e20-85",icmp_type=136,nd_target=fe80::f816:3eff:fee0:db8f action
table=24, priority=2,icmp6,in_port=9,icmp_type=136,nd_target=fe80::f816:3eff:fe38:f16f actions=NORMAL        |  table=24, priority=2,arp,in_port="qvo7054f143-33",arp_spa=172.18.0.13 actions=resubmit(,25)
table=24, priority=2,arp,in_port=5,arp_spa=172.18.0.24 actions=resubmit(,25)                                 |  table=24, priority=2,arp,in_port="qvo88365e20-85",arp_spa=172.18.0.14 actions=resubmit(,25)
table=24, priority=2,arp,in_port=7,arp_spa=172.18.0.22 actions=resubmit(,25)                                 <
table=24, priority=2,arp,in_port=9,arp_spa=192.168.88.157 actions=resubmit(,25)                              <
table=24, priority=0 actions=drop                                                                               table=24, priority=0 actions=drop
table=25, priority=2,in_port=5,dl_src=fa:16:3e:d1:bb:b0 actions=NORMAL                                       |  table=25, priority=2,in_port="qvo7054f143-33",dl_src=fa:16:3e:69:ec:b4 actions=resubmit(,60)
table=25, priority=2,in_port=7,dl_src=fa:16:3e:3f:f5:8c actions=NORMAL                                       |  table=25, priority=2,in_port="qvo88365e20-85",dl_src=fa:16:3e:e0:db:8f actions=resubmit(,60)
table=25, priority=2,in_port=9,dl_src=fa:16:3e:38:f1:6f actions=NORMAL                                       |  table=60, priority=3 actions=NORMAL
~~~

Comment 2 Aaron Conole 2020-05-11 17:44:42 UTC

Open vSwitch team doesn't program the flows - something in neutron or other component does.