Bug 2012932

Description of problem:
This issue was initially reproduced by the OSP job that is triggered with intermediate OVN versions (I mean, it's not officially released yet).
The job: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp_3net-ipv4-geneve-composable-vlan-provider-network/10/
The intermediate ovn version: ovn-2021.21.09.0-5



I have reproduced this issue in an openstack environment twice using the same method, although I don't understand some things yet:
- I ran two tempest tests in parallel that are very similar. They created a VM connected to a tenant network and with a FIP. I added a breakpoint before they try to connect to the VM via ssh to their FIPs.
- Both times, one of the VMs can be accessed via ssh and the other one can't. The problem is in the metadata. The faulty VM couldn't connect to the metadata and couldn't obtain the authorized public key.
- Some commands executed at the VM that works fine (169.254.169.254 is the metadata IP):
[root@tempest-broadcast-sender-258843568 ~]# ip r
default via 10.100.0.1 dev eth0 proto dhcp metric 100
10.100.0.0/28 dev eth0 proto kernel scope link src 10.100.0.14 metric 100
169.254.169.254 via 10.100.0.2 dev eth0 proto dhcp metric 100
[root@tempest-broadcast-sender-258843568 ~]# ping 10.100.0.2 -c1                                                                                                                                                                            
PING 10.100.0.2 (10.100.0.2) 56(84) bytes of data.
64 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=3.90 ms

--- 10.100.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.899/3.899/3.899/0.000 ms
[root@tempest-broadcast-sender-258843568 ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.100.0.2               ether   fa:16:3e:63:dd:56   C                     eth0
10.100.0.1               ether   fa:16:3e:80:81:fc   C                     eth0

- The same commands, executed at the faulty VM (169.254.169.254 is the metadata IP):
[root@localhost ~]# ip r
default via 10.100.0.1 dev eth0 proto dhcp metric 100
10.100.0.0/28 dev eth0 proto kernel scope link src 10.100.0.9 metric 100
169.254.169.254 via 10.100.0.2 dev eth0 proto dhcp metric 100
[root@localhost ~]# ping 10.100.0.2 -c1                                                                                                                                                                                                      
PING 10.100.0.2 (10.100.0.2) 56(84) bytes of data.
From 10.100.0.9 icmp_seq=1 Destination Host Unreachable

--- 10.100.0.2 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

[root@localhost ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.100.0.2                       (incomplete)                              eth0
10.100.0.1               ether   fa:16:3e:7d:b1:a4   C                     eth0

- The arp table from the "successful" ovnmetadata namespace includes an entry for the "successful" VM:
[root@compute-0 ~]# ip netns e ovnmeta-cba5f3c1-397c-47cb-b1af-da45a73e1b22 arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.100.0.14              ether   fa:16:3e:4a:0b:9c   C                     tapcba5f3c1-31

- The arp table from the "faulty" ovnmetadata namespace is empty:
[root@compute-1 ~]# ip netns e ovnmeta-12ff7d70-03a1-49c5-8bcd-98e28f0a4858 arp -n
[root@compute-1 ~]# 

- I don't understand why it affects only one VM/ovnmetadata namespace. Maybe a race condition? I fixed the problem by sending a ping from the "faulty" ovnmetadata namespace to the faulty VM.







Version-Release number of selected component (if applicable):
ovn-2021.21.09.0-5

How reproducible:
not always reproducible, but it affected 26 tempest tests from the job

Steps to Reproduce:
1. run tempest tests - I didn't reproduce it manually, but I guess creating several VMs on different tenant networks should reproduce the issue too


Additional info: