Dumitru has logged into the environment and found the root cause for this issue.

Neutron is setting "unknown" attribute in controller's logical port

    port dcf3cf5a-3e48-4082-bc7f-20976df0d398 (aka octavia-health-manager-controller-0.redhat.local-listen-port)                      
        addresses: ["fa:16:3e:f8:dd:30 172.24.1.216", "unknown"]

Because of this, core OVN is not adding local arp reply flow in the compute node. In the absence of local ARP reply flow,  ovs-vswitchd tried to broadcast the packet and hitting 4k limit as this network "lb-mgmt-net" has more than 5000 ports (i.e amphorae  VMs).  

2021-05-18T10:21:13.903Z|00419|ofproto_dpif_xlate(handler356)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,tun
_id=0x1,tun_src=0.0.0.0,tun_dst=172.17.2.47,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=0,tun_ttl=64,tun_erspa
n_ver=0,tun_flags=df|csum|key,tun_metadata0=0x18608000,in_port=1941,vlan_tci=0x0000,dl_src=fa:16:3e:20:71:28,dl_dst=ff:ff:ff:ff:ff:ff,a
rp_spa=172.24.19.180,arp_tpa=172.24.1.216,arp_op=1,arp_sha=fa:16:3e:20:71:28,arp_tha=00:00:00:00:00:00                                
2021-05-18T10:22:15.629Z|00740|ofproto_dpif_xlate(handler355)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,tun
_id=0x1,tun_src=0.0.0.0,tun_dst=172.17.2.47,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=0,tun_ttl=64,tun_erspa
n_ver=0,tun_flags=df|csum|key,tun_metadata0=0x149d8000,in_port=1480,vlan_tci=0x0000,dl_src=fa:16:3e:04:9f:a1,dl_dst=ff:ff:ff:ff:ff:ff,a
rp_spa=172.24.21.111,arp_tpa=172.24.0.151,arp_op=1,arp_sha=fa:16:3e:04:9f:a1,arp_tha=00:00:00:00:00:00     

If the neutron removes the "unknown" attribute, OVN can add the local arp reply flow.

As this neutron network is created by administrators  (tripleo is creating this for now), if any external ports (not controlled by neutron) try to reach the VM on this network, they will definitely hit this 4k resubmit issue. So OVN has the limitation for larger broadcast domain. 

We reached a limitation (5000) in Octavia on scaling the load balancers.

May be this can be the reason we see high cpu usage for unknown ports on lb-mgmt-net as reported in in https://bugzilla.redhat.com/show_bug.cgi?id=1961162

As we have discussed on the email chain around this issue, there is no limitation in Octavia that is being hit here. 5,000 is not an Octavia limit.

As was discussed above, Dumitru (thank you), found the issue as to why OVN was dropping the ARP requests from newly created nova VM instances.

At some number of ports in a neutron network, OVN will start dropping broadcast traffic to ports.
From what we understand, there is a flow chain limit in OVN that is being reached, where rules for newly created ports may stop being added to the chain. This will cause those new ports to be unable to communicate with any port configured with port security disabled on the network. This applies to IPv4 networks and was not tested with IPv6 (which does not use ARP).

When a port in neutron has port security disabled, the compute node ARP responder in OVN is not configured for the port (this is the "unknown" setting 
 in ovn-nb listed above). This leads to the ARP request from nova VMs to need to cross the OVN fabric, and getting dropped by the OVN limitation.

The workaround applied in this test environment was to add a security group and enable port security on the controller o-hm0 ports in neutron. Once that was done, new VMs booted in nova could communicate with the controllers again and new load balancers could be created.

With that said, maybe this should be closed as NOTABUG given it is a designed limit? Maybe it should remain open for this limit to be documented in the OVN OSP documentation?

(In reply to Michael Johnson from comment #4)
> As we have discussed on the email chain around this issue, there is no
> limitation in Octavia that is being hit here. 5,000 is not an Octavia limit.
> 
> As was discussed above, Dumitru (thank you), found the issue as to why OVN
> was dropping the ARP requests from newly created nova VM instances.
> 
> At some number of ports in a neutron network, OVN will start dropping
> broadcast traffic to ports.
> From what we understand, there is a flow chain limit in OVN that is being
> reached, where rules for newly created ports may stop being added to the
> chain. This will cause those new ports to be unable to communicate with any
> port configured with port security disabled on the network. This applies to
> IPv4 networks and was not tested with IPv6 (which does not use ARP).
> 
> When a port in neutron has port security disabled, the compute node ARP
> responder in OVN is not configured for the port (this is the "unknown"
> setting 
>  in ovn-nb listed above). This leads to the ARP request from nova VMs to
> need to cross the OVN fabric, and getting dropped by the OVN limitation.
> 
> The workaround applied in this test environment was to add a security group
> and enable port security on the controller o-hm0 ports in neutron.

Octavia team opened a private bug  https://bugzilla.redhat.com/show_bug.cgi?id=1961845 to apply this workaround.


 Once that
> was done, new VMs booted in nova could communicate with the controllers
> again and new load balancers could be created.
> 
> With that said, maybe this should be closed as NOTABUG given it is a
> designed limit? Maybe it should remain open for this limit to be documented
> in the OVN OSP documentation?

Hi,

The change has been made to the RHOSP 16.1 "Networking Guide." Customers can see
this change here:

   https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/networking_guide/index#con_limit-nonsecure-port-ovn_networking-concepts

Thanks,
--Greg