Bug 2037332

Description of problem:
Client has processes creating a lot of instances at the same time.
Where previously on setup with OVS it was working fine, now with OVN it can take up to 600 seconds for instances to become available.

When max_concurrent_builds is set to 2 and up, a lot of error are happening like:
openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://$URL:13774/v2.1/servers/$uuid, problems. Please try again later.: 503 Service Unavailable: The server is temporarily unable to service your: Service Unavailable: request due to maintenance downtime or capacity

In other environments with OVS, max_concurrent_builds is set to 3 and it works just fine.

Also a LOT of errors like those are seen in the logs /var/lib/containers/neutron/server.log
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance [req-c1605008-f3b7-4014-9665-7ea4455a457d - - - - -] Maintenance task: Failed to fix deleted resource $uuid (type: subnets): KeyError: 'uuid'
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 379, in check_for_inconsistencies
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance     self._ovn_client.delete_subnet(row.resource_uuid)
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 2044, in delete_subnet
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance     self._remove_subnet_dhcp_options(subnet_id, txn)
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1910, in _remove_subnet_dhcp_options
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance     dhcp_options['subnet']['uuid']))
2021-12-23 16:26:46.481 48 ERROR networking_ovn.common.maintenance KeyError: 'uuid'

We need your help into identifying the bottleneck here.

Anything else you need please let me know.

Thanks!

Version-Release number of selected component (if applicable):
OSP16.2.0

How reproducible:
100%

Steps to Reproduce:
1. Try to create a lot of instances at the same time
2.
3.

Actual results:
Slow instance creation.

Expected results:
Same speed as when using OVS network protocol.

Additional info:
We have sosreport with logs.