Bug 2119534

Description of problem:
Recently we attempted to run neutron/ovn db sync and encountered an error:

_group_handler-257722614'] for port, before_update _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2022-08-13 06:18:14.985 19969 CRITICAL neutron_ovn_db_sync_util [req-42769951-d311-4352-9a83-dc426ba75215 - - - - -] Unhandled error: neutron_lib.exceptions.IpAddressGenerationFailure: No more IP addresses available on network
68ef131a-5961-439f-8812-42b553e0d8f8.
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/neutron/db/ipam_pluggable_backend.py", line 138, in _ipam_allocate_ips
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util     ip_address, subnet_id = ipam_allocator.allocate(ip_request)
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/neutron/ipam/subnet_alloc.py", line 240, in allocate
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util     raise ipam_exc.IpAddressGenerationFailureAllSubnets()
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util neutron.ipam.exceptions.IpAddressGenerationFailureAllSubnets: No more IP addresses available.
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util
2022-08-13 06:18:14.985 19969 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:

The culprit is this network:

 openstack port list --network 68ef131a-5961-439f-8812-42b553e0d8f8 --long
+--------------------------------------+------------------------------+-------------------+----------------------------------------------------------------------------+--------+-----------------+--------------+------+
| ID                                   | Name                         | MAC Address       | Fixed IP Addresses                                                         | Status | Security Groups | Device Owner | Tags |
+--------------------------------------+------------------------------+-------------------+----------------------------------------------------------------------------+--------+-----------------+--------------+------+
| 0ea2ca2a-8c6a-4d5b-8874-07c7bf682bed | xxxxxxxx | fa:16:3e:f0:15:12 | ip_address='192.168.0.5', subnet_id='92ed69d0-eec7-4ab4-ba45-1b135d0c4733' | ACTIVE |                 | compute:nova |      |
| a28f3259-df36-4c2d-a230-cee8e61bab81 |                              | fa:16:3e:e7:09:2e |                                                                            | DOWN   |                 | network:dhcp |      |
| bd74c244-5a0c-4aca-8c72-e3db8b7f9100 | xxxxxxxx  | fa:16:3e:c7:1a:0b | ip_address='192.168.0.6', subnet_id='92ed69d0-eec7-4ab4-ba45-1b135d0c4733' | ACTIVE |                 | compute:nova |      |
+--------------------------------------+------------------------------+-------------------+----------------------------------------------------------------------------+--------+-----------------+--------------+------+

And from the port list it's obvious that the network is genuinely full. 

Version-Release number of selected component (if applicable):
16.1

How reproducible:
100%

Steps to Reproduce:
1.run db sync when network ip are full
2.
3.

Actual results:
failure

Expected results:
able to run db sync even if network has used all ip's

Additional info: