Cloned to track documentation of this issue +++ This bug was initially created as a clone of Bug #2184070 +++ Description of problem: OVS -> OVN migration fails due to no more IP on the external network. Error from /var/log/containers/neutron/neutron-ovn-db-sync-util.log: 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util Traceback (most recent call last): 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 138, in _ipam_allocate_ips 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util ip_address, subnet_id = ipam_allocator.allocate(ip_request) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/ipam/subnet_alloc.py", line 240, in allocate 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise ipam_exc.IpAddressGenerationFailureAllSubnets() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util neutron.ipam.exceptions.IpAddressGenerationFailureAllSubnets: No more IP addresses available. 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred: 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util Traceback (most recent call last): 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module> 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util sys.exit(main()) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 241, in main 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util synchronizer.do_sync() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 104, in do_sync 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.sync_networks_ports_and_dhcp_opts(ctx) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 1004, in sync_networks_ports_and_dhcp_opts 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self._ovn_client.create_network(ctx, network) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1885, in create_network 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.create_metadata_port(context, network) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2383, in create_metadata_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util p_utils.create_port(self._plugin, context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/plugins/utils.py", line 337, in create_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return core_plugin.create_port( 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 217, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return method(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util setattr(e, '_RETRY_EXCEEDED', True) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util ectxt.value = e.inner_exc 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util LOG.debug("Retry wrapper got retriable exception: %s", e) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*dup_args, **dup_kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 1489, in create_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util result, mech_context = self._create_port_db(context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 1456, in _create_port_db 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util port_db = self.create_port_db(context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/db_base_plugin_v2.py", line 1441, in create_port_db 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.ipam.allocate_ips_for_port_and_store( 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 219, in allocate_ips_for_port_and_store 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util ips = self.allocate_ips_for_port(context, port_copy) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 217, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return method(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1014, in wrapper 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return fn(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 226, in allocate_ips_for_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return self._allocate_ips_for_port(context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 288, in _allocate_ips_for_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return self._ipam_allocate_ips(context, ipam_driver, p, ips) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 148, in _ipam_allocate_ips 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 140, in _ipam_allocate_ips 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise n_exc.IpAddressGenerationFailure( 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util neutron_lib.exceptions.IpAddressGenerationFailure: No more IP addresses available on network 8fc29845-c4d7-4f16-87ec-9ed0dc94625a. 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util Error from automation: TASK [migration : Sync neutron db with OVN db (container) - Run 1] ************* task path: /home/stack/ovn_migration/playbooks/roles/migration/tasks/sync-dbs.yml:7 Monday 03 April 2023 12:45:04 +0000 (0:00:00.927) 0:31:02.801 ********** fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "052635d84e21", "neutron-ovn-db-sync-util", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "--ovn-neutron_sync_mode", "migrate"], "delta": "0:00:11.580294", "end": "2023-04-03 12:45:16.871626", "msg": "non-zero return code", "rc": 1, "start": "2023-04-03 12:45:05.291332", "stderr": "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences)\n util.warn_limited(", "stderr_lines": ["/usr/lib64/python3.9/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences)", " util.warn_limited("], "stdout": "", "stdout_lines": []} Version-Release number of selected component (if applicable): OSP17.1 How reproducible: 100% Steps to Reproduce: 1. deploy overcloud 2. run ovn migration 3. Actual results: OVN migration fails Expected results: OVN migration passing Additional info: --- Additional comment from Ella Shulman on 2023-04-03 14:57:13 UTC --- --- Additional comment from Miro Tomaska on 2023-04-10 20:18:42 UTC --- Hi Ella, Where is the Jenkins job that run this migration and run into this issue? Thank you --- Additional comment from Jakub Libosvar on 2023-04-12 13:35:54 UTC --- This was attempted to be fixed in Wallaby upstream https://review.opendev.org/c/openstack/neutron/+/854013 8 months ago. --- Additional comment from Jakub Libosvar on 2023-04-12 13:40:18 UTC --- (In reply to Jakub Libosvar from comment #3) > This was attempted to be fixed in Wallaby upstream > https://review.opendev.org/c/openstack/neutron/+/854013 8 months ago. disregard, this error comes from elsewhere. --- Additional comment from Ella Shulman on 2023-04-13 11:31:59 UTC --- Hi guys, This is the Jenkins job that failed: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/nfv/view/ovn-migration/job/DFG-nfv-17.1-director-3cont-2comp-ipv4-vxlan-dpdk-sriov-ctlplane-dataplane-bonding-hybrid-ovn-migration/2/ BR. --- Additional comment from Miro Tomaska on 2023-04-13 21:18:56 UTC --- Thanks Ella, The reason for the error is because when we create a network in OVN we ensure that each subnet in that network has metadata port. However, since the given network's subnet is completely depleted of IPs the exception is raised. The exeception is coming from a network called `external_net_403` with a subnet with cidr 10.46.141.112/28. I am not sure why the test run only specifies cidr /28. Is this a regression? Maybe we added more tests which require additional floating IPs but we never raised allocation pool of IPs for that network? Just for reference, here are some CLI outputs I pulled from one machine which run this job: No more IP addresses available on network cf193053-d46f-40ea-87b7-2ff3c69d5c07 (overcloud) [stack@undercloud-0 ~]$ openstack network show cf193053-d46f-40ea-87b7-2ff3c69d5c07 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2023-04-13T12:25:19Z | | description | | | dns_domain | | | id | cf193053-d46f-40ea-87b7-2ff3c69d5c07 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 9000 | | name | external_net_403 | | port_security_enabled | True | | project_id | 463bbb20e29d42ae922ba7738637d50f | | provider:network_type | vlan | | provider:physical_network | access | | provider:segmentation_id | 403 | | qos_policy_id | None | | revision_number | 2 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | 8834539d-64df-4f68-aafb-af9d51cb93a0 | | tags | | | updated_at | 2023-04-13T12:25:25Z | +---------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack subnet show 8834539d-64df-4f68-aafb-af9d51cb93a0 +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 10.46.141.113-10.46.141.121 | | cidr | 10.46.141.112/28 | | created_at | 2023-04-13T12:25:25Z | | description | | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | True | | gateway_ip | 10.46.141.126 | | host_routes | | | id | 8834539d-64df-4f68-aafb-af9d51cb93a0 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | access | | network_id | cf193053-d46f-40ea-87b7-2ff3c69d5c07 | | prefix_length | None | | project_id | 463bbb20e29d42ae922ba7738637d50f | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2023-04-13T12:25:25Z | +----------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port list --network cf193053-d46f-40ea-87b7-2ff3c69d5c07 +--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+ | 130db807-bce7-4a1d-9e9d-3c0107d71acd | | fa:16:3e:3d:e8:78 | ip_address='10.46.141.114', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | ACTIVE | | 27de137b-42fb-4168-9a5d-19fb6a0d0a8a | | fa:16:3e:73:6d:7e | ip_address='10.46.141.121', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | N/A | | 310ae5de-2339-4c13-a73e-94f882926046 | | fa:16:3e:dc:20:f2 | ip_address='10.46.141.119', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | N/A | | 718ff2d8-9a5d-45a4-bf50-a56ae8f72e38 | | fa:16:3e:83:99:cf | ip_address='10.46.141.115', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | ACTIVE | | b0ceaceb-d0ea-4142-bea1-991d47d96b49 | | fa:16:3e:57:b6:d7 | ip_address='10.46.141.113', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | ACTIVE | | baed317c-7673-46f8-8d31-eeeca996440f | | fa:16:3e:13:49:d9 | ip_address='10.46.141.116', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | ACTIVE | | bc75290d-d9f8-4de7-afa3-9cce76860c38 | | fa:16:3e:99:65:bd | ip_address='10.46.141.120', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | N/A | | c27e995d-2781-4170-a1c3-808088cb657c | | fa:16:3e:54:52:64 | ip_address='10.46.141.118', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | ACTIVE | | ef0793a0-eded-4fce-a767-7406348f1ee9 | | fa:16:3e:45:19:ff | ip_address='10.46.141.117', subnet_id='8834539d-64df-4f68-aafb-af9d51cb93a0' | N/A | +--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+ ---------------------------------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | 4fd92e9d-df96-402c-b337-b2db4e30f099 | 10.46.141.121 | 10.10.107.133 | 57765680-e2f2-4e26-bdd8-cb12527f42b9 | cf193053-d46f-40ea-87b7-2ff3c69d5c07 | 463bbb20e29d42ae922ba7738637d50f | | 72652584-dcba-4463-bfef-57f1bd60b6d8 | 10.46.141.119 | 192.168.168.76 | f055037a-40cd-4b3c-949b-5de186a6ba8b | cf193053-d46f-40ea-87b7-2ff3c69d5c07 | 463bbb20e29d42ae922ba7738637d50f | | a85ce9f7-e6cf-4e8d-b1f9-612a9547cb5a | 10.46.141.120 | 10.10.107.175 | 0f7ab415-36c8-4c52-b69c-697d9cb52ed0 | cf193053-d46f-40ea-87b7-2ff3c69d5c07 | 463bbb20e29d42ae922ba7738637d50f | | ff09bdcd-d477-41b8-874f-23dca51b97cd | 10.46.141.117 | 10.10.107.140 | 58dd2c5e-bf75-4f1a-b9a5-770de9d2677c | cf193053-d46f-40ea-87b7-2ff3c69d5c07 | 463bbb20e29d42ae922ba7738637d50f | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ 2023-04-13 13:33:28.598 85 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] Network found in Neutron but not in OVN DB, network_id=cf193053-d46f-40ea-87b7-2ff3c69d5c07 2023-04-13 13:33:28.598 85 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] Creating the network cf193053-d46f-40ea-87b7-2ff3c69d5c07 in OVN NB DB sync_networks_ports_and_dhcp_opts /usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1002 2023-04-13 13:33:28.612 85 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsAddCommand(switch=neutron-cf193053-d46f-40ea-87b7-2ff3c69d5c07, columns={'external_ids': {'neutron:network_name': 'external_net_403', 'neutron:mtu': '9000', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': ''}, 'other_config': {'mcast_snoop': 'true', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'}}, may_exist=True) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89 2023-04-13 13:33:28.612 85 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): AddLSwitchPortCommand(lport=provnet-0fbb8494-9608-422c-9c0e-8e35bd04eefc, lswitch=neutron-cf193053-d46f-40ea-87b7-2ff3c69d5c07, may_exist=True, columns={'addresses': ['unknown'], 'external_ids': {}, 'type': 'localnet', 'tag': 403, 'options': {'network_name': 'access', 'mcast_flood_reports': 'true', 'mcast_flood': 'false'}}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89 2023-04-13 13:33:28.636 85 WARNING neutron.db.ovn_revision_numbers_db [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] No revision row found for cf193053-d46f-40ea-87b7-2ff3c69d5c07 (type: networks) when bumping the revision number. Creating one. 2023-04-13 13:33:28.636 85 DEBUG neutron.db.ovn_revision_numbers_db [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] create_initial_revision uuid=cf193053-d46f-40ea-87b7-2ff3c69d5c07, type=networks, rev=-1 create_initial_revision /usr/lib/python3.9/site-packages/neutron/db/ovn_revision_numbers_db.py:104 2023-04-13 13:33:28.644 85 INFO neutron.db.ovn_revision_numbers_db [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] Successfully bumped revision number for resource cf193053-d46f-40ea-87b7-2ff3c69d5c07 (type: networks) to 2 2023-04-13 13:33:28.744 85 DEBUG neutron_lib.callbacks.manager [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] Notify callbacks ['neutron.cmd.ovn.neutron_ovn_db_sync_util.SecurityGroupDbMixin._ensure_default_security_group_handler-809008'] for port, before_create _notify_loop /usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192 2023-04-13 13:33:28.901 85 CRITICAL neutron_ovn_db_sync_util [req-ddd08dc8-7eed-4d47-87f1-9b5f265227c0 - - - - -] Unhandled error: neutron_lib.exceptions.IpAddressGenerationFailure: No more IP addresses available on network cf193053-d46f-40ea-87b7-2ff3c69d5c07. I would first look into the test that uses that network and figure out why the subnet cidr /28 is so small. There is nothing that we can do from the neutron side, except maybe catch and log when this condition occurs such that db_sync utility continues through the end. --- Additional comment from Ella Shulman on 2023-04-16 06:46:36 UTC --- Hi, We are using a /28 subnet because this is the external subnet we got from IT. We started testing ovn migration lately for 17.1 and we are using 3 VMs for tests and 2 pingers to verify network connectivity throughout the migration process. What is the minimum amount of available IPs we should have for a successful migration? should we document it for customers? how will this affect the migration process for customers with no available IPs on an external subnet? --- Additional comment from Miro Tomaska on 2023-04-17 16:36:29 UTC --- Ella, Thank you for the info. I think we can solve this in few ways, - Ask IT for a larger subnet - It looks like you only need 3 floating IPs for the "ping" VMs. I am assuming that rest of the IPs got consumed by some tempest tests? If so, then just make sure they get released/unset in the test tear down stage. This way you still have some IPs left when needed. In comment#4 I see there are 4 IPs with N/A status so maybe those ports can be deleted. Next, I will update the sync_utility code to catch and log when such condition occurs. However, this will be only a warning for the user and it will make sure the utility completes. I also agree that it would be nice to document this for the customer, I will work with a tech writter on that. --- Additional comment from Ella Shulman on 2023-04-20 09:14:09 UTC --- Miro can we mark this bug as assigned? --- Additional comment from RHEL Program Management on 2023-04-21 19:01:07 UTC --- This item has been properly Triaged and planned for the release, and Target Release is now set to match the release flag. --- Additional comment from Miro Tomaska on 2023-04-21 19:05:05 UTC --- (In reply to Ella Shulman from comment #9) > Miro can we mark this bug as assigned? Sure will do. I spoke to Eran about this BZ. How do you want to handle it? Do you want to expand the subnet cidr? Or should we look into clearing up some of those IPs before migration happens? --- Additional comment from Ella Shulman on 2023-05-04 06:17:53 UTC --- Hi Miro, I extended the subnet range and it seems like the ovn migration passes now. I think we should ask for docs for this bug. --- Additional comment from Miro Tomaska on 2023-05-17 02:40:10 UTC --- I am contemplating if we should add this to the migration docs. I am also not sure where that information will fit into existing docs but I can work with James on that. @Kuba, what is your opinion on adding something like this to the migration docs. “The ML2/OVN driver will implicitly create a metadata port for each network where the metadata port will claim one IP address from the IP pool. It is important that there is at least one available ip for each subnet in ML2/OVS before migration” . On one hand I think this information is useful but on other I feel like it might be confusing to some and creates lots noise in docs. BTW, I have a simple patch[1] U/S to log this condition and not raise. [1]https://review.opendev.org/c/openstack/neutron/+/880773 --- Additional comment from Ella Shulman on 2023-05-17 07:15:22 UTC --- IMHO I think we must document in some way (not sure what is the correct way), maybe in the preparation part we should ask costumers to make sure every network have at least a single ip that is free without explaining the "why". --- Additional comment from James Smith on 2023-05-19 18:41:01 UTC --- I made a doc bz [1] and a documentation merge requst [2] to explore the documentation proposal. In [1] I asked Miro for a quick review of [2] [1] https://bugzilla.redhat.com/show_bug.cgi?id=2208626 [2] https://gitlab.cee.redhat.com/rhci-documentation/docs-Red_Hat_Enterprise_Linux_OpenStack_Platform/-/merge_requests/11221 --- Additional comment from James Smith on 2023-05-23 13:29:56 UTC --- Hi Ella Please review the Release Note I added as Doc Text. I called it a Release Note rather than a Known Issue because it's not really a bug. Do you agree? I am also adding it to the migration guide as a prerequisite in [1]. Thanks! James [1] https://gitlab.cee.redhat.com/rhci-documentation/docs-Red_Hat_Enterprise_Linux_OpenStack_Platform/-/merge_requests/11221/diffs#note_6607108 --- Additional comment from Ella Shulman on 2023-05-24 09:07:29 UTC --- LGTM --- Additional comment from Eran Kuris on 2023-06-06 08:13:45 UTC --- Hi Miro, Can you backport this fix? I also would raise it as exception to have this fix for GA --- Additional comment from Paul Grist on 2023-06-12 14:56:36 UTC --- trac approved as exception - https://issues.redhat.com/browse/OSP-25735 - we would not block for the logging message, but should be able to get that in now. What we don't see is a BZ or tracker for the documentation needed if customer hits this, Is there a BZ?