Created attachment 1955483 [details] controller-0 neutron-ovn-db-sync-util.log Description of problem: OVS -> OVN migration fails due to no more IP on the external network. Error from /var/log/containers/neutron/neutron-ovn-db-sync-util.log: 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util Traceback (most recent call last): 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 138, in _ipam_allocate_ips 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util ip_address, subnet_id = ipam_allocator.allocate(ip_request) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/ipam/subnet_alloc.py", line 240, in allocate 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise ipam_exc.IpAddressGenerationFailureAllSubnets() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util neutron.ipam.exceptions.IpAddressGenerationFailureAllSubnets: No more IP addresses available. 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred: 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util Traceback (most recent call last): 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module> 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util sys.exit(main()) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 241, in main 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util synchronizer.do_sync() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 104, in do_sync 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.sync_networks_ports_and_dhcp_opts(ctx) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 1004, in sync_networks_ports_and_dhcp_opts 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self._ovn_client.create_network(ctx, network) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1885, in create_network 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.create_metadata_port(context, network) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2383, in create_metadata_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util p_utils.create_port(self._plugin, context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/plugins/utils.py", line 337, in create_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return core_plugin.create_port( 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 217, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return method(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util setattr(e, '_RETRY_EXCEEDED', True) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util ectxt.value = e.inner_exc 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util LOG.debug("Retry wrapper got retriable exception: %s", e) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return f(*dup_args, **dup_kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 1489, in create_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util result, mech_context = self._create_port_db(context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 1456, in _create_port_db 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util port_db = self.create_port_db(context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/db_base_plugin_v2.py", line 1441, in create_port_db 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.ipam.allocate_ips_for_port_and_store( 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 219, in allocate_ips_for_port_and_store 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util ips = self.allocate_ips_for_port(context, port_copy) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 217, in wrapped 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return method(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1014, in wrapper 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return fn(*args, **kwargs) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 226, in allocate_ips_for_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return self._allocate_ips_for_port(context, port) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 288, in _allocate_ips_for_port 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return self._ipam_allocate_ips(context, ipam_driver, p, ips) 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 148, in _ipam_allocate_ips 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util return 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util self.force_reraise() 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise self.value 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/db/ipam_pluggable_backend.py", line 140, in _ipam_allocate_ips 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util raise n_exc.IpAddressGenerationFailure( 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util neutron_lib.exceptions.IpAddressGenerationFailure: No more IP addresses available on network 8fc29845-c4d7-4f16-87ec-9ed0dc94625a. 2023-04-03 12:45:16.048 85 ERROR neutron_ovn_db_sync_util Error from automation: TASK [migration : Sync neutron db with OVN db (container) - Run 1] ************* task path: /home/stack/ovn_migration/playbooks/roles/migration/tasks/sync-dbs.yml:7 Monday 03 April 2023 12:45:04 +0000 (0:00:00.927) 0:31:02.801 ********** fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "052635d84e21", "neutron-ovn-db-sync-util", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "--ovn-neutron_sync_mode", "migrate"], "delta": "0:00:11.580294", "end": "2023-04-03 12:45:16.871626", "msg": "non-zero return code", "rc": 1, "start": "2023-04-03 12:45:05.291332", "stderr": "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences)\n util.warn_limited(", "stderr_lines": ["/usr/lib64/python3.9/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences)", " util.warn_limited("], "stdout": "", "stdout_lines": []} Version-Release number of selected component (if applicable): OSP17.1 How reproducible: 100% Steps to Reproduce: 1. deploy overcloud 2. run ovn migration 3. Actual results: OVN migration fails Expected results: OVN migration passing Additional info:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577