Description of problem: OVN migration fails in case d/s CI job runs full neutron tempest suite before starting ovn migration. Note: usually d/s CI jobs execute a small subset of tempest tests before starting ovn migration and the issue does not happen. I tried to trigger a full tempest run before ovn migration and the issue reproduced. Tried 2 times and result was the same. from /var/log/containers/neutron/neutron-ovn-db-sync-util.log on controller-0 2023-01-26 18:09:27.099 100 ERROR ovsdbapp.backend.ovs_idl.transaction [req-19e57e7b-e7bb-4dbc-98ac-2b3655838395 - - - - -] Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit command.run_idl(txn) File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl entity = self.api.lookup(self.lookup_table, self.entity) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup return self._lookup(table, record) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup row = idlutils.row_by_value(self, rl.table, rl.column, record) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value raise RowNotFound(table=table, col=column, match=match) ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_15f46bca_c429_4812_8eba_2c339c4ed41a 2023-01-26 18:09:27.100 100 CRITICAL neutron_ovn_db_sync_util [req-19e57e7b-e7bb-4dbc-98ac-2b3655838395 - - - - -] Unhandled error: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_15f46bca_c429_4812_8eba_2c339c4ed41a 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util Traceback (most recent call last): 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module> 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util sys.exit(main()) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 237, in main 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util synchronizer.do_sync() 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 106, in do_sync 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util self.sync_acls(ctx) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 304, in sync_acls 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util txn.add(self.ovn_api.pg_acl_add(**acla, may_exist=True)) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util next(self.gen) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 269, in transaction 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util yield t 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util next(self.gen) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 110, in transaction 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util del self._nested_txns_map[cur_thread_id] 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 61, in __exit__ 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util self.result = self.commit() 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util raise result.ex 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util txn.results.put(txn.do_commit()) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util command.run_idl(txn) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util entity = self.api.lookup(self.lookup_table, self.entity) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util return self._lookup(table, record) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util row = idlutils.row_by_value(self, rl.table, rl.column, record) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util raise RowNotFound(table=table, col=column, match=match) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_15f46bca_c429_4812_8eba_2c339c4ed41a 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20221130.n.1 python3-neutron-18.6.1-1.20221123160753.96d39b0.el9ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy ml2ovs HA environment, 3 controllers + 2 compute nodes. 2. Run full tempest run (networking tests from tempest repo + neutron upstream plugin + neutron downstream plugin test) 3. Start migration from OVS to OVN according to the official procedure https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.0/html/testing_migration_of_the_networking_service_to_the_ml2ovn_mechanism_driver/migrating-ml2ovs-to-ovn Actual results: Migration fails because of neutron-ovn-db-sync-util error. Expected results: No errors during neutron-ovn-db-sync-util execution. Migration completed successful. Additional info: