Bug 2164909
| Summary: | [17.1][OVN] neutron-ovn-db-sync-util fails during migration from OVS to OVN | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Roman Safronov <rsafrono> | |
| Component: | openstack-neutron | Assignee: | Miro Tomaska <mtomaska> | |
| Status: | CLOSED ERRATA | QA Contact: | Ella Shulman <eshulman> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | high | |||
| Version: | 17.1 (Wallaby) | CC: | chrisw, ekuris, jamsmith, jschluet, mariel, mtomaska, scohen | |
| Target Milestone: | z2 | Keywords: | Triaged | |
| Target Release: | 17.1 | Flags: | mtomaska:
needinfo-
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-neutron-18.6.1-17.1.20231025110804.85ff760.el9ost | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2169202 (view as bug list) | Environment: | ||
| Last Closed: | 2024-01-16 14:32:03 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2169202 | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 17.1.2 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:0209 |
Description of problem: OVN migration fails in case d/s CI job runs full neutron tempest suite before starting ovn migration. Note: usually d/s CI jobs execute a small subset of tempest tests before starting ovn migration and the issue does not happen. I tried to trigger a full tempest run before ovn migration and the issue reproduced. Tried 2 times and result was the same. from /var/log/containers/neutron/neutron-ovn-db-sync-util.log on controller-0 2023-01-26 18:09:27.099 100 ERROR ovsdbapp.backend.ovs_idl.transaction [req-19e57e7b-e7bb-4dbc-98ac-2b3655838395 - - - - -] Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit command.run_idl(txn) File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl entity = self.api.lookup(self.lookup_table, self.entity) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup return self._lookup(table, record) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup row = idlutils.row_by_value(self, rl.table, rl.column, record) File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value raise RowNotFound(table=table, col=column, match=match) ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_15f46bca_c429_4812_8eba_2c339c4ed41a 2023-01-26 18:09:27.100 100 CRITICAL neutron_ovn_db_sync_util [req-19e57e7b-e7bb-4dbc-98ac-2b3655838395 - - - - -] Unhandled error: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_15f46bca_c429_4812_8eba_2c339c4ed41a 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util Traceback (most recent call last): 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module> 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util sys.exit(main()) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 237, in main 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util synchronizer.do_sync() 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 106, in do_sync 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util self.sync_acls(ctx) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 304, in sync_acls 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util txn.add(self.ovn_api.pg_acl_add(**acla, may_exist=True)) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util next(self.gen) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 269, in transaction 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util yield t 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util next(self.gen) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 110, in transaction 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util del self._nested_txns_map[cur_thread_id] 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 61, in __exit__ 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util self.result = self.commit() 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util raise result.ex 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util txn.results.put(txn.do_commit()) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util command.run_idl(txn) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util entity = self.api.lookup(self.lookup_table, self.entity) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util return self._lookup(table, record) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util row = idlutils.row_by_value(self, rl.table, rl.column, record) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util raise RowNotFound(table=table, col=column, match=match) 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_15f46bca_c429_4812_8eba_2c339c4ed41a 2023-01-26 18:09:27.100 100 ERROR neutron_ovn_db_sync_util Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20221130.n.1 python3-neutron-18.6.1-1.20221123160753.96d39b0.el9ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy ml2ovs HA environment, 3 controllers + 2 compute nodes. 2. Run full tempest run (networking tests from tempest repo + neutron upstream plugin + neutron downstream plugin test) 3. Start migration from OVS to OVN according to the official procedure https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.0/html/testing_migration_of_the_networking_service_to_the_ml2ovn_mechanism_driver/migrating-ml2ovs-to-ovn Actual results: Migration fails because of neutron-ovn-db-sync-util error. Expected results: No errors during neutron-ovn-db-sync-util execution. Migration completed successful. Additional info: