Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2087721

Summary:	[OSP16.2]OVN migration failed due to unhandled error in neutron_ovn_db_sync_util
Product:	Red Hat OpenStack	Reporter:	Roman Safronov <rsafrono>
Component:	python-networking-ovn	Assignee:	Rodolfo Alonso <ralonsoh>
Status:	CLOSED ERRATA	QA Contact:	Roman Safronov <rsafrono>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	16.2 (Train)	CC:	apevec, froyo, lhh, majopela, ralonsoh, scohen, stchen
Target Milestone:	z4	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-networking-ovn-7.4.2-2.20220409154850.el8ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-12-07 19:22:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2139674
Bug Blocks:

Description Roman Safronov 2022-05-18 10:43:57 UTC

Description of problem:
While performing ovs2ovn migration the following exception occurred, from neutron/neutron-ovn-db-sync-util.log.1 on controller-0

2022-05-17 17:20:18.999 37 INFO networking_ovn.cmd.neutron_ovn_db_sync_util [req-dfcbb272-a1d7-434c-87b0-96d0eafd0a58 - - - - -] Migrating Neutron database from OVS to OVN
2022-05-17 17:20:19.363 37 CRITICAL neutron_ovn_db_sync_util [req-dfcbb272-a1d7-434c-87b0-96d0eafd0a58 - - - - -] Unhandled error: sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction.
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     sys.exit(main())
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/cmd/neutron_ovn_db_sync_util.py", line 235, in main
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     db_migration.migrate_neutron_database_to_ovn(core_plugin)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/db_migration.py", line 98, in migrate_neutron_database_to_ovn
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     pb_missed.add(port_id)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     next(self.gen)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1065, in _transaction_scope
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     yield resource
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     next(self.gen)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     self.session.rollback()
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     self.force_reraise()
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     raise value
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 664, in _session
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     self._end_session_transaction(self.session)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 692, in _end_session_transaction
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     session.commit()
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1026, in commit
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     self.transaction.commit()
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 491, in commit
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     self._assert_active(prepared_ok=True)
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 298, in _assert_active
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util     "This session is in 'inactive' state, due to the "
2022-05-17 17:20:19.363 37 ERROR neutron_ovn_db_sync_util sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction.



Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20220513.n.2
python3-networking-ovn-7.4.2-2.20220409154848.el8ost.noarch
python3-networking-ovn-migration-tool-7.4.2-2.20220409154848.el8ost.noarch
ovn-2021-21.12.0-42.el8fdp.x86_64

How reproducible:
Reproduced once for now.

Steps to Reproduce:
1. Deploy HA ML2OVS environment (3 controllers + 2 compute nodes), in my case DVR was not enabled
2. Create a workload, i.e. router, internal network, connect the network via router to external network, then spawn 2 VMs, one on each compute node
3. Apply changes from https://review.opendev.org/openstack/neutron refs/changes/38/835638/8 . This is a WIP patch intended to support backup and restore after OVN migration. It will add a backup control plane stage before starting the migration.
4. Follow the official documentation page for migrating to OVN, use no-DVR to no-DVR (centralized routing to centralized routing) scenario.

Actual results:
OVN migration script failed on TASK [migration : Sync neutron db with OVN db (container) - Run 1

Expected results:
OVN migration completed successfully

Additional info:

Comment 6 Arnau Verdaguer 2022-09-20 15:06:40 UTC

*** Bug 2128253 has been marked as a duplicate of this bug. ***

Comment 13 Roman Safronov 2022-11-27 09:21:33 UTC

Verified on puddle RHOS-16.2-RHEL-8-20221124.n.1 which uses python3-networking-ovn-7.4.2-2.20220409154865.el8ost.noarch.rpm
Verified that during downstream ovs2ovn CI jobs the issue does not happen.

Comment 19 errata-xmlrpc 2022-12-07 19:22:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8794