2089884 – [OSP17] DB sync script gets stuck when attempting to update port

Bug 2089884 - [OSP17] DB sync script gets stuck when attempting to update port

Summary: [OSP17] DB sync script gets stuck when attempting to update port

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	17.0 (Wallaby)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	beta
Target Release:	17.0
Assignee:	Jakub Libosvar
QA Contact:	Roman Safronov
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-05-24 16:06 UTC by Roman Safronov
Modified:	2022-09-21 12:22 UTC (History)
CC List:	3 users (show)
Fixed In Version:	openstack-neutron-18.4.1-0.20220529170342.3995b26.el9ost
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-09-21 12:21:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1950679	None	None	None	2022-05-24 17:03:50 UTC
OpenStack gerrit	843190	None	MERGED	[OVN] Fix deadlock in neutron_ovn_db_sync_util.py	2022-06-14 19:22:18 UTC
Red Hat Issue Tracker	OSP-15378	None	None	None	2022-05-24 16:16:15 UTC
Red Hat Product Errata	RHEA-2022:6543	None	None	None	2022-09-21 12:22:06 UTC

Description Roman Safronov 2022-05-24 16:06:59 UTC

Description of problem:
When OVN migration reach 'Sync neutron db with OVN db (container) - Run 1' it fails to proceed, just hands forever.

There are multiple traces in logs, e.g.

in the neutron server logs:

2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers [req-88e002f9-6d2e-4216-a871-a4ad83d9c727 - - - - -] Mechanism driver 'ovn' failed in create_port_postcommit: RuntimeError: neutron-0179741d-77a5-42f5-b879-9fef19d92a28 does not exist in name of Logical_Switch
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 198, in check_for_row_by_value_and_retry
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     idlutils.row_by_value(self.idl, table, column, match)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise RowNotFound(table=table, col=column, match=match)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-0179741d-77a5-42f5-b879-9fef19d92a28
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers 
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers 
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 792, in create_port_postcommit
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     self._ovn_client.create_port(context._plugin_context, port)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 356, in create_port
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     self._nb_idl.check_for_row_by_value_and_retry(
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     return self(f, *args, **kw)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     do = self.iter(retry_state=retry_state)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 372, in iter
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise retry_exc.reraise()
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 189, in reraise
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise self.last_attempt.result()
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     return self.__get_result()
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise self._exception
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     result = fn(*args, **kwargs)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 202, in check_for_row_by_value_and_retry
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers RuntimeError: neutron-0179741d-77a5-42f5-b879-9fef19d92a28 does not exist in name of Logical_Switch
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers 
2022-05-24 12:02:08.933 21 ERROR neutron.plugins.ml2.plugin [req-88e002f9-6d2e-4216-a871-a4ad83d9c727 - - - - -] mechanism_manager.create_port_postcommit failed, deleting port '0968fc66-3bb4-4f81-ba26-fcbf43efffd3': neutron.plugins.ml2.common.exceptions.MechanismDriverError


in neutron-ovn-db-sync-util.log

l2/managers.py:479
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers [req-2b68417d-f0a2-4fda-80e7-0a191b7cce06 - - - - -] Mechanism driver 'ovn-sync' failed in update_port_postcommit: bdb.BdbQuit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 479, in _call_on_drivers
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 89, in update_port_postcommit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     self.ovn_client.update_port(context._plugin_context, port,
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 89, in update_port_postcommit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     self.ovn_client.update_port(context._plugin_context, port,
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/bdb.py", line 88, in trace_dispatch
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     return self.dispatch_line(frame)
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/bdb.py", line 113, in dispatch_line
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     if self.quitting: raise BdbQuit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers bdb.BdbQuit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers 
2022-05-24 15:38:08.778 3075 ERROR neutron.plugins.ml2.plugin [req-2b68417d-f0a2-4fda-80e7-0a191b7cce06 - - - - -] mechanism_manager.update_port_postcommit failed for port 264d1d73-061f-4bae-a1e4-97cc8dd6642a: neutron.plugins.ml2.common.exceptions.MechanismDriverError





Version-Release number of selected component (if applicable):
RHOS-17.0-RHEL-9-20220519.n.1
python3-neutron-18.3.1-0.20220508170403.a88c2b2.el9ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy ml2/ovs HA environment, 3 controllers + 3 compute nodes
2. Create a basic workload: a network, subnet, router, security group and 2 VMs
3. Try to perform migration to OVN according to the official documentation

Actual results:
OVN migration get stuck on  'Sync neutron db with OVN db (container) - Run 1' task

Expected results:
OVN migration succeeds


Additional info:

Comment 4 Roman Safronov 2022-06-26 10:55:16 UTC

Verified that the problem does not occur when performing ovs2ovn migration on RHOS-17.0-RHEL-9-20220621.n.2

Comment 8 errata-xmlrpc 2022-09-21 12:21:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543

Note You need to log in before you can comment on or make changes to this bug.