Bug 2089884

Summary: [OSP17] DB sync script gets stuck when attempting to update port
Product: Red Hat OpenStack Reporter: Roman Safronov <rsafrono>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Roman Safronov <rsafrono>
Severity: high Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: chrisw, jlibosva, scohen
Target Milestone: betaKeywords: Regression, Triaged
Target Release: 17.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-18.4.1-0.20220529170342.3995b26.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-21 12:21:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roman Safronov 2022-05-24 16:06:59 UTC 
Description of problem:
When OVN migration reach 'Sync neutron db with OVN db (container) - Run 1' it fails to proceed, just hands forever.

There are multiple traces in logs, e.g.

in the neutron server logs:

2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers [req-88e002f9-6d2e-4216-a871-a4ad83d9c727 - - - - -] Mechanism driver 'ovn' failed in create_port_postcommit: RuntimeError: neutron-0179741d-77a5-42f5-b879-9fef19d92a28 does not exist in name of Logical_Switch
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 198, in check_for_row_by_value_and_retry
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     idlutils.row_by_value(self.idl, table, column, match)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise RowNotFound(table=table, col=column, match=match)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-0179741d-77a5-42f5-b879-9fef19d92a28
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers 
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers 
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 792, in create_port_postcommit
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     self._ovn_client.create_port(context._plugin_context, port)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 356, in create_port
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     self._nb_idl.check_for_row_by_value_and_retry(
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     return self(f, *args, **kw)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     do = self.iter(retry_state=retry_state)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 372, in iter
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise retry_exc.reraise()
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 189, in reraise
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise self.last_attempt.result()
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     return self.__get_result()
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise self._exception
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     result = fn(*args, **kwargs)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 202, in check_for_row_by_value_and_retry
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers     raise RuntimeError(msg)
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers RuntimeError: neutron-0179741d-77a5-42f5-b879-9fef19d92a28 does not exist in name of Logical_Switch
2022-05-24 12:02:08.930 21 ERROR neutron.plugins.ml2.managers 
2022-05-24 12:02:08.933 21 ERROR neutron.plugins.ml2.plugin [req-88e002f9-6d2e-4216-a871-a4ad83d9c727 - - - - -] mechanism_manager.create_port_postcommit failed, deleting port '0968fc66-3bb4-4f81-ba26-fcbf43efffd3': neutron.plugins.ml2.common.exceptions.MechanismDriverError


in neutron-ovn-db-sync-util.log

l2/managers.py:479
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers [req-2b68417d-f0a2-4fda-80e7-0a191b7cce06 - - - - -] Mechanism driver 'ovn-sync' failed in update_port_postcommit: bdb.BdbQuit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 479, in _call_on_drivers
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 89, in update_port_postcommit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     self.ovn_client.update_port(context._plugin_context, port,
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python3.9/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 89, in update_port_postcommit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     self.ovn_client.update_port(context._plugin_context, port,
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/bdb.py", line 88, in trace_dispatch
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     return self.dispatch_line(frame)
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers   File "/usr/lib64/python3.9/bdb.py", line 113, in dispatch_line
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers     if self.quitting: raise BdbQuit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers bdb.BdbQuit
2022-05-24 15:38:08.777 3075 ERROR neutron.plugins.ml2.managers 
2022-05-24 15:38:08.778 3075 ERROR neutron.plugins.ml2.plugin [req-2b68417d-f0a2-4fda-80e7-0a191b7cce06 - - - - -] mechanism_manager.update_port_postcommit failed for port 264d1d73-061f-4bae-a1e4-97cc8dd6642a: neutron.plugins.ml2.common.exceptions.MechanismDriverError





Version-Release number of selected component (if applicable):
RHOS-17.0-RHEL-9-20220519.n.1
python3-neutron-18.3.1-0.20220508170403.a88c2b2.el9ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy ml2/ovs HA environment, 3 controllers + 3 compute nodes
2. Create a basic workload: a network, subnet, router, security group and 2 VMs
3. Try to perform migration to OVN according to the official documentation

Actual results:
OVN migration get stuck on  'Sync neutron db with OVN db (container) - Run 1' task

Expected results:
OVN migration succeeds


Additional info:
Comment 4 Roman Safronov 2022-06-26 10:55:16 UTC 
Verified that the problem does not occur when performing ovs2ovn migration on RHOS-17.0-RHEL-9-20220621.n.2
Comment 8 errata-xmlrpc 2022-09-21 12:21:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543