Bug 1638715 - During migration, neutron-ovn-db-sync-util is failing
Summary: During migration, neutron-ovn-db-sync-util is failing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 14.0 (Rocky)
Assignee: Daniel Alvarez Sanchez
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks: 1546994 1629430 1925290
TreeView+ depends on / blocked
 
Reported: 2018-10-12 10:14 UTC by Numan Siddique
Modified: 2022-03-11 15:07 UTC (History)
10 users (show)

Fixed In Version: python-networking-ovn-5.0.2-0.20181009120341.99b02f6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 11:53:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1798028 0 None None None 2018-10-16 07:19:13 UTC
OpenStack gerrit 599426 0 None MERGED Fix ovn db sync tool to support Port Groups 2020-10-28 05:09:32 UTC
OpenStack gerrit 611786 0 None MERGED Fix ovn db sync tool to support Port Groups 2020-10-28 05:09:17 UTC
Red Hat Issue Tracker OSP-13517 0 None None None 2022-03-11 15:07:27 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:54:03 UTC

Description Numan Siddique 2018-10-12 10:14:20 UTC
Description of problem:

The OVN migration script, after updating the stack to OVN (with br-migration), creates the OVN db by running the 'neutron-ovn-db-sync-util' inside the neutron-server-ovn container in the bootstrap controller (controller-0) and it fails with the below exceptio

*****
2018-10-12 10:06:18.633 3669 INFO networking_ovn.cmd.neutron_ovn_db_sync_util [req-91093acc-99d9-4343-82dd-3ce4843bc15e - - - - -] Sync for Northbound db started with mode : repair
2018-10-12 10:06:18.633 3669 DEBUG networking_ovn.ovn_db_sync [req-91093acc-99d9-4343-82dd-3ce4843bc15e - - - - -] Starting OVN-Northbound DB sync process do_sync /usr/lib/python2.7/site-packages/networking_ovn/ovn_db_sync.py:86
2018-10-12 10:06:18.634 3669 DEBUG networking_ovn.ovn_db_sync [req-91093acc-99d9-4343-82dd-3ce4843bc15e - - - - -] Port Groups Migration task started migrate_to_port_groups /usr/lib/python2.7/site-packages/networking_ovn/ovn_db_sync.py:1028
2018-10-12 10:06:18.760 3669 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): PgAddCommand(columns={'acls': []}, may_exist=False, name=neutron_pg_drop) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2018-10-12 10:06:18.761 3669 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): PgAclAddCommand(direction=from-lport, log=False, name=[], may_exist=False, entity=neutron_pg_drop, priority=1001, action=drop, external_ids={}, match=inport == @neutron_pg_drop && ip, severity=[]) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2018-10-12 10:06:18.762 3669 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): PgAclAddCommand(direction=to-lport, log=False, name=[], may_exist=False, entity=neutron_pg_drop, priority=1001, action=drop, external_ids={}, match=outport == @neutron_pg_drop && ip, severity=[]) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2018-10-12 10:06:18.763 3669 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): PgAddPortCommand(if_exists=False, lsp=[u'4a370dd0-9ee8-4c1e-bcd4-3b96b20bed3f', u'782de2ee-c33f-48ac-9949-69720f662f1d'], port_group=neutron_pg_drop) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2018-10-12 10:06:18.767 3669 ERROR ovsdbapp.backend.ovs_idl.transaction [req-91093acc-99d9-4343-82dd-3ce4843bc15e - - - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python2.7/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 1281, in run_idl
    self._run_method(pg, lsp)
  File "/usr/lib/python2.7/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 1269, in _run_method
    'Port %s does not exist' % port)
RuntimeError: Port 4a370dd0-9ee8-4c1e-bcd4-3b96b20bed3f does not exist

2018-10-12 10:06:18.768 3669 CRITICAL neutron_ovn_db_sync_util [req-91093acc-99d9-4343-82dd-3ce4843bc15e - - - - -] Unhandled error: RuntimeError: Port 4a370dd0-9ee8-4c1e-bcd4-3b96b20bed3f does not exist
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     sys.exit(main())
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/networking_ovn/cmd/neutron_ovn_db_sync_util.py", line 205, in main
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/networking_ovn/ovn_db_sync.py", line 89, in do_sync
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     self.migrate_to_port_groups(ctx)
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/networking_ovn/ovn_db_sync.py", line 1038, in migrate_to_port_groups
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     self._create_default_drop_port_group(db_ports)
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/networking_ovn/ovn_db_sync.py", line 967, in _create_default_drop_port_group
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     txn.add(self.ovn_api.pg_add_ports(pg_name, ports_ids))
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     self.gen.next()
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 171, in transaction
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     yield t
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     self.gen.next()
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 112, in transaction
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     del self._nested_txns_map[cur_thread_id]
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 69, in __exit__
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     self.result = self.commit()
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util     raise result.ex
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util RuntimeError: Port 4a370dd0-9ee8-4c1e-bcd4-3b96b20bed3f does not exist
2018-10-12 10:06:18.768 3669 ERROR neutron_ovn_db_sync_util 
******

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. On a ML2OVS deployment run ovn migration script.
2.
3.

Actual results:
The OVN north db is empty because of which migration fails.


Expected results:
The neutron-db-sync-util should be successful and the OVN north db should be created.

Additional info:

Comment 2 Daniel Alvarez Sanchez 2018-10-22 06:49:44 UTC
Upstream patch merged in master. stable/rocky backport here: https://review.openstack.org/#/c/611786/1

Comment 8 Miguel Angel Ajo 2018-11-16 15:09:18 UTC
Ok, I've verified this doesn't happen anymore during migration.

Comment 10 errata-xmlrpc 2019-01-11 11:53:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.