Bug 1956074 - check_for_mcast_flood_reports periodic in maintenace task fails because of ovsdb lock
Summary: check_for_mcast_flood_reports periodic in maintenace task fails because of ov...
Keywords:
Status: ON_DEV
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z7
: 16.1 (Train on RHEL 8.2)
Assignee: Lucas Alvares Gomes
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-02 14:32 UTC by Jakub Libosvar
Modified: 2021-05-14 04:32 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 789547 0 None NEW [OVN] Check for lock in check_for_mcast_flood_reports 2021-05-04 10:53:16 UTC

Description Jakub Libosvar 2021-05-02 14:32:02 UTC 
Description of problem:
2021-05-02 14:23:29.193 599 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     yield self._nested_txns_map[cur_thread_id]
2021-05-02 14:23:29.193 599 ERROR futurist.periodics KeyError: 139844066981736
2021-05-02 14:23:29.193 599 ERROR futurist.periodics
2021-05-02 14:23:29.193 599 ERROR futurist.periodics During handling of the above exception, another exception occurred:
2021-05-02 14:23:29.193 599 ERROR futurist.periodics
2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     work()
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     return self.callback(*self.args, **self.kwargs)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     return f(*args, **kwargs)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     for port in self._nb_idl.lsp_list().execute(check_error=True):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     t.add(self)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     next(self.gen)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 196, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     yield t
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     next(self.gen)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     del self._nested_txns_map[cur_thread_id]
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     self.result = self.commit()
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     raise result.ex
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     txn.results.put(txn.do_commit())
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     raise RuntimeError(msg)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2021-05-02 14:23:29.193 599 ERROR futurist.periodics

It seems the check_for_mcast_flood_reports method doesn't use has_lock

Version-Release number of selected component (if applicable):
OSP 16.1.5

How reproducible:
Always

Steps to Reproduce:
1. Install openstack with at least 3 controllers
2.
3.

Actual results:
All controllers that don't hold the lock fail

Expected results:


Additional info:
Note You need to log in before you can comment on or make changes to this bug.