Bug 2168874 - ValueError executing withdraw/expose operations on controllers
Summary: ValueError executing withdraw/expose operations on controllers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: ovn-bgp-agent
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: beta
: 17.1
Assignee: Luis Tomas Bolivar
QA Contact: Eduardo Olivares
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-02-10 11:50 UTC by Eduardo Olivares
Modified: 2023-08-16 01:14 UTC (History)
3 users (show)

Fixed In Version: ovn-bgp-agent-0.3.1-1.20230308161522.3c21775.el9ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-16 01:13:48 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 873084 0 None MERGED Avoid race condition on is_provider_network checkings 2023-02-14 11:30:48 UTC
OpenStack gerrit 876277 0 None MERGED More protection from router deletion races 2023-03-08 13:57:59 UTC
Red Hat Issue Tracker OSP-22239 0 None None None 2023-02-10 11:51:18 UTC
Red Hat Product Errata RHEA-2023:4577 0 None None None 2023-08-16 01:14:11 UTC

Description Eduardo Olivares 2023-02-10 11:50:06 UTC 
Description of problem:
While running the tempest neutron regression, some unhandled exceptions have been found on the controllers nodes:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-ovn-bgp-agent-17.1_director-rhel-virthost-basic_topology-ipv4-geneve/135/ctrl-1-0/var/log/containers/stdouts/ovn_bgp_agent.log.1.gz
2023-02-07T13:01:26.601331405+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.backend.ovs_idl.command [00m
2023-02-07T13:01:26.601331405+00:00 stdout F 2023-02-07 13:01:26.600 100487 DEBUG oslo_concurrency.lockutils [-] Lock "bgp" released by "ovn_bgp_agent.drivers.openstack.ovn_bgp_driver.OVNBGPDriver.withdraw_remote_ip" :: held 0.001s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:367[00m
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: ValueError: Column type and condition operand do not match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event Traceback (most recent call last):
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/event.py", line 159, in notify_loop
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     match.run(event, row, updates)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/watchers/bgp_watcher.py", line 315, in run
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     self.agent.withdraw_remote_ip(ips, row, chassis)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return f(*args, **kwargs)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py", line 641, in withdraw_remote_ip
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     if (self.sb_idl.is_provider_network(row.datapath) or
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/ovn.py", line 141, in is_provider_network
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return bool(cmd.execute(check_error=True))
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 39, in execute
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     self.run_idl(None)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 314, in run_idl
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     self.result = [
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 319, in <listcomp>
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     for r in rows if idlutils.row_match(r, self.conditions)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 412, in row_match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return all(condition_match(row, cond) for cond in conditions)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 412, in <genexpr>
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return all(condition_match(row, cond) for cond in conditions)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 359, in condition_match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     raise ValueError(
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event ValueError: Column type and condition operand do not match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event [00m


They happen during the execution of the withdraw_remote_ip method.

Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230131.n.2
ovn-bgp-agent-0.3.1-1.20230120160941.62a04d4.el9ost


How reproducible:
it is reproduced several times during the execution of the tempest neutron tests

Steps to Reproduce:
Unfortunately, we don't have a more accurate/simple reproducer yet - it is reproduced by running the tempest neutron tests
Comment 8 Eduardo Olivares 2023-04-13 10:39:53 UTC 
Verified on RHOS-17.1-RHEL-9-20230404.n.1 / ovn-bgp-agent-0.3.1-1.20230324161202.ddb740e.el9ost
I have reviewed both DVR and non-DVR jobs (two builds triggered with that compose per job), which ran frrempest, tempest and tobiko. The errors from the previous comments are not found in the ovn-bgp-agent logs from those jobs.
Comment 18 errata-xmlrpc 2023-08-16 01:13:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577
Note You need to log in before you can comment on or make changes to this bug.