Bug 2168874

Summary:	ValueError executing withdraw/expose operations on controllers
Product:	Red Hat OpenStack	Reporter:	Eduardo Olivares <eolivare>
Component:	ovn-bgp-agent	Assignee:	Luis Tomas Bolivar <ltomasbo>
Status:	CLOSED ERRATA	QA Contact:	Eduardo Olivares <eolivare>
Severity:	medium	Docs Contact:
Priority:	high
Version:	17.1 (Wallaby)	CC:	dalvarez, lmartins, ltomasbo
Target Milestone:	beta	Keywords:	Triaged
Target Release:	17.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovn-bgp-agent-0.3.1-1.20230308161522.3c21775.el9ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-16 01:13:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eduardo Olivares 2023-02-10 11:50:06 UTC

Description of problem:
While running the tempest neutron regression, some unhandled exceptions have been found on the controllers nodes:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-ovn-bgp-agent-17.1_director-rhel-virthost-basic_topology-ipv4-geneve/135/ctrl-1-0/var/log/containers/stdouts/ovn_bgp_agent.log.1.gz
2023-02-07T13:01:26.601331405+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.backend.ovs_idl.command [00m
2023-02-07T13:01:26.601331405+00:00 stdout F 2023-02-07 13:01:26.600 100487 DEBUG oslo_concurrency.lockutils [-] Lock "bgp" released by "ovn_bgp_agent.drivers.openstack.ovn_bgp_driver.OVNBGPDriver.withdraw_remote_ip" :: held 0.001s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:367[00m
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: ValueError: Column type and condition operand do not match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event Traceback (most recent call last):
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/event.py", line 159, in notify_loop
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     match.run(event, row, updates)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/watchers/bgp_watcher.py", line 315, in run
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     self.agent.withdraw_remote_ip(ips, row, chassis)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return f(*args, **kwargs)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py", line 641, in withdraw_remote_ip
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     if (self.sb_idl.is_provider_network(row.datapath) or
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/ovn.py", line 141, in is_provider_network
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return bool(cmd.execute(check_error=True))
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 39, in execute
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     self.run_idl(None)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 314, in run_idl
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     self.result = [
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 319, in <listcomp>
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     for r in rows if idlutils.row_match(r, self.conditions)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 412, in row_match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return all(condition_match(row, cond) for cond in conditions)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 412, in <genexpr>
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     return all(condition_match(row, cond) for cond in conditions)
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 359, in condition_match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event     raise ValueError(
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event ValueError: Column type and condition operand do not match
2023-02-07T13:01:26.601437566+00:00 stdout F 2023-02-07 13:01:26.600 100487 ERROR ovsdbapp.event [00m


They happen during the execution of the withdraw_remote_ip method.

Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230131.n.2
ovn-bgp-agent-0.3.1-1.20230120160941.62a04d4.el9ost


How reproducible:
it is reproduced several times during the execution of the tempest neutron tests

Steps to Reproduce:
Unfortunately, we don't have a more accurate/simple reproducer yet - it is reproduced by running the tempest neutron tests

Comment 8 Eduardo Olivares 2023-04-13 10:39:53 UTC

Verified on RHOS-17.1-RHEL-9-20230404.n.1 / ovn-bgp-agent-0.3.1-1.20230324161202.ddb740e.el9ost
I have reviewed both DVR and non-DVR jobs (two builds triggered with that compose per job), which ran frrempest, tempest and tobiko. The errors from the previous comments are not found in the ovn-bgp-agent logs from those jobs.

Comment 18 errata-xmlrpc 2023-08-16 01:13:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577