Bug 2223914

Summary: Error 500 while updating DSCP marking rule
Product: Red Hat OpenStack Reporter: Slawek Kaplonski <skaplons>
Component: openstack-neutronAssignee: Rodolfo Alonso <ralonsoh>
Status: NEW --- QA Contact: Maor <mblue>
Severity: low Docs Contact:
Priority: medium    
Version: 17.1 (Wallaby)CC: apevec, chrisw, jlibosva, lhh, majopela, mblue, ralonsoh, scohen
Target Milestone: z2Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Slawek Kaplonski 2023-07-19 09:13:15 UTC 
I have seen that in the CI job https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-neutron-17.1_director-rhel-virthost-3cont_2comp-ipv4-geneve-gate-ovn/367//artifact/tempest-results/tempest-results-neutron.1.html where one test failed due to error 500 returned from neutron server.

In the neutron logs http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-neutron-17.1_director-rhel-virthost-3cont_2comp-ipv4-geneve-gate-ovn/367/controller-0/var/log/containers/neutron/server.log.1.gz I found stacktrace:

2023-07-13 20:20:41.690 16 ERROR ovsdbapp.backend.ovs_idl.transaction [req-7a3a9947-941f-4130-b42b-ec4f960d534e ae4395b71d784cb08045c1dd5e5088c2 44ae1f6f6a07483aab7f5499b0aa60c3 - default default] Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 131, in do_commit
    self.post_commit(txn)
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 72, in post_commit
    command.post_commit(txn)
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 94, in post_commit
    row = self.api.tables[self.table_name].rows[real_uuid]
  File "/usr/lib64/python3.9/collections/__init__.py", line 1058, in __getitem__
    raise KeyError(key)
KeyError: UUID('d30e854c-6105-4a2e-b23f-560310ef94fd')

2023-07-13 20:20:41.690 18 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-4ff08062-d8ab-4230-a44c-3b05ad87ab51 - - - - -] Hash Ring: Node b41047f8-b124-44bb-881e-ea3dc129ab43 (host: controller-0.redhat.local) handling event "delete" for row d30e854c-6105-4a2e-b23f-560310ef94fd (table: QoS) notify /usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:769
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager [req-7a3a9947-941f-4130-b42b-ec4f960d534e ae4395b71d784cb08045c1dd5e5088c2 44ae1f6f6a07483aab7f5499b0aa60c3 - default default] Extension driver 'OVNQosDriver' failed in update_policy: KeyError: UUID('d30e854c-6105-4a2e-b23f-560310ef94fd')
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager Traceback (most recent call last):
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/neutron/services/qos/drivers/manager.py", line 105, in call
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     getattr(driver, method_name)(*args, **kwargs)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/neutron/services/qos/drivers/ovn/driver.py", line 70, in update_policy
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     self._driver._ovn_client._qos_driver.update_policy(context, policy)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/extensions/qos.py", line 501, in update_policy
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     self.update_router(txn, router)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     next(self.gen)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 269, in transaction
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     yield t
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     next(self.gen)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 110, in transaction
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     del self._nested_txns_map[cur_thread_id]
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 61, in __exit__
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     self.result = self.commit()
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     raise result.ex
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     txn.results.put(txn.do_commit())
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 131, in do_commit
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     self.post_commit(txn)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 72, in post_commit
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     command.post_commit(txn)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 94, in post_commit
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     row = self.api.tables[self.table_name].rows[real_uuid]
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager   File "/usr/lib64/python3.9/collections/__init__.py", line 1058, in __getitem__
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager     raise KeyError(key)
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager KeyError: UUID('d30e854c-6105-4a2e-b23f-560310ef94fd')
2023-07-13 20:20:41.691 16 ERROR neutron.services.qos.drivers.manager 
2023-07-13 20:20:41.694 18 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-4ff08062-d8ab-4230-a44c-3b05ad87ab51 - - - - -] Hash Ring: Node b41047f8-b124-44bb-881e-ea3dc129ab43 (host: controller-0.redhat.local) handling event "update" for row 1fc372ad-8b21-4ba8-b4c9-5ad443a9a0b5 (table: QoS) notify /usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:769
2023-07-13 20:20:41.694 18 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-4ff08062-d8ab-4230-a44c-3b05ad87ab51 - - - - -] Hash Ring: Node b41047f8-b124-44bb-881e-ea3dc129ab43 (host: controller-0.redhat.local) handling event "update" for row a4ebc050-7ca3-4013-b1a3-401d82fcbd74 (table: QoS) notify /usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:769
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource [req-7a3a9947-941f-4130-b42b-ec4f960d534e ae4395b71d784cb08045c1dd5e5088c2 44ae1f6f6a07483aab7f5499b0aa60c3 - default default] update failed: No details.: neutron_lib.exceptions.DriverCallError
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource Traceback (most recent call last):
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/api/v2/resource.py", line 98, in resource
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/api/v2/base.py", line 630, in update
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     return self._update(request, id, body, **kwargs)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     self.force_reraise()
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     raise self.value
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     self.force_reraise()
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     raise self.value
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     self.force_reraise()
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     raise self.value
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/api/v2/base.py", line 695, in _update
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     obj = obj_updater(request.context, id, **kwargs)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/extensions/qos.py", line 182, in <lambda>
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     return lambda *args, **kwargs: _make_call(
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/extensions/qos.py", line 178, in _make_call
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     return getattr(self, method_name)(
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/db/db_base_plugin_common.py", line 48, in inner
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     result = f(*args, **kwargs)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/services/qos/qos_plugin.py", line 635, in update_policy_rule
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     self.driver_manager.call(qos_consts.UPDATE_POLICY, context, policy)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource   File "/usr/lib/python3.9/site-packages/neutron/services/qos/drivers/manager.py", line 114, in call
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource     raise exceptions.DriverCallError(exc_list=exc_list)
2023-07-13 20:20:41.694 16 ERROR neutron.api.v2.resource neutron_lib.exceptions.DriverCallError
Comment 1 Eran Kuris 2023-07-19 12:38:16 UTC 
Mostly related to CI, we ran it several times without reproducer