Bug 2063177

Summary: Deleting a non-existing security group rule returns an incorrect error
Product: Red Hat OpenStack Reporter: Gregory Thiemonge <gthiemon>
Component: python-networking-ovnAssignee: Terry Wilson <twilson>
Status: CLOSED ERRATA QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: apevec, bbonguar, bcafarel, dhill, egarciar, jschluet, lhh, majopela, scohen, twilson
Target Milestone: z4Keywords: AutomationBlocker, Triaged
Target Release: 16.2 (Train on RHEL 8.4)Flags: twilson: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-networking-ovn-7.4.2-2.20220409154863.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-07 19:21:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gregory Thiemonge 2022-03-11 12:52:47 UTC 
Description of problem:
This bug was observed in an Octavia CI job.
Octavia tries to delete a security group rule that was already deleted and neutron returns a Conflict exception instead of a NotFound exception:

2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     result = task.execute(**arguments)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 512, in execute
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     self.network_driver.update_vip(loadbalancer, for_delete=True)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 627, in update_vip
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     sec_grp.get(constants.ID))
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 210, in _update_security_group_rules
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     self.neutron_client.delete_security_group_rule(rule_id)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 1013, in delete_security_group_rule
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     (security_group_rule))
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 350, in delete
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     headers=headers, params=params)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     headers=headers, params=params)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     self._handle_fault_response(status_code, replybody, resp)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     exception_handler_v20(status_code, error_body)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker     request_ids=request_ids)
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker neutronclient.common.exceptions.Conflict: Security Group Rule bdb82146-b9c7-4c0c-bfd9-5bb43266b0d4 cannot perform before_delete due to Callback networking_ovn.ml2.mech_driver.OVNMechanismDriver._process_sg_rule_notification--9223372036847947572 failed with "Security group rule bdb82146-b9c7-4c0c-bfd9-5bb43266b0d4 does not exist".
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker Neutron server returns request_ids: ['req-b57825a5-a237-4634-a3df-b65c0f2bce54']
2022-03-06 04:53:59.805 22 ERROR octavia.controller.worker.v1.controller_worker 
2022-03-06 04:53:59.818 22 DEBUG octavia.network.drivers.neutron.base [req-b040c510-291d-4d09-9bd9-5b0216b9fd86 - 40e497b59ef246fdbe9c41418d000b51 - - -] Neutron extension security-group found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66
2022-03-06 04:53:59.826 22 DEBUG octavia.network.drivers.neutron.base [req-5fc7d207-bbfa-4496-b821-041ea10502f5 - e3f939325d94401e892a0cf7ab010e7d - - -] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66
2022-03-06 04:53:59.833 22 DEBUG octavia.network.drivers.neutron.base [req-7d29a510-2814-4523-a23c-3b3ff8d84faf - 40e497b59ef246fdbe9c41418d000b51 - - -] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66
2022-03-06 04:53:59.834 22 DEBUG octavia.network.drivers.neutron.base [req-36598b9b-8bc2-4734-b1d6-52591dde4f00 - 40e497b59ef246fdbe9c41418d000b51 - - -] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66
2022-03-06 04:53:59.883 22 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'delete_update_vip_listener_ac77ab07-d10a-4337-8b7f-014f433ac334' (a9af9af3-be4d-4d26-b423-25f37dcf3e97) transitioned into state 'FAILURE' from state 'RUNNING': neutronclient.common.exceptions.Conflict: Security Group Rule bdb82146-b9c7-4c0c-bfd9-5bb43266b0d4 cannot perform before_delete due to Callback networking_ovn.ml2.mech_driver.OVNMechanismDriver._process_sg_rule_notification--9223372036852139332

Octavia catches the NotFound exception but does catch any other exception.

The bug was reported upstream at https://bugs.launchpad.net/neutron/+bug/1933638 but the fix was not backported to train.


Version-Release number of selected component (if applicable):
16.2

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Gregory Thiemonge 2022-03-14 07:29:59 UTC 
Not 100% reproducible, it seems that this is a race condition that was caught last week, not a regression.
Comment 16 errata-xmlrpc 2022-12-07 19:21:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8794