Description of problem: When using a 3rd party provider, Octavia resource statuses might be updated multiple times for some reasons (concurrent requests, or successive requests that trigger errors) For instance the ovn-provider, a cascade deletion request triggered an error in OVN and the provider sent a status update to Octavia with some newly deleted resources: 2022-09-02 07:32:51.290 16 INFO octavia.api.v2.controllers.load_balancer [req-8965a055-f039-43bb-94c6-19c71b67c8e0 - 79f6bc108da84222b42dea61f0a97116 - default default] Sending delete Load Balancer 3fe6f3c3-bc9b-4a7a-aee3-617513ef1b8f to provider ovn 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper [-] Exception occurred during deletion of loadbalancer: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with uuid=42019590-84d6-4aea-80d5-5805ee939f1c 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper Traceback (most recent call last): 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py", line 1093, in lb_delete 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper status = self._lb_delete(loadbalancer, ovn_lb, status) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py", line 1168, in _lb_delete 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper self._execute_commands(commands) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py", line 505, in _execute_commands 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper txn.add(command) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper next(self.gen) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 165, in transaction 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper yield t 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper next(self.gen) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 110, in transaction 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper del self._nested_txns_map[cur_thread_id] 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/api.py", line 61, in __exit__ 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper self.result = self.commit() 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper raise result.ex 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper txn.results.put(txn.do_commit()) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper command.run_idl(txn) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 1220, in run_idl 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper ls = self.api.lookup('Logical_Switch', self.switch) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper return self._lookup(table, record) 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 240, in _lookup 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper raise idlutils.RowNotFound(table=table, col='uuid', 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with uuid=42019590-84d6-4aea-80d5-5805ee939f1c 2022-09-02 07:32:51.364 16 ERROR ovn_octavia_provider.helper 2022-09-02 07:32:51.366 16 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '3fe6f3c3-bc9b-4a7a-aee3-617513ef1b8f', 'provisioning_status': 'ERROR', 'operating_status': 'ERROR'}], 'listeners': [{'id': '4e0f41a0-fc84-4f1a-b29c-1e2999ce67f4', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}, {'id': 'd7cf0ad1-06f4-4e7f-9178-a61df93be0af', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}], 'pools': [{'id': '15a8b3ce-ab48-46f4-9bb5-4399cdc00268', 'provisioning_status': 'DELETED'}, {'id': '4c8b7c7a-8187-4495-87c4-899492d1d03c', 'provisioning_status': 'DELETED'}], 'members': [{'id': 'b355ab65-c427-4439-aecb-4bc5db414ada', 'provisioning_status': 'DELETED'}, {'id': '0c4a6b5c-f874-4d26-9101-9f574a07994b', 'provisioning_status': 'DELETED'}]} _update_status_to_octavia /usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py:370 The LB was then in ERROR, another deletion request was sent, ovn-provider sent a new status update: 2022-09-02 07:37:58.442 17 INFO octavia.api.v2.controllers.load_balancer [req-58e71239-90c7-4250-a734-947bba62f46c - 79f6bc108da84222b42dea61f0a97116 - default default] Sending delete Load Balancer 3fe6f3c3-bc9b-4a7a-aee3-617513ef1b8f to provider ovn 2022-09-02 07:37:58.444 17 DEBUG ovn_octavia_provider.helper [-] Handling request lb_delete with info {'id': '3fe6f3c3-bc9b-4a7a-aee3-617513ef1b8f', 'cascade': True} request_handler /usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py:343 2022-09-02 07:37:59.846 17 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '3fe6f3c3-bc9b-4a7a-aee3-617513ef1b8f', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}], 'listeners': [{'id': '4e0f41a0-fc84-4f1a-b29c-1e2999ce67f4', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}, {'id': 'd7cf0ad1-06f4-4e7f-9178-a61df93be0af', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}], 'pools': [{'id': '15a8b3ce-ab48-46f4-9bb5-4399cdc00268', 'provisioning_status': 'DELETED'}, {'id': '4c8b7c7a-8187-4495-87c4-899492d1d03c', 'provisioning_status': 'DELETED'}], 'members': [{'id': 'b355ab65-c427-4439-aecb-4bc5db414ada', 'provisioning_status': 'DELETED'}, {'id': '0c4a6b5c-f874-4d26-9101-9f574a07994b', 'provisioning_status': 'DELETED'}]} _update_status_to_octavia /usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py:370 2022-09-02 07:48:55.134 17 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '3fe6f3c3-bc9b-4a7a-aee3-617513ef1b8f', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}], 'listeners': [{'id': '4e0f41a0-fc84-4f1a-b29c-1e2999ce67f4', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}, {'id': 'd7cf0ad1-06f4-4e7f-9178-a61df93be0af', 'provisioning_status': 'DELETED', 'operating_status': 'OFFLINE'}], 'pools': [{'id': '15a8b3ce-ab48-46f4-9bb5-4399cdc00268', 'provisioning_status': 'DELETED'}, {'id': '4c8b7c7a-8187-4495-87c4-899492d1d03c', 'provisioning_status': 'DELETED'}], 'members': [{'id': 'b355ab65-c427-4439-aecb-4bc5db414ada', 'provisioning_status': 'DELETED'}, {'id': '0c4a6b5c-f874-4d26-9101-9f574a07994b', 'provisioning_status': 'DELETED'}]} _update_status_to_octavia /usr/lib/python3.9/site-packages/ovn_octavia_provider/helper.py:370 2022-09-02 07:48:55.330 17 ERROR ovn_octavia_provider.helper [-] Error while updating the load balancer status: 'NoneType' object has no attribute 'project_id': octavia_lib.api.drivers.exceptions.UpdateStatusError: 'NoneType' object has no attribute 'project_id' The exception "UpdateStatusError: 'NoneType' object has no attribute 'project_id'" was raised, it appears that NoneType was returned when getting the listener from DB (the listener was deleted from the DB in the previous call) Version-Release number of selected component (if applicable): 17.0 How reproducible: randomly
Note that after the 2nd request, the LB was stuck in PENDING_DELETE
*** Bug 2123657 has been marked as a duplicate of this bug. ***
Ran the following: 1. git clone https://github.com/cloud-bulldozer/e2e-benchmarking.git 2. cd e2e-benchmarking/workloads/kube-burner 3. export WORKLOAD=cluster-density; export JOB_ITERATIONS=40 4. ./run.sh Checked with: OSP RHOS-17.0-RHEL-9-20221213.n.1 OCP 4.12.0-rc.4 There was a LB stuck in PENDING_UPDATE state In Octavia logs: 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (LsGetCommand): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-d275ba0b-e5d2-44f4-94f7-146d30efbf42 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 39, in execute 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command self.run_idl(None) 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 329, in run_idl 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.api.lookup(self.table, self.record) 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command return self._lookup(table, record) 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command row = idlutils.row_by_value(self, rl.table, rl.column, record) 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match) 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-d275ba0b-e5d2-44f4-94f7-146d30efbf42 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command
(In reply to Itzik Brown from comment #10) > Ran the following: > 1. git clone https://github.com/cloud-bulldozer/e2e-benchmarking.git > 2. cd e2e-benchmarking/workloads/kube-burner > 3. export WORKLOAD=cluster-density; export JOB_ITERATIONS=40 > 4. ./run.sh > > Checked with: > OSP RHOS-17.0-RHEL-9-20221213.n.1 > OCP 4.12.0-rc.4 > > There was a LB stuck in PENDING_UPDATE state > > In Octavia logs: > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command [-] Error > executing command (LsGetCommand): > ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch > with name=neutron-d275ba0b-e5d2-44f4-94f7-146d30efbf42 > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command Traceback > (most recent call last): > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File > "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line > 39, in execute > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command > self.run_idl(None) > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File > "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line > 329, in run_idl > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command > self.result = self.api.lookup(self.table, self.record) > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File > "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", > line 208, in lookup > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command return > self._lookup(table, record) > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File > "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", > line 268, in _lookup > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command row = > idlutils.row_by_value(self, rl.table, rl.column, record) > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command File > "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", > line 114, in row_by_value > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command raise > RowNotFound(table=table, col=column, match=match) > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command > ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch > with name=neutron-d275ba0b-e5d2-44f4-94f7-146d30efbf42 > 2022-12-18 08:57:40.892 9 ERROR ovsdbapp.backend.ovs_idl.command This looks like a different issue, not related with this bugzilla. It is ok to have those logs (RowNotFound), the loadbalancer will transition to error, and a further delete action will move it to deleted. If it was an error due to this bug not being fixed, the loadbalancer would be in PENDING_DELETE, not in PENDING_UPDATE
Per Luis` comment since no LBs were in ERROR/PENDING_DELETE status I'm close it as verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 17.0.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:0271