1940870 – networking-ovn unable to delete stale neutron resources from OVN NBDB

Bug 1940870 - networking-ovn unable to delete stale neutron resources from OVN NBDB

Summary: networking-ovn unable to delete stale neutron resources from OVN NBDB

Keywords:
Status:	CLOSED DUPLICATE of bug 1940724
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-networking-ovn
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-03-19 12:40 UTC by anil venkata
Modified:	2021-03-23 14:37 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-03-23 14:37:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description anil venkata 2021-03-19 12:40:15 UTC

Description of problem:
During our scale testing, we found that router and its port got deleted from neutron DB but not OVN NBDB.
Networking-ovn maintenance task is trying to fix it but throwing below traceback

2021-03-19 08:01:00.271 39 DEBUG futurist.periodics [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Submitting periodic callback 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronizing Neutron and OVN databases check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:341
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Number of inconsistencies found at delete: router_ports=1 _log /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:322
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Fixing resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports) at delete check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:375
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Failed to fix deleted resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports): neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 198, in _get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     context, l3_models.Router, router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 159, in get_by_id
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return query.filter(model.id == object_id).one()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3282, in one
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise orm_exc.NoResultFound("No row was found for one()")
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance During handling of the above exception, another exception occurred:
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 380, in check_for_inconsistencies
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self._fix_delete(row)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 261, in _fix_delete
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     res_map['ovn_delete'](row.resource_uuid)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1502, in delete_router_port
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     router = self._l3_plugin.get_router(context, router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return method(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     setattr(e, '_RETRY_EXCEEDED', True)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     ectxt.value = e.inner_exc
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*dup_args, **dup_kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 533, in get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     router = self._get_router(context, id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 200, in _get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise l3_exc.RouterNotFound(router_id=router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.350 39 INFO networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronization finished (took 0.06 seconds)

In 3 hours duration, this maintenance task tried 32 times and thrown the same traceback. I could still see this stale neutron router in OVN NBDB.

(overcloud) [stack@undercloud 20210319-084055]$ grep -r "networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found" .
./controller-0/root/neutron/server.log:2021-03-19 09:01:41.790 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log:2021-03-19 09:06:16.477 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log:2021-03-19 09:11:00.441 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.1:2021-03-19 08:36:00.398 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.1:2021-03-19 08:41:00.405 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.1:2021-03-19 08:46:31.021 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.1:2021-03-19 08:51:46.241 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.1:2021-03-19 08:56:41.927 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:26:00.282 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:31:00.253 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:36:00.266 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:41:00.255 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:46:00.565 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:51:00.260 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 06:56:12.919 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:01:00.266 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:06:00.267 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:11:00.264 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:16:00.281 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:21:00.302 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:26:56.713 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:32:01.688 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:36:50.438 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:41:09.465 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:46:00.337 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:51:00.330 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 07:56:00.338 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-0/root/neutron/server.log.2:2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-2/root/neutron/server.log.2:2021-03-19 06:12:52.048 40 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-2/root/neutron/server.log.2:2021-03-19 06:17:52.074 40 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
./controller-2/root/neutron/server.log.2:2021-03-19 06:22:52.076 40 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found

Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20210205.n.0
ovn2.13-host-20.12.0-20pvt_dp_groups.el8fdp.x86_64
rhosp-ovn-central-2.13-8.el8ost.noarch
ovn2.13-central-20.12.0-20pvt_dp_groups.el8fdp.x86_64
ovn2.13-vtep-20.12.0-20pvt_dp_groups.el8fdp.x86_64
puppet-ovn-15.4.1-1.20201113171358.192ac4e.el8ost.noarch
rhosp-ovn-2.13-8.el8ost.noarch
ovn2.13-20.12.0-20pvt_dp_groups.el8fdp.x86_64

Note You need to log in before you can comment on or make changes to this bug.