1737803 – Subnets without networks at the database breaking the neutron_api

Bug 1737803 - Subnets without networks at the database breaking the neutron_api

Summary: Subnets without networks at the database breaking the neutron_api

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Rodolfo Alonso
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1784333 1858960
TreeView+	depends on / blocked

Reported:	2019-08-06 09:11 UTC by Luis Tomas Bolivar
Modified:	2023-03-24 15:10 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openstack-neutron-13.0.4-0.20190508213456.de95bc6.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1783076 1858960 (view as bug list)
Environment:
Last Closed:	2020-02-27 20:42:09 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1839658	None	None	None	2019-11-05 18:22:10 UTC
OpenStack gerrit	677221	None	MERGED	Force "network_id" in "subnet" DB registers	2020-12-08 01:54:33 UTC
Red Hat Issue Tracker	OSP-20471	None	None	None	2022-11-25 14:28:22 UTC
Red Hat Knowledge Base (Solution)	5237651	None	None	None	2020-07-20 22:12:37 UTC

Description Luis Tomas Bolivar 2019-08-06 09:11:48 UTC

When installing OCP on top of OSP we use tagging to identify what resources belong to a specific OCP cluster. Then, to delete such cluster the openshift installer simply looks for the resources tagged with the specific ID and triggers their deletion (ports, subnets, networks, security groups).

In the case when kuryr is enable, such OCP environments will have many networks/subnets (on per namespace) and therefore there is a lot of concurrent calls to the neutron_api to delete the associated networks/subnets.

In such case, the 'openstack subnet list' command stop working with the following log at the neutron_api:

019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200  len: 248 time: 0.0012720
2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return self.get(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     **query_params)}
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     for subnet_object in subnet_objs
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     'network_id': subnet['network_id'],
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return getattr(self, name)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return getattr(self, attrname)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 
2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500  len: 344 time: 0.7188911
2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200  len: 248 time: 0.0012681


Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet:
MariaDB [ovs_neutron]> select * from subnets;
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
| project_id                       | id                                   | name                                        | network_id                           | ip_version | cidr           | gateway_ip  | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id                        | standard_attr_id | segment_id |
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
| 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet                              | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 |          4 | 172.24.0.0/16  | 172.24.0.1  |           1 | NULL         | NULL              | NULL                                 |                8 | NULL       |
| dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL                                 |          4 | 10.128.34.0/24 | 10.128.34.1 |           0 | NULL         | NULL              | cc48743c-e44c-48ee-b29c-5ceb53f60fbd |             9493 | NULL       |
| 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet                             | 004f76a6-3867-46f4-8718-5498634945a3 |          4 | 10.0.0.0/24    | 10.0.0.1    |           0 | NULL         | NULL              | NULL                                 |               27 | NULL       |
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+


This is only happening rarely, not 100% of the time.

Comment 17 August Simonelli 2019-12-12 23:51:15 UTC

I've created https://bugzilla.redhat.com/show_bug.cgi?id=1783076 as a backport to OSP 13 request for this. I think we need it for OCP on OSP.

Note You need to log in before you can comment on or make changes to this bug.