Bug 1737803 - Subnets without networks at the database breaking the neutron_api
Summary: Subnets without networks at the database breaking the neutron_api
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Rodolfo Alonso
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks: 1784333 1858960
TreeView+ depends on / blocked
 
Reported: 2019-08-06 09:11 UTC by Luis Tomas Bolivar
Modified: 2020-07-20 22:12 UTC (History)
8 users (show)

Fixed In Version: openstack-neutron-13.0.4-0.20190508213456.de95bc6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1783076 1858960 (view as bug list)
Environment:
Last Closed: 2020-02-27 20:42:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1839658 0 None None None 2019-11-05 18:22:10 UTC
OpenStack gerrit 677221 0 None MERGED Force "network_id" in "subnet" DB registers 2020-12-08 01:54:33 UTC
Red Hat Knowledge Base (Solution) 5237651 0 None None None 2020-07-20 22:12:37 UTC

Description Luis Tomas Bolivar 2019-08-06 09:11:48 UTC
When installing OCP on top of OSP we use tagging to identify what resources belong to a specific OCP cluster. Then, to delete such cluster the openshift installer simply looks for the resources tagged with the specific ID and triggers their deletion (ports, subnets, networks, security groups).

In the case when kuryr is enable, such OCP environments will have many networks/subnets (on per namespace) and therefore there is a lot of concurrent calls to the neutron_api to delete the associated networks/subnets.

In such case, the 'openstack subnet list' command stop working with the following log at the neutron_api:

019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200  len: 248 time: 0.0012720
2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return self.get(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     **query_params)}
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     for subnet_object in subnet_objs
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     'network_id': subnet['network_id'],
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return getattr(self, name)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return getattr(self, attrname)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 
2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500  len: 344 time: 0.7188911
2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200  len: 248 time: 0.0012681


Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet:
MariaDB [ovs_neutron]> select * from subnets;
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
| project_id                       | id                                   | name                                        | network_id                           | ip_version | cidr           | gateway_ip  | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id                        | standard_attr_id | segment_id |
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
| 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet                              | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 |          4 | 172.24.0.0/16  | 172.24.0.1  |           1 | NULL         | NULL              | NULL                                 |                8 | NULL       |
| dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL                                 |          4 | 10.128.34.0/24 | 10.128.34.1 |           0 | NULL         | NULL              | cc48743c-e44c-48ee-b29c-5ceb53f60fbd |             9493 | NULL       |
| 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet                             | 004f76a6-3867-46f4-8718-5498634945a3 |          4 | 10.0.0.0/24    | 10.0.0.1    |           0 | NULL         | NULL              | NULL                                 |               27 | NULL       |
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+


This is only happening rarely, not 100% of the time.

Comment 17 August Simonelli 2019-12-12 23:51:15 UTC
I've created https://bugzilla.redhat.com/show_bug.cgi?id=1783076 as a backport to OSP 13 request for this. I think we need it for OCP on OSP.


Note You need to log in before you can comment on or make changes to this bug.