| Summary: |
Subnets without networks at the database breaking the neutron_api |
| Product: |
Red Hat OpenStack
|
Reporter: |
Luis Tomas Bolivar <ltomasbo> |
| Component: |
openstack-neutron | Assignee: |
Rodolfo Alonso <ralonsoh> |
| Status: |
CLOSED
CURRENTRELEASE
|
QA Contact: |
Eran Kuris <ekuris> |
| Severity: |
high
|
Docs Contact: |
|
| Priority: |
high
|
|
|
| Version: |
14.0 (Rocky) | CC: |
amuller, asimonel, chrisw, dhill, jappleii, njohnston, scohen, skaplons
|
| Target Milestone: |
--- | Keywords: |
Reopened, Triaged, ZStream |
| Target Release: |
--- | |
|
| Hardware: |
Unspecified | |
|
| OS: |
Unspecified | |
|
| Whiteboard: |
|
|
Fixed In Version:
|
openstack-neutron-13.0.4-0.20190508213456.de95bc6.el7ost
|
Doc Type:
|
If docs needed, set a value
|
|---|
|
Doc Text:
|
|
Story Points:
|
---
|
|---|
|
Clone Of:
|
|
|---|
|
:
|
1783076 1858960
(view as bug list) |
Environment:
|
|
|
Last Closed:
|
2020-02-27 20:42:09 UTC
|
Type:
|
Bug
|
|---|
|
Regression:
|
---
|
Mount Type:
|
---
|
|---|
|
Documentation:
|
---
|
CRM:
|
|
|---|
|
Verified Versions:
|
|
Category:
|
---
|
|---|
|
oVirt Team:
|
---
|
RHEL 7.3 requirements from Atomic Host:
|
|
|---|
|
Cloudforms Team:
|
---
|
Target Upstream Version:
|
|
|---|
|
Embargoed:
|
|
| |
|---|
| Bug Depends On: |
|
|
|
| Bug Blocks: |
1784333, 1858960
|
|
|
When installing OCP on top of OSP we use tagging to identify what resources belong to a specific OCP cluster. Then, to delete such cluster the openshift installer simply looks for the resources tagged with the specific ID and triggers their deletion (ports, subnets, networks, security groups). In the case when kuryr is enable, such OCP environments will have many networks/subnets (on per namespace) and therefore there is a lot of concurrent calls to the neutron_api to delete the associated networks/subnets. In such case, the 'openstack subnet list' command stop working with the following log at the neutron_api: 019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012720 2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id' 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation **query_params)} 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation for subnet_object in subnet_objs 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 'network_id': subnet['network_id'], 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, name) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, attrname) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id' 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500 len: 344 time: 0.7188911 2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012681 Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet: MariaDB [ovs_neutron]> select * from subnets; +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ | project_id | id | name | network_id | ip_version | cidr | gateway_ip | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id | standard_attr_id | segment_id | +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ | 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 | 4 | 172.24.0.0/16 | 172.24.0.1 | 1 | NULL | NULL | NULL | 8 | NULL | | dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL | 4 | 10.128.34.0/24 | 10.128.34.1 | 0 | NULL | NULL | cc48743c-e44c-48ee-b29c-5ceb53f60fbd | 9493 | NULL | | 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet | 004f76a6-3867-46f4-8718-5498634945a3 | 4 | 10.0.0.0/24 | 10.0.0.1 | 0 | NULL | NULL | NULL | 27 | NULL | +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ This is only happening rarely, not 100% of the time.