1511394 – FlushError on IPAllocations when multiple subnets in the same network deleted

Bug 1511394 - FlushError on IPAllocations when multiple subnets in the same network deleted

Summary: FlushError on IPAllocations when multiple subnets in the same network deleted

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	10.0 (Newton)
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	z9
Target Release:	10.0 (Newton)
Assignee:	Slawek Kaplonski
QA Contact:	Toni Freger
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1572253 1572254
TreeView+	depends on / blocked

Reported:	2017-11-09 09:51 UTC by Sergii Mykhailushko
Modified:	2022-07-09 11:47 UTC (History)
CC List:	12 users (show)
Fixed In Version:	openstack-neutron-9.4.1-21.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1572253 1572254 (view as bug list)
Environment:
Last Closed:	2018-09-17 16:52:18 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	539022	None	ABANDONED	add scenario to create / delete subnets in parallel threads	2020-04-28 10:56:39 UTC
OpenStack gerrit	543042	None	ABANDONED	Refresh port fixed_ips after adding/removing ip allocations	2020-04-28 10:56:39 UTC
Red Hat Issue Tracker	OSP-4758	None	None	None	2022-07-09 11:47:46 UTC
Red Hat Product Errata	RHSA-2018:2715	None	None	None	2018-09-17 16:54:15 UTC

Description Sergii Mykhailushko 2017-11-09 09:51:31 UTC

- Description of problem:

When deleting a stack for a tenant network Heat engine reports  an error: "Delete failed internal server error".

neutron server log:

~~~
[26/Sep/2017 07:55:21] "GET /v2.0/subnets.json?id=74f06b6b-fb9d-4564-86a5-3e5b1720a852 HTTP/1.1" 200 936 0.384428
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource [req-8e698ac2-fad0-404a-bcca-5585684fe81d deace90de3284745b5bcaeffb3234218 f8382e80c05744c2a4ee6b101f9bf345 - - -] dele
te failed: No details.
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 93, in resource
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 556, in delete
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     traceback.format_exc())
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 578, in _delete
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 804, in inner
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1060, in delete_subnet
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return super(Ml2Plugin, self).delete_subnet(context, id)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 159, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     traceback.format_exc())
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 917, in delete_subnet
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self._remove_subnet_ip_allocations_from_ports(context, id)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 159, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     traceback.format_exc())
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 910, in _remove_subnet_ip_allocations_from_ports
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     auto_subnet=is_auto_addr_subnet)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 159, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     traceback.format_exc())
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 875, in _remove_subnet_from_port
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.update_port(context, port_id, data)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 804, in inner
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 159, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     traceback.format_exc())
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.force_reraise()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1289, in update_port
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     port)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 159, in wrapped
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1248, in update_port
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     raise os_db_exc.RetryRequest(e)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 502, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.rollback()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 499, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.commit()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self._prepare_impl()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self.session.flush()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     self._flush(objects)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     transaction.rollback(_capture_exception=True)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     flush_context.execute()
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     rec.execute(self)
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     uow
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 149, in save_obj
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     base_mapper, states, uowtransaction
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 301, in _organize_states_for_save
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource     state_str(existing)))
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource FlushError: New instance <IPAllocation at 0x6a38c10> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('
fd5b:1fd5:8295:5339::5', u'19891a0b-4450-425e-8398-1121675dfc14', u'1a7ca8dd-c03e-4c50-a632-ebbef479d0fc')) conflicts with persistent instance <IPAllocation at 0x8005cd0>
2017-09-26 07:55:23.044 334498 ERROR neutron.api.v2.resource
~~~

This trace possibly leads to upstream bug 1706750 (https://bugs.launchpad.net/neutron/+bug/1706750).

From that discussion we see that the issue was fixed in the openstack/neutron 11.0.0.0rc1 release candidate. Still we have 10.0.3-1 available from OpenStack 11 channels, and 11.0.1-3 in OpenStack beta channels. Is it possible to have this fix backported to RHOSP10?

- Version-Release number of selected component (if applicable):

openstack-neutron-9.4.0-2.el7ost.noarch
openstack-neutron-bigswitch-agent-9.42.7-1.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.42.7-1.el7ost.noarch
openstack-neutron-common-9.4.0-2.el7ost.noarch
openstack-neutron-lbaas-9.2.1-3.el7ost.noarch
openstack-neutron-metering-agent-9.4.0-2.el7ost.noarch
openstack-neutron-ml2-9.4.0-2.el7ost.noarch
openstack-neutron-openvswitch-9.4.0-2.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.4.0-2.el7ost.noarch
puppet-neutron-9.5.0-3.el7ost.noarch
python-neutron-9.4.0-2.el7ost.noarch
python-neutron-lbaas-9.2.1-3.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
python-neutron-tests-9.4.0-2.el7ost.noarch
python-neutronclient-6.0.0-2.el7ost.noarch

Comment 2 Ihar Hrachyshka 2017-11-13 19:00:15 UTC

The patch linked is not the only one needed. I am working on two more that hopefully should help the failure. The work in in scope of https://bugzilla.redhat.com/show_bug.cgi?id=1470784 Once I have all patches set and proved to work, I will update the bug.

Comment 4 Ihar Hrachyshka 2018-01-24 22:29:52 UTC

I initially thought that the bug would need more than a single fix, including https://review.openstack.org/#/c/529616/ but it seems that the latter patch is for a bug that should affect Pike+ only, so we shouldn't need it for Newton.

I will have another look at whether we have all patches needed lined up for Newton this week. Expect some results till Friday.

Comment 5 Ihar Hrachyshka 2018-01-24 23:47:15 UTC

I just checked and neither of fixes for the upstream bug apply to Newton. It is some other bug I believe.

Comment 6 Ihar Hrachyshka 2018-01-26 23:09:55 UTC

Today I made some progress on the issue. Some details below.

===

1. I don't think upstream bugs linked before are related to this failure despite similarity. Both patches we have in upstream would apply for Ocata+ only while here we deal with Newton.

2. The network in question has two subnets - ipv4 and ipv6 - before stack removal.

3. It seems that heat doesn't remove the network but instead cleans up each subnet first. It triggers deletions for both subnets in parallel.

First, deletion for IPv4 subnet is handled by controller0:

2017-10-31 10:50:59.482 300543 DEBUG neutron.db.db_base_plugin_v2 [req-e34a8732-1164-46dd-a4af-2c1ab0979d40 484fa285e8d449458cd7c492627b8eed f0bb88b3ea144d4ca633abc090c2aaf9 - - -] Deleting subnet 8f195510-ccce-43c7-b88d-46d4e493ef95 delete_subnet /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:914
2017-10-31 10:50:59.781 300543 DEBUG neutron.db.db_base_plugin_common [req-e34a8732-1164-46dd-a4af-2c1ab0979d40 484fa285e8d449458cd7c492627b8eed f0bb88b3ea144d4ca633abc090c2aaf9 - - -] Delete allocated IP 10.10.0.5 (b5495763-8744-444b-9cfd-6db2db192b08/8f195510-ccce-43c7-b88d-46d4e493ef95) _delete_ip_allocation /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py:99

In parallel but slightly later, deletion for IPv6 subnet arrives on controller1:

2017-10-31 10:51:00.168 127873 Delete allocated IP fd5b:1fd5:8295:5339::5
2017-10-31 10:51:00.173 127873 req-16a6baf7-6469-47c4-b23e-449ca6026a4c Allocated IP 10.10.0.5 (b5495763-8744-444b-9cfd-6db2db192b08/8f195510-ccce-43c7-b88d-46d4e493ef95/70c9da67-a70b-422f-99c9-32346085bbda) _store_ip_allocation /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py:115
2017-10-31 10:51:00.179 127873 ERROR neutron.api.v2.resource FlushError: New instance <IPAllocation at 0x6602610> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('10.10.0.5', u'8f195510-ccce-43c7-b88d-46d4e493ef95', u'b5495763-8744-444b-9cfd-6db2db192b08')) conflicts with persistent instance <IPAllocation at 0x7405290>

The root of the problem is that delete_subnet tries to deallocate IP addresses for the to-be-deleted subnet. It gets all ports that have an allocation for this subnet, then for each port, it fetches fixed_ips, removes subnet-to-be-deleted allocations from the list, and calls update_port with the list.

I think what happens is that fixed_ips are fetched before the other deletion thread completes its work, so fixed_ips will contain allocation for that other subnet. But then when update_port actually gets to remove allocation for ipv6 subnet, it fetches fixed_ips for the port again, now detecting that there is no ipv4 address there, so it thinks that the caller of update_port meant to ADD the just-deleted ipv4 allocation, so it proceeds doing it. This is where we actually hit the error, because the subnet and its allocation is gone.

I think what we should do is make sure that for each new fixed_ips we receive in update_port, corresponding subnets actually exist. If not, we should raise a legit error like SubnetNotFound that will make _remove_subnet_from_port fetch fixed_ips and call update_port again, now with correct allocations.

I will continue looking at the right fix for the issue on Monday.

Comment 7 Ihar Hrachyshka 2018-01-26 23:11:34 UTC

Somewhat related is, why does heat clean up each subnet separately if it could just kill the network in one go? Sounds not very optimal.

Comment 8 Ihar Hrachyshka 2018-01-26 23:19:05 UTC

Another lead to consider is that https://review.openstack.org/#/c/460696/ was merged in 9.4.0 Newton (the version that the customer uses). I probably need to talk to Miguel Lavalle about potential impact of this change in Newton. Another patch of interest is: https://review.openstack.org/#/c/321152/

Comment 9 Ihar Hrachyshka 2018-01-29 20:46:59 UTC

Update on progress here: I am currently working on a rally scenario that would trigger parallel deletions for subnets from the same network. No current rally or tempest test cases do it right now. I hope that with the scenario ready, we will be able to trigger the issue easily, which is the first step to validate the fix I have in mind.

Comment 10 Ihar Hrachyshka 2018-01-29 21:50:18 UTC

I am exploring a new rally scenario here: https://review.openstack.org/#/c/539022

Comment 14 Ihar Hrachyshka 2018-04-16 17:28:46 UTC

There was some progress here with building rally scenario for deleting subnets but we couldn't trigger this exact failure mode with it anyway, so no lead into what can cause the issue right now.

I am moving the bug back to NEW / untriaged / unassigned so that triage team is able to revisit the bug again. We need to find additional engineers to look into the bug, hence I am moving the bug to untriaged to give the team a chance to reassess it.

Comment 15 Ihar Hrachyshka 2018-04-16 17:34:54 UTC

Nevermind my last comment, Slawomir will take it over from me, as per Assaf.

Slawomir, feel free to reach out if you have questions about the bug.

Comment 26 Alex McLeod 2018-09-03 07:57:08 UTC

Hi there,

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field.

The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Thanks,
Alex

Comment 29 errata-xmlrpc 2018-09-17 16:52:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2715

Note You need to log in before you can comment on or make changes to this bug.