1506668 – IPv6 address association tests frequently reaches 5xx

Bug 1506668 - IPv6 address association tests frequently reaches 5xx

Summary: IPv6 address association tests frequently reaches 5xx

Keywords:
Status:	CLOSED DUPLICATE of bug 1470784
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	12.0 (Pike)
Assignee:	Ihar Hrachyshka
QA Contact:	Toni Freger
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-10-26 14:10 UTC by Attila Fazekas
Modified:	2017-11-10 19:24 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-11-10 19:24:45 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Attila Fazekas 2017-10-26 14:10:49 UTC

Description of problem:
For example tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_stateless_eui64 or tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_two_subnets test can fail with 5xx :

tempest side exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/network/test_dhcp_ipv6.py", line 111, in test_dhcpv6_stateless_eui64
    real_ip, eui_ip = self._get_ips_from_subnet(**kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/api/network/test_dhcp_ipv6.py", line 92, in _get_ips_from_subnet
    subnet = self.create_subnet(self.network, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/api/network/base.py", line 181, in create_subnet
    **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnets_client.py", line 27, in create_subnet
    return self.create_resource(uri, post_data)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 61, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 270, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 836, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault

Version-Release number of selected component (if applicable):
openstack-neutron.noarch           1:11.0.2-0.20171020230401.el7ost
openstack-neutron-common.noarch    1:11.0.2-0.20171020230401.el7ost
openstack-neutron-lbaas.noarch     1:11.0.2-0.20170927152439.743c1db.el7ost
openstack-neutron-lbaas-ui.noarch  3.0.1-2.el7ost      @rhos-12.0-signed        
openstack-neutron-linuxbridge.noarch
openstack-neutron-metering-agent.noarch
openstack-neutron-ml2.noarch       1:11.0.2-0.20171020230401.el7ost
openstack-neutron-openvswitch.noarch
openstack-neutron-sriov-nic-agent.noarch
puppet-neutron.noarch              11.3.1-0.20171005205442.83e8ac7.el7ost
python-neutron.noarch              1:11.0.2-0.20171020230401.el7ost
python-neutron-lbaas.noarch        1:11.0.2-0.20170927152439.743c1db.el7ost
python-neutron-lib.noarch          1.9.1-0.20170821170222.0ef54c3.el7ost
python-neutronclient.noarch        6.5.0-0.20170814170137.355983d.el7ost

20171024.1

How reproducible:
Not always


Additional info:
One of the neutron server side exception:

2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 246, in release_dhcp_port
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     plugin.delete_ports_by_device_id(context, device_id, network_id)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1348, in delete_ports_by_device_id
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.delete_port(context, port_id)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 665, in inner
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1546, in delete_port
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     super(Ml2Plugin, self).delete_port(context, id)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1336, in delete_port
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.ipam.delete_port(context, id)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 429, in delete_port
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     port['fixed_ips'])
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 95, in _ipam_deallocate_ips
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     "external system for %s", addresses)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 72, in _ipam_deallocate_ips
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     ipam_subnet.deallocate(ip['ip_address'])
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 227, in deallocate
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server     ip_address=address)
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server IpAddressAllocationNotFound: Unable to find IP address 2003::f816:3eff:fe73:8a72 on subnet 0c005759-7305-419e-9d0e-3b1ddd4c7373
2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server 
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation [req-6821aae0-ee44-4a2a-afe9-6e26dc1e0589 b2c91cc438c346aaa7fa7627a472fa2b 9d6a5a311c0441c8b2331c7f2daca581 - default default] POST failed.: FlushError: New instance <IPAllocation at 0x8e7f0d0> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('2003::f816:3eff:fe73:8a72', '0c005759-7305-419e-9d0e-3b1ddd4c7373', 'a73d8aa6-5913-4cbd-9282-3f28506b4eb6')) conflicts with persistent instance <IPAllocation at 0x8949850>
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 569, in invoke_controller
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 159, in post
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return self.create(resources)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 177, in create
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return {key: creator(*creator_args, **creator_kwargs)}
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 665, in inner
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1048, in create_subnet
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     result, mech_context = self._create_subnet_db(context, subnet)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1040, in _create_subnet_db
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self._create_subnet_postcommit(context, result, net_db, ipam_sub)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 689, in _create_subnet_postcommit
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     result, ipam_subnet)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 480, in add_auto_addrs_on_network_ports
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     allocated.create()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 225, in decorator
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     res = func(self, *args, **kwargs)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 633, in create
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.modify_fields_to_db(fields))
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 61, in create_object
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     context.session.add(db_obj)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 567, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.rollback()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     compat.reraise(exc_type, exc_value, exc_tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 564, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.commit()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 461, in commit
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self._prepare_impl()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self.session.flush()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in flush
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     self._flush(objects)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     transaction.rollback(_capture_exception=True)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     compat.reraise(exc_type, exc_value, exc_tb)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     flush_context.execute()
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     rec.execute(self)
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     uow
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 156, in save_obj
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     base_mapper, states, uowtransaction
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 312, in _organize_states_for_save
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation     state_str(existing)))
2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation FlushError: New instance <IPAllocation at 0x8e7f0d0> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('2003::f816:3eff:fe73:8a72', '0c005759-7305-419e-9d0e-3b1ddd4c7373', 'a73d8aa6-5913-4cbd-9282-3f28506b4eb6')) conflicts with persistent instance <IPAllocation at 0x8949850>

Comment 2 Jakub Libosvar 2017-10-30 13:29:49 UTC

Brian will have a look and will triage.

Comment 3 Brian Haley 2017-10-30 19:21:08 UTC

This looks like a new issue, but similar to an upstream bug that was fixed recently, https://bugs.launchpad.net/neutron/+bug/1706750 (FlushError for automatic IPAllocations when subnet create is retried)

Looking at the server.log in comment #1 above we can see a subnet-create is ongoing, but has triggered a retry at about 2017-10-25 21:54:45.617 and is in _create_subnet_postcommit().

At the same time a dhcp-agent is trying to delete a port from that same subnet via an RPC call and is in _ipam_deallocate_ips().  This delete port can't find the IP:

IpAddressAllocationNotFound: Unable to find IP address 2003::f816:3eff:fe73:8a72 on subnet 0c005759-7305-419e-9d0e-3b1ddd4c7373

And the subnet-create fails as well trying to allocate the same IP address:

FlushError: New instance <IPAllocation at 0x8e7f0d0> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('2003::f816:3eff:fe73:8a72', '0c005759-7305-419e-9d0e-3b1ddd4c7373', 'a73d8aa6-5913-4cbd-9282-3f28506b4eb6')) conflicts with persistent instance <IPAllocation at 0x8949850>

Sure enough, controller-2 is trying to delete it's dhcp port around 2017-10-25 21:54:44.705 in this log file:

http://cougar11.scl.lab.tlv.redhat.com/phase2-12_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-gre-ceph-containers/60/controller-2.tar.gz?controller-2/var/log/neutron/dhcp-agent.log

It seems to be complaining about reading /var/lib/neutron/dhcp/{network_id}/pid, but doesn't complain when writing the host file, could be a red herring.

BTW, I see a similar 500 error in the controller-2 server.log as well.

Comment 4 Ihar Hrachyshka 2017-11-10 19:24:45 UTC


*** This bug has been marked as a duplicate of bug 1470784 ***

Note You need to log in before you can comment on or make changes to this bug.