Bug 1506668
| Summary: | IPv6 address association tests frequently reaches 5xx | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Attila Fazekas <afazekas> |
| Component: | openstack-neutron | Assignee: | Ihar Hrachyshka <ihrachys> |
| Status: | CLOSED DUPLICATE | QA Contact: | Toni Freger <tfreger> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 12.0 (Pike) | CC: | amuller, chrisw, ihrachys, jlibosva, nyechiel, srevivo, tvignaud |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-10 19:24:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Brian will have a look and will triage. This looks like a new issue, but similar to an upstream bug that was fixed recently, https://bugs.launchpad.net/neutron/+bug/1706750 (FlushError for automatic IPAllocations when subnet create is retried) Looking at the server.log in comment #1 above we can see a subnet-create is ongoing, but has triggered a retry at about 2017-10-25 21:54:45.617 and is in _create_subnet_postcommit(). At the same time a dhcp-agent is trying to delete a port from that same subnet via an RPC call and is in _ipam_deallocate_ips(). This delete port can't find the IP: IpAddressAllocationNotFound: Unable to find IP address 2003::f816:3eff:fe73:8a72 on subnet 0c005759-7305-419e-9d0e-3b1ddd4c7373 And the subnet-create fails as well trying to allocate the same IP address: FlushError: New instance <IPAllocation at 0x8e7f0d0> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('2003::f816:3eff:fe73:8a72', '0c005759-7305-419e-9d0e-3b1ddd4c7373', 'a73d8aa6-5913-4cbd-9282-3f28506b4eb6')) conflicts with persistent instance <IPAllocation at 0x8949850> Sure enough, controller-2 is trying to delete it's dhcp port around 2017-10-25 21:54:44.705 in this log file: http://cougar11.scl.lab.tlv.redhat.com/phase2-12_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-gre-ceph-containers/60/controller-2.tar.gz?controller-2/var/log/neutron/dhcp-agent.log It seems to be complaining about reading /var/lib/neutron/dhcp/{network_id}/pid, but doesn't complain when writing the host file, could be a red herring. BTW, I see a similar 500 error in the controller-2 server.log as well. *** This bug has been marked as a duplicate of bug 1470784 *** |
Description of problem: For example tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_stateless_eui64 or tempest.api.network.test_dhcp_ipv6.NetworksTestDHCPv6.test_dhcpv6_two_subnets test can fail with 5xx : tempest side exception: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tempest/api/network/test_dhcp_ipv6.py", line 111, in test_dhcpv6_stateless_eui64 real_ip, eui_ip = self._get_ips_from_subnet(**kwargs) File "/usr/lib/python2.7/site-packages/tempest/api/network/test_dhcp_ipv6.py", line 92, in _get_ips_from_subnet subnet = self.create_subnet(self.network, **kwargs) File "/usr/lib/python2.7/site-packages/tempest/api/network/base.py", line 181, in create_subnet **kwargs) File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnets_client.py", line 27, in create_subnet return self.create_resource(uri, post_data) File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 61, in create_resource resp, body = self.post(req_uri, req_post_data) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 270, in post return self.request('POST', url, extra_headers, headers, body, chunked) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request self._error_checker(resp, resp_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 836, in _error_checker message=message) tempest.lib.exceptions.ServerFault: Got server fault Version-Release number of selected component (if applicable): openstack-neutron.noarch 1:11.0.2-0.20171020230401.el7ost openstack-neutron-common.noarch 1:11.0.2-0.20171020230401.el7ost openstack-neutron-lbaas.noarch 1:11.0.2-0.20170927152439.743c1db.el7ost openstack-neutron-lbaas-ui.noarch 3.0.1-2.el7ost @rhos-12.0-signed openstack-neutron-linuxbridge.noarch openstack-neutron-metering-agent.noarch openstack-neutron-ml2.noarch 1:11.0.2-0.20171020230401.el7ost openstack-neutron-openvswitch.noarch openstack-neutron-sriov-nic-agent.noarch puppet-neutron.noarch 11.3.1-0.20171005205442.83e8ac7.el7ost python-neutron.noarch 1:11.0.2-0.20171020230401.el7ost python-neutron-lbaas.noarch 1:11.0.2-0.20170927152439.743c1db.el7ost python-neutron-lib.noarch 1.9.1-0.20170821170222.0ef54c3.el7ost python-neutronclient.noarch 6.5.0-0.20170814170137.355983d.el7ost 20171024.1 How reproducible: Not always Additional info: One of the neutron server side exception: 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 246, in release_dhcp_port 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server plugin.delete_ports_by_device_id(context, device_id, network_id) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1348, in delete_ports_by_device_id 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.delete_port(context, port_id) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 665, in inner 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1546, in delete_port 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server super(Ml2Plugin, self).delete_port(context, id) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1336, in delete_port 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.ipam.delete_port(context, id) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 429, in delete_port 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server port['fixed_ips']) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 95, in _ipam_deallocate_ips 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server "external system for %s", addresses) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server self.force_reraise() 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 72, in _ipam_deallocate_ips 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server ipam_subnet.deallocate(ip['ip_address']) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 227, in deallocate 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server ip_address=address) 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server IpAddressAllocationNotFound: Unable to find IP address 2003::f816:3eff:fe73:8a72 on subnet 0c005759-7305-419e-9d0e-3b1ddd4c7373 2017-10-25 21:54:45.861 130093 ERROR oslo_messaging.rpc.server 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation [req-6821aae0-ee44-4a2a-afe9-6e26dc1e0589 b2c91cc438c346aaa7fa7627a472fa2b 9d6a5a311c0441c8b2331c7f2daca581 - default default] POST failed.: FlushError: New instance <IPAllocation at 0x8e7f0d0> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('2003::f816:3eff:fe73:8a72', '0c005759-7305-419e-9d0e-3b1ddd4c7373', 'a73d8aa6-5913-4cbd-9282-3f28506b4eb6')) conflicts with persistent instance <IPAllocation at 0x8949850> 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 569, in invoke_controller 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 159, in post 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 177, in create 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)} 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 665, in inner 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1048, in create_subnet 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation result, mech_context = self._create_subnet_db(context, subnet) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1040, in _create_subnet_db 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self._create_subnet_postcommit(context, result, net_db, ipam_sub) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 163, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 689, in _create_subnet_postcommit 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation result, ipam_subnet) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 480, in add_auto_addrs_on_network_ports 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation allocated.create() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 225, in decorator 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation res = func(self, *args, **kwargs) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 633, in create 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.modify_fields_to_db(fields)) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 61, in create_object 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation context.session.add(db_obj) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 567, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.rollback() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation compat.reraise(exc_type, exc_value, exc_tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 564, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.commit() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 461, in commit 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self._prepare_impl() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self.session.flush() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in flush 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation self._flush(objects) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation transaction.rollback(_capture_exception=True) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation compat.reraise(exc_type, exc_value, exc_tb) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation flush_context.execute() 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation rec.execute(self) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation uow 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 156, in save_obj 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation base_mapper, states, uowtransaction 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 312, in _organize_states_for_save 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation state_str(existing))) 2017-10-25 21:54:45.884 130087 ERROR neutron.pecan_wsgi.hooks.translation FlushError: New instance <IPAllocation at 0x8e7f0d0> with identity key (<class 'neutron.db.models_v2.IPAllocation'>, ('2003::f816:3eff:fe73:8a72', '0c005759-7305-419e-9d0e-3b1ddd4c7373', 'a73d8aa6-5913-4cbd-9282-3f28506b4eb6')) conflicts with persistent instance <IPAllocation at 0x8949850>