Description of problem: The issue seems to occur randomly. The environment has been up and running for 11 months, and the issue has been noticed twice in the last two months. During instance creation the IP assignment fails with: ~~~ 2018-01-10 15:46:21.450 11453 ERROR neutron.api.v2.resource DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '10.10.10.206-1097775a-ea41-4f3b-9951-ca3ac476baa8-66d8b759-3ec' for key 'PRIMARY'") [SQL: u'INSERT INTO ipallocations (port_id, ip_address, subnet_id, network_id) VALUES (%s, %s, %s, %s)'] [parameters: ('aa0accc0-d743-4620-bf50-e9dcff5716f4', u'10.10.10.206', u'1097775a-ea41-4f3b-9951-ca3ac476baa8', u'66d8b759-3ec5-4dac-ab2c-f6c5012ee848')] 2018-01-10 15:46:21.450 11453 ERROR neutron.api.v2.resource ~~~ Full log attached to the bz. Version-Release number of selected component (if applicable): OSP 9 How reproducible: Due to it's randomness no specific steps. If this environment was not production, spam creation of instances should yield this error more often. Additional info: Other bugs having similar issues and none are resolved https://bugs.launchpad.net/neutron/+bug/1369590 https://bugs.launchpad.net/neutron/+bug/1484379 https://bugs.launchpad.net/neutron/+bug/1341875
Openstack installed rpms openstack-aodh-api-2.0.5-1.el7ost.noarch openstack-aodh-common-2.0.5-1.el7ost.noarch openstack-aodh-evaluator-2.0.5-1.el7ost.noarch openstack-aodh-listener-2.0.5-1.el7ost.noarch openstack-aodh-notifier-2.0.5-1.el7ost.noarch openstack-ceilometer-api-6.1.3-2.el7ost.noarch openstack-ceilometer-central-6.1.3-2.el7ost.noarch openstack-ceilometer-collector-6.1.3-2.el7ost.noarch openstack-ceilometer-common-6.1.3-2.el7ost.noarch openstack-ceilometer-compute-6.1.3-2.el7ost.noarch openstack-ceilometer-notification-6.1.3-2.el7ost.noarch openstack-ceilometer-polling-6.1.3-2.el7ost.noarch openstack-cinder-8.1.1-4.el7ost.noarch openstack-dashboard-9.0.1-3.el7ost.noarch openstack-dashboard-theme-9.0.1-3.el7ost.noarch openstack-glance-12.0.0-2.el7ost.noarch openstack-gnocchi-api-2.1.3-3.el7ost.noarch openstack-gnocchi-carbonara-2.1.3-3.el7ost.noarch openstack-gnocchi-common-2.1.3-3.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-2.1.3-3.el7ost.noarch openstack-gnocchi-metricd-2.1.3-3.el7ost.noarch openstack-gnocchi-statsd-2.1.3-3.el7ost.noarch openstack-heat-api-6.0.0-12.el7ost.noarch openstack-heat-api-cfn-6.0.0-12.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-12.el7ost.noarch openstack-heat-common-6.0.0-12.el7ost.noarch openstack-heat-engine-6.0.0-12.el7ost.noarch openstack-keystone-9.0.2-1.el7ost.noarch openstack-manila-2.0.0-6.el7ost.noarch openstack-manila-share-2.0.0-6.el7ost.noarch openstack-neutron-8.1.2-13.el7ost.noarch openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch openstack-neutron-common-8.1.2-13.el7ost.noarch openstack-neutron-lbaas-8.0.0-1.el7ost.noarch openstack-neutron-metering-agent-8.1.2-13.el7ost.noarch openstack-neutron-ml2-8.1.2-13.el7ost.noarch openstack-neutron-openvswitch-8.1.2-13.el7ost.noarch openstack-nova-api-13.1.2-9.el7ost.noarch openstack-nova-cert-13.1.2-9.el7ost.noarch openstack-nova-common-13.1.2-9.el7ost.noarch openstack-nova-compute-13.1.2-9.el7ost.noarch openstack-nova-conductor-13.1.2-9.el7ost.noarch openstack-nova-console-13.1.2-9.el7ost.noarch openstack-nova-novncproxy-13.1.2-9.el7ost.noarch openstack-nova-scheduler-13.1.2-9.el7ost.noarch openstack-puppet-modules-8.1.10-1.el7ost.noarch openstack-sahara-4.0.1-2.el7ost.noarch openstack-sahara-api-4.0.1-2.el7ost.noarch openstack-sahara-common-4.0.1-2.el7ost.noarch openstack-sahara-engine-4.0.1-2.el7ost.noarch openstack-sahara-ui-4.0.0-3.el7ost.noarch openstack-selinux-0.7.11-1.el7ost.noarch openstack-swift-2.7.0-2.el7ost.noarch openstack-swift-account-2.7.0-2.el7ost.noarch openstack-swift-container-2.7.0-2.el7ost.noarch openstack-swift-object-2.7.0-2.el7ost.noarch openstack-swift-plugin-swift3-1.10-1.el7ost.noarch openstack-swift-proxy-2.7.0-2.el7ost.noarch openstack-trove-api-5.0.1-1.el7ost.noarch openstack-trove-common-5.0.1-1.el7ost.noarch openstack-trove-conductor-5.0.1-1.el7ost.noarch openstack-trove-taskmanager-5.0.1-1.el7ost.noarch openstack-utils-2015.2-1.el7ost.noarch
The patch I linked to should fix the issue. That being said, it's a very invasive patch that changes not only internal implementation (that would need significant regression testing) but also observed behavior of IPAM layer (the order in which addresses are allocated is now not fully predictable; the observed change is well captured by the following change to tempest that was needed to remove assumptions about the order from existing API tests: https://review.openstack.org/#/c/312771). We should consider whether the benefit of getting this fixed outweighs potential risks of breaking IP allocation layer. (Though on first look, the patch is more or less cleanly applicable to Mitaka codebase, so at least producing the backport shouldn't be that big of a deal, unless I miss some implicit dependencies of the fix on some other Newton+ work.) We will discuss applicability of the backport on next bug triage call.
There is another patch worth a try: when DuplicateError is bubbled up to api layer, it should make it retry instead of spill 500 on user.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0537