Bug 1533592
| Summary: | Neutron ERROR DBDuplicateEntry ipallocations | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marc Methot <mmethot> | |
| Component: | openstack-neutron | Assignee: | Ihar Hrachyshka <ihrachys> | |
| Status: | CLOSED ERRATA | QA Contact: | Toni Freger <tfreger> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 9.0 (Mitaka) | CC: | amuller, ccollett, chrisw, ihrachys, jlibosva, jraju, mbayer, nyechiel, oblaut, ragiman, srevivo | |
| Target Milestone: | zstream | Keywords: | Triaged, ZStream | |
| Target Release: | 9.0 (Mitaka) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-neutron-8.4.0-17.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
When multiple API requests that allocate IP addresses in the same subnet are issued, sometimes the IPAM layer raises a duplicate error. Previously, when this happened, the API layer would not retry the allocation; instead it bubbled up the error as HTTP error code 5xx to the API consumer. Consequently, API requests would sometimes fail with error code 5xx.
With this update, the Neutron API layer now retries all duplicate errors. This allows the IPAM layer to allocate a new, unused address, and successfully fulfil the API request. As a result, when the IPAM layer hits a duplicate error from allocating two addresses from the same subnet at the same time, no 5xx errors are bubbled up to the API consumers.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1544561 (view as bug list) | Environment: | ||
| Last Closed: | 2018-03-15 12:41:29 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1544561 | |||
Openstack installed rpms openstack-aodh-api-2.0.5-1.el7ost.noarch openstack-aodh-common-2.0.5-1.el7ost.noarch openstack-aodh-evaluator-2.0.5-1.el7ost.noarch openstack-aodh-listener-2.0.5-1.el7ost.noarch openstack-aodh-notifier-2.0.5-1.el7ost.noarch openstack-ceilometer-api-6.1.3-2.el7ost.noarch openstack-ceilometer-central-6.1.3-2.el7ost.noarch openstack-ceilometer-collector-6.1.3-2.el7ost.noarch openstack-ceilometer-common-6.1.3-2.el7ost.noarch openstack-ceilometer-compute-6.1.3-2.el7ost.noarch openstack-ceilometer-notification-6.1.3-2.el7ost.noarch openstack-ceilometer-polling-6.1.3-2.el7ost.noarch openstack-cinder-8.1.1-4.el7ost.noarch openstack-dashboard-9.0.1-3.el7ost.noarch openstack-dashboard-theme-9.0.1-3.el7ost.noarch openstack-glance-12.0.0-2.el7ost.noarch openstack-gnocchi-api-2.1.3-3.el7ost.noarch openstack-gnocchi-carbonara-2.1.3-3.el7ost.noarch openstack-gnocchi-common-2.1.3-3.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-2.1.3-3.el7ost.noarch openstack-gnocchi-metricd-2.1.3-3.el7ost.noarch openstack-gnocchi-statsd-2.1.3-3.el7ost.noarch openstack-heat-api-6.0.0-12.el7ost.noarch openstack-heat-api-cfn-6.0.0-12.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-12.el7ost.noarch openstack-heat-common-6.0.0-12.el7ost.noarch openstack-heat-engine-6.0.0-12.el7ost.noarch openstack-keystone-9.0.2-1.el7ost.noarch openstack-manila-2.0.0-6.el7ost.noarch openstack-manila-share-2.0.0-6.el7ost.noarch openstack-neutron-8.1.2-13.el7ost.noarch openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch openstack-neutron-common-8.1.2-13.el7ost.noarch openstack-neutron-lbaas-8.0.0-1.el7ost.noarch openstack-neutron-metering-agent-8.1.2-13.el7ost.noarch openstack-neutron-ml2-8.1.2-13.el7ost.noarch openstack-neutron-openvswitch-8.1.2-13.el7ost.noarch openstack-nova-api-13.1.2-9.el7ost.noarch openstack-nova-cert-13.1.2-9.el7ost.noarch openstack-nova-common-13.1.2-9.el7ost.noarch openstack-nova-compute-13.1.2-9.el7ost.noarch openstack-nova-conductor-13.1.2-9.el7ost.noarch openstack-nova-console-13.1.2-9.el7ost.noarch openstack-nova-novncproxy-13.1.2-9.el7ost.noarch openstack-nova-scheduler-13.1.2-9.el7ost.noarch openstack-puppet-modules-8.1.10-1.el7ost.noarch openstack-sahara-4.0.1-2.el7ost.noarch openstack-sahara-api-4.0.1-2.el7ost.noarch openstack-sahara-common-4.0.1-2.el7ost.noarch openstack-sahara-engine-4.0.1-2.el7ost.noarch openstack-sahara-ui-4.0.0-3.el7ost.noarch openstack-selinux-0.7.11-1.el7ost.noarch openstack-swift-2.7.0-2.el7ost.noarch openstack-swift-account-2.7.0-2.el7ost.noarch openstack-swift-container-2.7.0-2.el7ost.noarch openstack-swift-object-2.7.0-2.el7ost.noarch openstack-swift-plugin-swift3-1.10-1.el7ost.noarch openstack-swift-proxy-2.7.0-2.el7ost.noarch openstack-trove-api-5.0.1-1.el7ost.noarch openstack-trove-common-5.0.1-1.el7ost.noarch openstack-trove-conductor-5.0.1-1.el7ost.noarch openstack-trove-taskmanager-5.0.1-1.el7ost.noarch openstack-utils-2015.2-1.el7ost.noarch The patch I linked to should fix the issue. That being said, it's a very invasive patch that changes not only internal implementation (that would need significant regression testing) but also observed behavior of IPAM layer (the order in which addresses are allocated is now not fully predictable; the observed change is well captured by the following change to tempest that was needed to remove assumptions about the order from existing API tests: https://review.openstack.org/#/c/312771). We should consider whether the benefit of getting this fixed outweighs potential risks of breaking IP allocation layer. (Though on first look, the patch is more or less cleanly applicable to Mitaka codebase, so at least producing the backport shouldn't be that big of a deal, unless I miss some implicit dependencies of the fix on some other Newton+ work.) We will discuss applicability of the backport on next bug triage call. There is another patch worth a try: when DuplicateError is bubbled up to api layer, it should make it retry instead of spill 500 on user. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0537 |
Description of problem: The issue seems to occur randomly. The environment has been up and running for 11 months, and the issue has been noticed twice in the last two months. During instance creation the IP assignment fails with: ~~~ 2018-01-10 15:46:21.450 11453 ERROR neutron.api.v2.resource DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '10.10.10.206-1097775a-ea41-4f3b-9951-ca3ac476baa8-66d8b759-3ec' for key 'PRIMARY'") [SQL: u'INSERT INTO ipallocations (port_id, ip_address, subnet_id, network_id) VALUES (%s, %s, %s, %s)'] [parameters: ('aa0accc0-d743-4620-bf50-e9dcff5716f4', u'10.10.10.206', u'1097775a-ea41-4f3b-9951-ca3ac476baa8', u'66d8b759-3ec5-4dac-ab2c-f6c5012ee848')] 2018-01-10 15:46:21.450 11453 ERROR neutron.api.v2.resource ~~~ Full log attached to the bz. Version-Release number of selected component (if applicable): OSP 9 How reproducible: Due to it's randomness no specific steps. If this environment was not production, spam creation of instances should yield this error more often. Additional info: Other bugs having similar issues and none are resolved https://bugs.launchpad.net/neutron/+bug/1369590 https://bugs.launchpad.net/neutron/+bug/1484379 https://bugs.launchpad.net/neutron/+bug/1341875