Bug 1269201 - Backport request: Improve DVR scale performance
Summary: Backport request: Improve DVR scale performance
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 6.0 (Juno)
Hardware: x86_64
OS: Linux
Target Milestone: async
: 6.0 (Juno)
Assignee: Jakub Libosvar
QA Contact: Itzik Brown
Depends On:
TreeView+ depends on / blocked
Reported: 2015-10-06 15:37 UTC by Randy Rubins
Modified: 2019-08-15 05:36 UTC (History)
13 users (show)

Fixed In Version: openstack-neutron-2014.2.3-19.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2015-10-15 16:10:25 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1413314 0 None None None Never
OpenStack gerrit 150110 0 None None None Never
Red Hat Product Errata RHSA-2015:1909 0 normal SHIPPED_LIVE Moderate: openstack-neutron security and bug fix update 2015-10-15 20:09:00 UTC

Description Randy Rubins 2015-10-06 15:37:17 UTC
Description of problem:

Need to backport upstream Neutron fix 150110 "Improve DVR scale performance" to RHEL-OSP6.

Version-Release number of selected component (if applicable):
RHEL-OSP-6 (Juno) A4 (latest)

How reproducible:

Steps to Reproduce:
1. Create 5 or more instances
2. Attach floating ips to the instances
3. Delete instances

Actual results:
The deletion causes ERROR in l3-agent (PortNotFound), which in turn triggers ERROR 'Failed to fetch router information' for the involved router.  This in turn causes 'fullsync' of all Neutron routers, which is a very cpu intensive process in an environment with ~100-1000 routers.  The high load averages (70-80% system time) cause the OpenStack environment to become sluggish as the Neutron L3 agent and neutron-server are running on the 3 HA controllers (co-located with other OpenStack services).

Expected results:
The deletion of multiple instances should NOT cause PortNotFound error, thus avoiding unnecessary full router sync.

Additional info:
The issue discovered/confirmed in IRC session with Assaf Muller.
According to amuller, the fix is contained in

Comment 2 Assaf Muller 2015-10-06 15:42:48 UTC
Patch is present in OSP 7+, we need an OSP 6 downstream backport.

Comment 16 Itzik Brown 2015-10-15 13:57:48 UTC
Checked with openstack-neutron-2014.2.3-19.el7ost.noarch

Comment 18 errata-xmlrpc 2015-10-15 16:10:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.