Bug 1269201 - Backport request: Improve DVR scale performance
Backport request: Improve DVR scale performance
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
6.0 (Juno)
x86_64 Linux
high Severity high
: async
: 6.0 (Juno)
Assigned To: Jakub Libosvar
Itzik Brown
: Triaged, ZStream
Depends On:
  Show dependency treegraph
Reported: 2015-10-06 11:37 EDT by Randy Rubins
Modified: 2016-04-26 13:29 EDT (History)
12 users (show)

See Also:
Fixed In Version: openstack-neutron-2014.2.3-19.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-10-15 12:10:25 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1413314 None None None Never
OpenStack gerrit 150110 None None None Never

  None (edit)
Description Randy Rubins 2015-10-06 11:37:17 EDT
Description of problem:

Need to backport upstream Neutron fix 150110 "Improve DVR scale performance" to RHEL-OSP6.

Version-Release number of selected component (if applicable):
RHEL-OSP-6 (Juno) A4 (latest)

How reproducible:

Steps to Reproduce:
1. Create 5 or more instances
2. Attach floating ips to the instances
3. Delete instances

Actual results:
The deletion causes ERROR in l3-agent (PortNotFound), which in turn triggers ERROR 'Failed to fetch router information' for the involved router.  This in turn causes 'fullsync' of all Neutron routers, which is a very cpu intensive process in an environment with ~100-1000 routers.  The high load averages (70-80% system time) cause the OpenStack environment to become sluggish as the Neutron L3 agent and neutron-server are running on the 3 HA controllers (co-located with other OpenStack services).

Expected results:
The deletion of multiple instances should NOT cause PortNotFound error, thus avoiding unnecessary full router sync.

Additional info:
The issue discovered/confirmed in IRC session with Assaf Muller.
According to amuller, the fix is contained in
Comment 2 Assaf Muller 2015-10-06 11:42:48 EDT
Patch is present in OSP 7+, we need an OSP 6 downstream backport.
Comment 16 Itzik Brown 2015-10-15 09:57:48 EDT
Checked with openstack-neutron-2014.2.3-19.el7ost.noarch
Comment 18 errata-xmlrpc 2015-10-15 12:10:25 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.