Bug 1269201

Summary: Backport request: Improve DVR scale performance
Product: Red Hat OpenStack Reporter: Randy Rubins <rrubins>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Itzik Brown <itbrown>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: amuller, bhoefer, chenders, chrisw, dcadzow, dmaley, ihrachys, jlibosva, nyechiel, ohochman, tfreger, yeylon
Target Milestone: asyncKeywords: FeatureBackport, Triaged, ZStream
Target Release: 6.0 (Juno)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-2014.2.3-19.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-15 16:10:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Randy Rubins 2015-10-06 15:37:17 UTC
Description of problem:

Need to backport upstream Neutron fix 150110 "Improve DVR scale performance" to RHEL-OSP6.

Version-Release number of selected component (if applicable):
RHEL-OSP-6 (Juno) A4 (latest)

How reproducible:

Steps to Reproduce:
1. Create 5 or more instances
2. Attach floating ips to the instances
3. Delete instances

Actual results:
The deletion causes ERROR in l3-agent (PortNotFound), which in turn triggers ERROR 'Failed to fetch router information' for the involved router.  This in turn causes 'fullsync' of all Neutron routers, which is a very cpu intensive process in an environment with ~100-1000 routers.  The high load averages (70-80% system time) cause the OpenStack environment to become sluggish as the Neutron L3 agent and neutron-server are running on the 3 HA controllers (co-located with other OpenStack services).

Expected results:
The deletion of multiple instances should NOT cause PortNotFound error, thus avoiding unnecessary full router sync.

Additional info:
The issue discovered/confirmed in IRC session with Assaf Muller.
According to amuller, the fix is contained in
https://review.openstack.org/#/c/150110/

Comment 2 Assaf Muller 2015-10-06 15:42:48 UTC
Patch is present in OSP 7+, we need an OSP 6 downstream backport.

Comment 16 Itzik Brown 2015-10-15 13:57:48 UTC
Checked with openstack-neutron-2014.2.3-19.el7ost.noarch

Comment 18 errata-xmlrpc 2015-10-15 16:10:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1909.html