Bug 1372470

Summary: Neutron API Performance regression
Product: Red Hat OpenStack Reporter: Joe Talerico <jtaleric>
Component: openstack-tripleo-heat-templatesAssignee: Assaf Muller <amuller>
Status: CLOSED ERRATA QA Contact: Toni Freger <tfreger>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: amuller, chrisw, hmlnarik, jschluet, mburns, nyechiel, rhel-osp-director-maint, srevivo
Target Milestone: z1Keywords: ZStream
Target Release: 10.0 (Newton)   
Hardware: All   
OS: Linux   
Whiteboard: scale_lab
Fixed In Version: openstack-tripleo-heat-templates-5.2.0-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-01 14:46:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joe Talerico 2016-09-01 20:10:07 UTC
Description of problem:
Neutron API response times between Mitaka and Newton has significantly increased.

Looking at Mitaka: http://10.12.23.141/goto/102518200d0e04774e4fe6ffec47961e
Looking at Newton: http://10.12.23.141:80/goto/883c5b0b082fa4c863bf00ed6a338046

Looking at upstream Rally scenarios, we do see a change in response times as well (pointed out by Ihar) 
http://logs.openstack.org/58/363558/1/check/gate-rally-dsvm-neutron-neutron/e911424/console.html
http://logs.openstack.org/23/364123/2/check/gate-rally-dsvm-neutron-neutron/b83c8b3/console.html

However Brent Eagles points out that OOO incorrectly deploys Neutron with API workers set to 0, which would default to 1.

Version-Release number of selected component (if applicable):
openstack-neutron-9.0.0-0.20160820234947.14ebfb5.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install a with OOO
2. Run Browbeat with supplied config file : https://gist.github.com/jtaleric/eb8c090c2562c8842d8a8183b29ffc2f


Additional info:

Comment 2 Assaf Muller 2016-09-01 20:30:24 UTC
Brent has a TripleO patch to change the default of api_workers from 0 (No workers) to the empty string, so that TripleO won't set a value at all and use Neutron's default, which is the number of CPUs. Meanwhile Joe is re-running the tests with api_workers set to ''.

Comment 3 Hynek Mlnarik 2016-09-02 18:09:44 UTC
Following up on the references to upstream Rally - some of the upstream Rally scenarios differ significantly in Rally settings, eg. ports_per_network in create_and_list_ports is set to 2 in mitaka (PS 363558) but to 100 in master (PS 364123) so the results need detailed inspection w.r.t. their settings.

That said, IMHO they are still worth inspection. E.g. in the case of create_and_list_networks ([1] - master, [2] - mitaka), the settings is almost the same, the only rally differences seem the number of iterations (100 in master, 40 in mitaka), in network quota (100 in master, -1 in mitaka), and in sla (but that is rather irrelevant for the results). The results however show rather a significant difference in the times.

[1] http://logs.openstack.org/23/364123/2/check/gate-rally-dsvm-neutron-neutron/b83c8b3/rally-plot/results.html.gz#/NeutronNetworks.create_and_list_networks/task
[2] http://logs.openstack.org/58/363558/1/check/gate-rally-dsvm-neutron-neutron/e911424/rally-plot/results.html.gz#/NeutronNetworks.create_and_list_networks/task

Comment 6 Toni Freger 2017-01-24 09:40:06 UTC
Tested in openstack-tripleo-heat-templates-5.2.0-3.el7ost.noarch
The amount of workers configured according to CPU amount of the controller, in my setup 4.

/etc/neutron/neutron.conf:api_workers=4
/etc/neutron/neutron.conf:rpc_workers=4

Comment 9 errata-xmlrpc 2017-02-01 14:46:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0234.html