Bug 1372470 - Neutron API Performance regression
Summary: Neutron API Performance regression
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: All
OS: Linux
high
high
Target Milestone: z1
: 10.0 (Newton)
Assignee: Assaf Muller
QA Contact: Toni Freger
URL:
Whiteboard: scale_lab
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-01 20:10 UTC by Joe Talerico
Modified: 2017-02-01 14:46 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.2.0-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-01 14:46:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 364483 0 None None None 2016-09-01 20:15:48 UTC
Red Hat Product Errata RHBA-2017:0234 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 director Bug Fix Advisory 2017-02-01 19:44:43 UTC

Description Joe Talerico 2016-09-01 20:10:07 UTC
Description of problem:
Neutron API response times between Mitaka and Newton has significantly increased.

Looking at Mitaka: http://10.12.23.141/goto/102518200d0e04774e4fe6ffec47961e
Looking at Newton: http://10.12.23.141:80/goto/883c5b0b082fa4c863bf00ed6a338046

Looking at upstream Rally scenarios, we do see a change in response times as well (pointed out by Ihar) 
http://logs.openstack.org/58/363558/1/check/gate-rally-dsvm-neutron-neutron/e911424/console.html
http://logs.openstack.org/23/364123/2/check/gate-rally-dsvm-neutron-neutron/b83c8b3/console.html

However Brent Eagles points out that OOO incorrectly deploys Neutron with API workers set to 0, which would default to 1.

Version-Release number of selected component (if applicable):
openstack-neutron-9.0.0-0.20160820234947.14ebfb5.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install a with OOO
2. Run Browbeat with supplied config file : https://gist.github.com/jtaleric/eb8c090c2562c8842d8a8183b29ffc2f


Additional info:

Comment 2 Assaf Muller 2016-09-01 20:30:24 UTC
Brent has a TripleO patch to change the default of api_workers from 0 (No workers) to the empty string, so that TripleO won't set a value at all and use Neutron's default, which is the number of CPUs. Meanwhile Joe is re-running the tests with api_workers set to ''.

Comment 3 Hynek Mlnarik 2016-09-02 18:09:44 UTC
Following up on the references to upstream Rally - some of the upstream Rally scenarios differ significantly in Rally settings, eg. ports_per_network in create_and_list_ports is set to 2 in mitaka (PS 363558) but to 100 in master (PS 364123) so the results need detailed inspection w.r.t. their settings.

That said, IMHO they are still worth inspection. E.g. in the case of create_and_list_networks ([1] - master, [2] - mitaka), the settings is almost the same, the only rally differences seem the number of iterations (100 in master, 40 in mitaka), in network quota (100 in master, -1 in mitaka), and in sla (but that is rather irrelevant for the results). The results however show rather a significant difference in the times.

[1] http://logs.openstack.org/23/364123/2/check/gate-rally-dsvm-neutron-neutron/b83c8b3/rally-plot/results.html.gz#/NeutronNetworks.create_and_list_networks/task
[2] http://logs.openstack.org/58/363558/1/check/gate-rally-dsvm-neutron-neutron/e911424/rally-plot/results.html.gz#/NeutronNetworks.create_and_list_networks/task

Comment 6 Toni Freger 2017-01-24 09:40:06 UTC
Tested in openstack-tripleo-heat-templates-5.2.0-3.el7ost.noarch
The amount of workers configured according to CPU amount of the controller, in my setup 4.

/etc/neutron/neutron.conf:api_workers=4
/etc/neutron/neutron.conf:rpc_workers=4

Comment 9 errata-xmlrpc 2017-02-01 14:46:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0234.html


Note You need to log in before you can comment on or make changes to this bug.