Red Hat Bugzilla – Bug 1290562
[perf] neutron list_ports grows in timing as number of ports increases
Last modified: 2016-04-26 23:41:35 EDT
Created attachment 1104489 [details] Rally html output displaying the timings for each action performed against neutron Description of problem: Testing neutron through browbeat (https://github.com/jtaleric/browbeat) create and list scenarios for networks, ports, routers, and subnets. List-ports test shows a drastic increase in response timing of neutron.list_ports. The benchmark ran 500 iterations of: create network Create port x 4 list ports Timing values for each action are include in the rally html output attached to this bz Testing was conducted on an OSPd deployed OSP 8 cloud with a single controller and a single compute node. Pacemaker was deployed dispite the single controller configuration to accurate measure response timings with ha enabled. Version-Release number of selected component (if applicable): OSP 8 openstack-neutron-common-7.0.0-5.el7ost.noarch python-neutron-lbaas-7.0.0-2.el7ost.noarch python-neutronclient-3.1.0-1.el7ost.noarch openstack-neutron-lbaas-7.0.0-2.el7ost.noarch openstack-neutron-openvswitch-7.0.0-5.el7ost.noarch openstack-neutron-metering-agent-7.0.0-5.el7ost.noarch python-neutron-7.0.0-5.el7ost.noarch openstack-neutron-7.0.0-5.el7ost.noarch openstack-neutron-ml2-7.0.0-5.el7ost.noarch How reproducible: Every time Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: See details tab to view the drastic increase in response timing for listing ports as the number of ports created increases.
I tried reproducing on devstack master. Following the Rally scenario, I created 500 networks and 4 ports in each network. No subnets or any other resources, everything is in the same tenant. I issued 'time neutron port-list': real 0m1.325s user 0m0.572s sys 0m0.038s No dice... Creating a devstack setup based off Liberty now.
There's some u/s traction, adding Launchpad bugs. Fixes will be backported to OSP 8. The summary is that newly added features (RBAC and AZs) added DB look ups on a per-resource basic when listing that resource type. So for example when listing networks, RBAC added a per-network DB look up, so the time to list networks increases linearly.
Backported the following issues to osp-8: https://bugs.launchpad.net/bugs/1525423 (https://bugzilla.redhat.com/show_bug.cgi?id=1292167) https://bugs.launchpad.net/bugs/1525295 (https://bugzilla.redhat.com/show_bug.cgi?id=1292179) Removing issue https://bugs.launchpad.net/bugs/1525740 back-port is irrelevant to osp-8, as its patch fixes network availability zones db, which didn't make it into osp-8. Leaving https://bugs.launchpad.net/bugs/1513782 to track this issue.
Package openstack-neutron-7.0.1-4 has two patches merged (Links in the dependent RHBZs). Joe Talerico installed it in an OSP 8 system and re-ran Rally tests, with these results: http://10.16.31.132/rook-neutron-test-neutron-24/neutron/neutron-create-list-port-cc/run-1/rook-neutron-test-neutron-24-iteration_1-neutron-create-list-port-cc-0008.html#/NeutronNetworks.create_and_list_ports/details It looks like it still takes 50 seconds to list 500 ports.
Added an upstream patch by Ihar that will resolve the issue for most cases. We'll push it upstream and backport it to OSP 8 as fast as we can.
Joe - Package is available for testing. Did you want to re-run Rally?
Assaf - We are getting a OSP8 env up for this.
You can find my result here - http://file.tlv.redhat.com/~tfreger/rally_logs/output_1.html Have been running this task on Controller+Compute setup, that installed via OSPd openstack-neutron-7.0.1-10.el7ost.noarch From my results it seems to be improved as well.
Assaf, from your opinion what result we should get in order to verify this bug?
Looking at your attached results vs. the attached results in comment 0, the linear coefficient has clearly reduced dramatically. We can always improve, but I personally consider this bug closed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0603.html