Bug 1737012 - Neutron API List Ports Performance regression
Summary: Neutron API List Ports Performance regression
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 15.0 (Stein)
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Slawek Kaplonski
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-02 09:53 UTC by Poul Udovenko
Modified: 2019-08-12 08:29 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-12 08:07:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Poul Udovenko 2019-08-02 09:53:20 UTC
Description of problem:

Have 70 ports on 2 net:
If get list port - API get info sow long ~ 10 sec

Have 1800 port on 70 net:
If get list port - API get info sow long > 200 sec


Version-Release number of selected component (if applicable):
python2-neutron-lib-1.25.0-1.el7.noarch
openstack-neutron-ml2-14.0.2-1.el7.noarch
openstack-neutron-linuxbridge-14.0.2-1.el7.noarch
openstack-neutron-common-14.0.2-1.el7.noarch
openstack-neutron-14.0.2-1.el7.noarch
python2-neutronclient-6.12.0-1.el7.noarch
python2-neutron-14.0.2-1.el7.noarch

uname -r
3.10.0-957.12.2.el7.x86_64

How reproducible:
New install stein release and create many port

Steps to Reproduce:
1. Deploy Steins release an create many ports 
2.
3.

Actual results:
Rally has client expectations due to neutron taking a long time to list ports
Expected results:
Clientside timeouts because of neutron server taking too much time to list ports

Expected results:


Additional info:

dev install with:

# time openstack network list 
-
real	0m2.463s
user	0m1.737s
sys	0m0.262s

openstack port list | wc -l
78

2019-08-02 11:39:34.242 13112 INFO neutron.wsgi [req-1eed7391-d230-4777-85bf-0ad885fa29cc ***** 5adea387957546a79b4c4ddcac95ca33 - default default] 10.1.88.150 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 10.7024050
2019-08-02 11:40:23.730 13073 INFO neutron.wsgi [req-09b6adbe-ee4a-4629-beb2-11c2ba57e757 ***** 5adea387957546a79b4c4ddcac95ca33 - default default] 10.1.88.150 "GET /v2.0/ports HTTP/1.1" status: 200  len: 66597 time: 10.1765039
2019-08-02 11:40:40.989 13098 INFO neutron.wsgi [req-6d53a840-0413-424b-8fe4-759cc4a50c74 ***** 5adea387957546a79b4c4ddcac95ca33 - default default] 10.1.88.150 "GET /v2.0/ports HTTP/1.1" status: 200  len: 66597 time: 9.6410010


On another install:
time openstack network list | wc -l
70

real	0m3.323s
user	0m1.913s
sys	0m0.365s

time openstack port list | wc -l
1804

real	4m28.873s
user	0m6.788s
sys	0m0.398s

2019-07-30 19:15:31.354 14047 INFO neutron.wsgi [req-488e863d-ada4-4b30-8d1e-ad6df27e587d **** cb4e62f368824ec4a3ad250eeeff9b00 - default default] 10.1.84.150 "GET /v2.0/ports HTTP/1.1" status: 200  len: 1447729 time: 218.5887561
2019-07-30 19:16:37.485 14052 INFO neutron.wsgi [req-980f1592-4b75-4ca7-833d-b045a3ff41b9 **** 2e3642a48ea843f7b1853e778ac5bd76 - default default] 10.1.84.150 "GET /v2.0/ports HTTP/1.1" status: 200  len: 1447729 time: 217.6485591


I see Bug 1721273
Apply patch but API response time very slow

Comment 1 Poul Udovenko 2019-08-02 13:57:27 UTC
I use:
service_plugins = router, neutron.services.qos.qos_plugin.QoSPlugin

If use:
service_plugins = router

[root@ctrl1 paveludovenko]# time neutron port-list | wc -l
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
78

real	0m2.335s
user	0m1.510s
sys	0m0.248s

Comment 4 Slawek Kaplonski 2019-08-12 08:07:22 UTC
Today I tested it on neutron packages installed from https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=947015 - it's version openstack-neutron-14.0.3-0.20190809100451.2e15b18 and I had results like below:


(overcloud) [stack@undercloud-0 ~]$ time neutron port-list | wc -l
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
222

real    0m4.341s
user    0m0.993s
sys     0m0.094s
(overcloud) [stack@undercloud-0 ~]$ time neutron port-list | wc -l
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
222

real    0m4.337s
user    0m1.075s
sys     0m0.130s
(overcloud) [stack@undercloud-0 ~]$ time neutron port-list | wc -l
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
222

real    0m2.973s
user    0m1.020s
sys     0m0.099s

So IMO this problem is now fixed and I will close this BZ as CURRENTRELEASE
If You have other way to reproduce this issue on current OSP-15 released package, feel free to reopen this BZ or create new one.

Comment 5 Poul Udovenko 2019-08-12 08:26:20 UTC
(In reply to Slawek Kaplonski from comment #4)
> Today I tested it on neutron packages installed from
> https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=947015 - it's
> version openstack-neutron-14.0.3-0.20190809100451.2e15b18 and I had results
> like below:
> 
...
> So IMO this problem is now fixed and I will close this BZ as CURRENTRELEASE
> If You have other way to reproduce this issue on current OSP-15 released
> package, feel free to reopen this BZ or create new one.


Yu use neutron.services.qos.qos_plugin.QoSPlugin in service plugin?

Comment 6 Slawek Kaplonski 2019-08-12 08:29:40 UTC
@Poul: Yes, I had QoS service plugin enabled in my config.


Note You need to log in before you can comment on or make changes to this bug.