Bug 1721273 - Neutron API List Ports Performance regression
Summary: Neutron API List Ports Performance regression
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 15.0 (Stein)
Assignee: Rodolfo Alonso
QA Contact: Candido Campos
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-17 19:29 UTC by Sai Sindhur Malleni
Modified: 2019-09-26 10:52 UTC (History)
14 users (show)

Fixed In Version: openstack-neutron-14.0.3-0.20190708220434.1229be9.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1834484 0 None None None 2019-06-27 13:24:42 UTC
OpenStack gerrit 667981 0 None MERGED Add qos_network_policy_id to Port OVO 2020-09-24 10:03:23 UTC
OpenStack gerrit 667998 0 None MERGED Refactor qos_plugin._extend_port_resource_request 2020-09-24 10:03:22 UTC
OpenStack gerrit 669360 0 None MERGED Add qos_network_policy_id to Port OVO 2020-09-24 10:03:21 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:23:36 UTC

Description Sai Sindhur Malleni 2019-06-17 19:29:21 UTC 
Description of problem: Testing an OSP15 build with OVN as the ML2 backend and running rally through browbeat,

When running a scenario that creates 500 networks with 4 ports per networks and lists all ports, several list ports we are seeing several errors in rally on listing ports once there are more than 600-700 ports with client timeouts. Looking at neutron server logs, it looks like neutron is eventually listing ports but taking way more time than the client timeout. In some cases neutron is taking more than 500s to list ports. Also, all 2000 ports create without problems, only listing ports is the problem.


Version-Release number of selected component (if applicable):
15 with OVN as backend
How reproducible:
100%

Tested on a fresh deploy OSP 15 and after seeing this problem, redeployed overcloud and retested but still hitting the same issues



Steps to Reproduce:
1. Deploy OSP 15 with ML2/OVN as the backend
2.
3.

Actual results:

Rally has client expectations due to neutron taking a long time to list ports
Expected results:
Clientside timeouts because of neutron server taking too much time to list ports

Additional info:

Some logs showing high response times
c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 736044 time: 81.7121129
/var/log/containers/neutron/server.log:135144:2019-06-15 03:52:45.936 27 INFO neutron.wsgi [req-9de046af-6669-4450-b254-c6ac68336f62 2a6be1344b3548ef8a0f88218e504d5b 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 751134 time: 83.0072837
/var/log/containers/neutron/server.log:135262:2019-06-15 03:53:22.908 28 INFO neutron.wsgi [req-a8f45a33-3065-4bda-9716-3adbe82b4d93 2a6be1344b3548ef8a0f88218e504d5b 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 795705 time: 84.0019901
/var/log/containers/neutron/server.log:135300:2019-06-15 03:53:28.252 32 INFO neutron.wsgi [req-872947a8-76ca-4b7c-8800-6e5903dfc1ff 13d22cde19fa475e8df18d5fe7238a5a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 801882 time: 84.9408801
/var/log/containers/neutron/server.log:135319:2019-06-15 03:53:32.465 33 INFO neutron.wsgi [req-f31e2aa6-2541-48cd-a8e3-0077ebdef4e2 d8a60209680a4b3b8da492fbc7ea370a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 806004 time: 85.5383320
/var/log/containers/neutron/server.log:135468:2019-06-15 03:54:17.873 25 INFO neutron.wsgi [req-e55d2049-d0e2-4d35-af35-8e234b991101 8b02224a0a3d4b76bed7a4a797955d94 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 840288 time: 90.6322405
/var/log/containers/neutron/server.log:135476:2019-06-15 03:54:20.323 29 INFO neutron.wsgi [req-88c35620-a661-467f-ab8a-cc09c443165c 2a6be1344b3548ef8a0f88218e504d5b 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 843711 time: 90.6403577
/var/log/containers/neutron/server.log:135615:2019-06-15 03:55:04.518 24 INFO neutron.wsgi [req-3ba96413-7696-44bc-a28f-9097c3190235 2a6be1344b3548ef8a0f88218e504d5b 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 880056 time: 98.3140750
/var/log/containers/neutron/server.log:135644:2019-06-15 03:55:07.002 32 INFO neutron.wsgi [req-245e8556-93ee-415e-97e3-2d98c62629c3 13d22cde19fa475e8df18d5fe7238a5a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 886233 time: 97.5110650
/var/log/containers/neutron/server.log:135663:2019-06-15 03:55:11.058 33 INFO neutron.wsgi [req-b3b648e6-df07-4baa-a098-b99558811540 d8a60209680a4b3b8da492fbc7ea370a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 888975 time: 97.6181471
/var/log/containers/neutron/server.log:135745:2019-06-15 03:55:25.317 34 INFO neutron.wsgi [req-1dcadc0a-4be3-4eec-b282-c144571a1fa2 70a7ec3af3914562afab4ef3d801191a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 902697 time: 99.0537875
/var/log/containers/neutron/server.log:139681:2019-06-15 04:12:41.421 28 INFO neutron.wsgi [req-5c3ab4a7-7553-4f45-a495-8a4f12360cf6 d8a60209680a4b3b8da492fbc7ea370a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 525.0647051
/var/log/containers/neutron/server.log:139809:2019-06-15 04:13:38.716 28 INFO neutron.wsgi [req-ab88e99c-bd54-439c-a53d-9e475171e195 ea2d8c1f456c491c8a73a44ff977a8d4 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 563.4730241
/var/log/containers/neutron/server.log:139865:2019-06-15 04:13:57.203 28 INFO neutron.wsgi [req-56763959-e3d6-4cea-8f21-80e01f890e21 13d22cde19fa475e8df18d5fe7238a5a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 569.1214509
/var/log/containers/neutron/server.log:139959:2019-06-15 04:14:41.150 26 INFO neutron.wsgi [req-b42a9440-b72f-4784-8817-6c92822a46aa 2a6be1344b3548ef8a0f88218e504d5b 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 519.7558904
/var/log/containers/neutron/server.log:140011:2019-06-15 04:14:56.362 26 INFO neutron.wsgi [req-e55a1e3d-9286-420c-9984-2a18fa4fb83d 13d22cde19fa475e8df18d5fe7238a5a 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 528.1388409
/var/log/containers/neutron/server.log:140105:2019-06-15 04:15:40.392 23 INFO neutron.wsgi [req-a534977e-2a94-4071-aa76-f325baf8edd1 8b02224a0a3d4b76bed7a4a797955d94 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 581.0299084
/var/log/containers/neutron/server.log:140137:2019-06-15 04:15:52.379 23 INFO neutron.wsgi [req-7e6a390b-b243-4689-acb8-d1427560c766 8b02224a0a3d4b76bed7a4a797955d94 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 588.6950381
/var/log/containers/neutron/server.log:140181:2019-06-15 04:16:12.104 23 INFO neutron.wsgi [req-c68979d9-ed4e-49dc-bf85-25f5d3c4e75a 8b02224a0a3d4b76bed7a4a797955d94 6006531c87974244a34d6d938d605661 - default default] 172.16.0.187 "GET /v2.0/ports HTTP/1.1" status: 200  len: 0 time: 584.6853387
Comment 1 Sai Sindhur Malleni 2019-06-19 13:15:57 UTC 
Just as a datapoint, seeing similar results with ML2/OVS as well on OSP15. However, in OSP 13, ware able to list > 60K ports without issues.
Comment 3 Sai Sindhur Malleni 2019-06-25 16:35:23 UTC 
Hi Neutron team,

This seems like a very serious regression. Any update?
Comment 14 Nate Johnston 2019-07-05 14:11:54 UTC 
Upstream change in master [1] has merged; backported to stein [2].

[1] https://review.opendev.org/#/c/667981/
[2] https://review.opendev.org/#/c/669360/
Comment 21 errata-xmlrpc 2019-09-21 11:23:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811
Note You need to log in before you can comment on or make changes to this bug.