1622469 – Creating ports using OVN as a backend doesn't scale

Bug 1622469 - Creating ports using OVN as a backend doesn't scale

Summary: Creating ports using OVN as a backend doesn't scale

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openvswitch
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Mark Michelson
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:	1545344
Blocks:	1551016
TreeView+	depends on / blocked

Reported:	2018-08-27 09:44 UTC by Daniel Alvarez Sanchez
Modified:	2021-02-01 14:48 UTC (History)
CC List:	18 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1545344
Environment:
Last Closed:	2020-06-23 13:29:42 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1752897	None	None	None	2018-08-27 09:44:03 UTC
OpenStack gerrit	575388	None	MERGED	Support Port Groups in networking-ovn	2021-02-01 14:48:46 UTC
OpenStack gerrit	594137	None	ABANDONED	Maintenance: Add functional tests for database sync task	2021-02-01 14:48:46 UTC

Comment 5 Daniel Alvarez Sanchez 2018-10-19 09:51:38 UTC

I think that there was something weird with Joe's tests or with my port group backport into OVS 2.9.

I sent a patch [0] to test the rally results in the gate without port groups. Comparing this to any other patch that ran today, here are the results:


WITHOUT PORT GROUPS:

--------------------------------------------------------------------------------
Task 1ebd3d3e-f195-4064-9b37-e813eda206c9 has 0 error(s)
--------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------------------------------+
|                                                     Response Times (sec)                                                     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                    | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_network    | 1.208     | 3.995        | 5.437        | 5.62         | 5.802     | 3.791     | 100.0%  | 8     |
| neutron.create_port (x50) | 145.483   | 154.186      | 172.954      | 173.903      | 174.852   | 157.766   | 100.0%  | 8     |
| neutron.list_ports        | 0.704     | 1.64         | 2.284        | 2.523        | 2.762     | 1.62      | 100.0%  | 8     |
| total                     | 149.249   | 161.61       | 179.235      | 180.037      | 180.839   | 163.181   | 100.0%  | 8     |
|  -> duration              | 149.249   | 161.61       | 179.235      | 180.037      | 180.839   | 163.181   | 100.0%  | 8     |
|  -> idle_duration         | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 8     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

+---------------------------------------------------------------------------------------------------------+
|                                          Atomics per iteration                                          |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| iteration | duration | 1. neutron.create_network | 2. neutron.create_port (x50) | 3. neutron.list_ports |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| 1         | 180.839  | 5.281                     | 0.0                          | 0.704                 |
| 2         | 173.028  | 3.455                     | 0.0                          | 1.681                 |
| 3         | 178.547  | 4.805                     | 0.0                          | 1.598                 |
| 4         | 165.178  | 5.802                     | 0.0                          | 1.733                 |
| 5         | 158.043  | 4.536                     | 0.0                          | 2.762                 |
| 6         | 149.249  | 1.208                     | 0.0                          | 2.079                 |
| 7         | 150.866  | 2.129                     | 0.0                          | 1.303                 |
| 8         | 149.696  | 3.109                     | 0.0                          | 1.101                 |
+-----------+----------+---------------------------+------------------------------+-----------------------+

Load duration: 330.544
Full duration: 498.878
--------------------------------------------------------------------------------



WITH PORT GROUPS:

+------------------------------------------------------------------------------------------------------------------------------+
|                                                     Response Times (sec)                                                     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                    | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_network    | 2.737     | 4.506        | 6.051        | 6.061        | 6.072     | 4.529     | 100.0%  | 8     |
| neutron.create_port (x50) | 81.721    | 91.642       | 96.327       | 96.636       | 96.945    | 90.449    | 100.0%  | 8     |
| neutron.list_ports        | 0.846     | 1.297        | 1.759        | 1.852        | 1.945     | 1.298     | 100.0%  | 8     |
| total                     | 88.642    | 96.484       | 102.252      | 102.45       | 102.648   | 96.279    | 100.0%  | 8     |
|  -> duration              | 88.642    | 96.484       | 102.252      | 102.45       | 102.648   | 96.279    | 100.0%  | 8     |
|  -> idle_duration         | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 8     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

+---------------------------------------------------------------------------------------------------------+
|                                          Atomics per iteration                                          |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| iteration | duration | 1. neutron.create_network | 2. neutron.create_port (x50) | 3. neutron.list_ports |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| 1         | 88.642   | 6.072                     | 0.0                          | 0.846                 |
| 2         | 98.318   | 2.737                     | 0.0                          | 1.567                 |
| 3         | 100.508  | 5.868                     | 0.0                          | 0.849                 |
| 4         | 94.65    | 6.041                     | 0.0                          | 0.905                 |
| 5         | 102.648  | 4.021                     | 0.0                          | 1.68                  |
| 6         | 102.082  | 4.99                      | 0.0                          | 1.027                 |
| 7         | 89.14    | 3.327                     | 0.0                          | 1.945                 |
| 8         | 94.244   | 3.179                     | 0.0                          | 1.569                 |
+-----------+----------+---------------------------+------------------------------+-----------------------+

Load duration: 196.751
Full duration: 367.979
--------------------------------------------------------------------------------




As you can see the 95%ile of creating x50 ports is 96.636 vs 173.903. Almost 100% of speedup using port groups.

This difference is much more noticeable as the number of port grows. Also, CPU utilization will drop a lot with the Port Groups.
 

[0] https://review.openstack.org/#/c/611793/

Comment 6 Shelley Dunne 2018-10-26 15:08:18 UTC

Not in Advisory removing Target Milestone z3 planned to ship 14-Nov

Note You need to log in before you can comment on or make changes to this bug.