Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1622469

Summary: Creating ports using OVN as a backend doesn't scale
Product: Red Hat OpenStack Reporter: Daniel Alvarez Sanchez <dalvarez>
Component: openvswitchAssignee: Mark Michelson <mmichels>
Status: CLOSED CURRENTRELEASE QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: amuller, apevec, atragler, chrisw, dalvarez, dcbw, ekuris, fhallal, fleitner, ihrachys, jlibosva, lhh, majopela, mmichels, njohnston, oblaut, rhos-maint, srevivo
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1545344 Environment:
Last Closed: 2020-06-23 13:29:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1545344    
Bug Blocks: 1551016    

Comment 5 Daniel Alvarez Sanchez 2018-10-19 09:51:38 UTC 
I think that there was something weird with Joe's tests or with my port group backport into OVS 2.9.

I sent a patch [0] to test the rally results in the gate without port groups. Comparing this to any other patch that ran today, here are the results:


WITHOUT PORT GROUPS:

--------------------------------------------------------------------------------
Task 1ebd3d3e-f195-4064-9b37-e813eda206c9 has 0 error(s)
--------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------------------------------+
|                                                     Response Times (sec)                                                     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                    | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_network    | 1.208     | 3.995        | 5.437        | 5.62         | 5.802     | 3.791     | 100.0%  | 8     |
| neutron.create_port (x50) | 145.483   | 154.186      | 172.954      | 173.903      | 174.852   | 157.766   | 100.0%  | 8     |
| neutron.list_ports        | 0.704     | 1.64         | 2.284        | 2.523        | 2.762     | 1.62      | 100.0%  | 8     |
| total                     | 149.249   | 161.61       | 179.235      | 180.037      | 180.839   | 163.181   | 100.0%  | 8     |
|  -> duration              | 149.249   | 161.61       | 179.235      | 180.037      | 180.839   | 163.181   | 100.0%  | 8     |
|  -> idle_duration         | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 8     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

+---------------------------------------------------------------------------------------------------------+
|                                          Atomics per iteration                                          |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| iteration | duration | 1. neutron.create_network | 2. neutron.create_port (x50) | 3. neutron.list_ports |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| 1         | 180.839  | 5.281                     | 0.0                          | 0.704                 |
| 2         | 173.028  | 3.455                     | 0.0                          | 1.681                 |
| 3         | 178.547  | 4.805                     | 0.0                          | 1.598                 |
| 4         | 165.178  | 5.802                     | 0.0                          | 1.733                 |
| 5         | 158.043  | 4.536                     | 0.0                          | 2.762                 |
| 6         | 149.249  | 1.208                     | 0.0                          | 2.079                 |
| 7         | 150.866  | 2.129                     | 0.0                          | 1.303                 |
| 8         | 149.696  | 3.109                     | 0.0                          | 1.101                 |
+-----------+----------+---------------------------+------------------------------+-----------------------+

Load duration: 330.544
Full duration: 498.878
--------------------------------------------------------------------------------



WITH PORT GROUPS:

+------------------------------------------------------------------------------------------------------------------------------+
|                                                     Response Times (sec)                                                     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                    | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_network    | 2.737     | 4.506        | 6.051        | 6.061        | 6.072     | 4.529     | 100.0%  | 8     |
| neutron.create_port (x50) | 81.721    | 91.642       | 96.327       | 96.636       | 96.945    | 90.449    | 100.0%  | 8     |
| neutron.list_ports        | 0.846     | 1.297        | 1.759        | 1.852        | 1.945     | 1.298     | 100.0%  | 8     |
| total                     | 88.642    | 96.484       | 102.252      | 102.45       | 102.648   | 96.279    | 100.0%  | 8     |
|  -> duration              | 88.642    | 96.484       | 102.252      | 102.45       | 102.648   | 96.279    | 100.0%  | 8     |
|  -> idle_duration         | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 8     |
+---------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

+---------------------------------------------------------------------------------------------------------+
|                                          Atomics per iteration                                          |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| iteration | duration | 1. neutron.create_network | 2. neutron.create_port (x50) | 3. neutron.list_ports |
+-----------+----------+---------------------------+------------------------------+-----------------------+
| 1         | 88.642   | 6.072                     | 0.0                          | 0.846                 |
| 2         | 98.318   | 2.737                     | 0.0                          | 1.567                 |
| 3         | 100.508  | 5.868                     | 0.0                          | 0.849                 |
| 4         | 94.65    | 6.041                     | 0.0                          | 0.905                 |
| 5         | 102.648  | 4.021                     | 0.0                          | 1.68                  |
| 6         | 102.082  | 4.99                      | 0.0                          | 1.027                 |
| 7         | 89.14    | 3.327                     | 0.0                          | 1.945                 |
| 8         | 94.244   | 3.179                     | 0.0                          | 1.569                 |
+-----------+----------+---------------------------+------------------------------+-----------------------+

Load duration: 196.751
Full duration: 367.979
--------------------------------------------------------------------------------




As you can see the 95%ile of creating x50 ports is 96.636 vs 173.903. Almost 100% of speedup using port groups.

This difference is much more noticeable as the number of port grows. Also, CPU utilization will drop a lot with the Port Groups.
 

[0] https://review.openstack.org/#/c/611793/
Comment 6 Shelley Dunne 2018-10-26 15:08:18 UTC 
Not in Advisory removing Target Milestone z3 planned to ship 14-Nov