Bug 1551016 - Creating ports using OVN as a backend doesn't scale
Summary: Creating ports using OVN as a backend doesn't scale
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openvswitch
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Timothy Redaelli
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On: 1545344 1622469
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-02 14:53 UTC by Daniel Alvarez Sanchez
Modified: 2019-09-09 15:13 UTC (History)
15 users (show)

Fixed In Version: openvswitch-2.9.0-21.el7fdn
Doc Type: If docs needed, set a value
Doc Text:
The creation time of OVN ports grew linearly as ports were created. The creation time now remains constant, regardless of the number of ports in the cloud.
Clone Of: 1545344
Environment:
Last Closed: 2018-08-29 16:32:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github https://github.com/openvswitch/ovs/commit/eab138764e3aaa9f98a725a4636dd30e2a41a896 0 None None None 2018-03-02 14:55:42 UTC
Launchpad 1752897 0 None None None 2018-03-02 14:53:50 UTC
Red Hat Product Errata RHBA-2018:2573 0 None None None 2018-08-29 16:33:24 UTC

Comment 1 Daniel Alvarez Sanchez 2018-03-02 14:55:42 UTC
This is greatly improved by this patch [0]. Patch got merged in master and I asked for the backport upstream to branch 2.9 (ie OSP13). Link to the upstream discussion at [1].

[0] https://github.com/openvswitch/ovs/commit/eab138764e3aaa9f98a725a4636dd30e2a41a896
[1] https://mail.openvswitch.org/pipermail/ovs-discuss/2018-February/046149.html

Comment 18 Daniel Alvarez Sanchez 2018-08-09 13:26:13 UTC
If you want to measure the times for creating a port in OVN:

diff --git a/networking_ovn/ml2/mech_driver.py b/networking_ovn/ml2/mech_driver.py
index 2fb7d35..0ebbc0e 100644
--- a/networking_ovn/ml2/mech_driver.py
+++ b/networking_ovn/ml2/mech_driver.py
@@ -27,6 +27,7 @@ from neutron_lib.services.qos import constants as qos_consts
 from oslo_config import cfg
 from oslo_db import exception as os_db_exc
 from oslo_log import log
+from oslo_utils import timeutils
 
 from neutron.common import utils as n_utils
 from neutron.db import provisioning_blocks
@@ -471,7 +472,11 @@ class OVNMechanismDriver(api.MechanismDriver):
         result in the deletion of the resource.
         """
         port = context.current
+        _timer = timeutils.StopWatch()
+        _timer.restart()
         self._ovn_client.create_port(port)
+        _timer.stop()
+        LOG.debug('[*] create_port = %.2f seconds', _timer.elapsed())
         self._notify_dhcp_updated(port['id'])




And then parse the logs:



import sys
import re

PATTERN = re.compile(r'create_port\ =\ (\d+\.\d+)\ seconds')

with open(sys.argv[1]) as logf:
    for line in logf:
        match = PATTERN.findall(line)
        if match:
            print(match[0])

Comment 19 Joanne O'Flynn 2018-08-13 10:32:11 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 20 Timothy Redaelli 2018-08-13 10:49:24 UTC
I only backported the patch, so I think the original creator of the feature should add the doctext

Comment 21 Eran Kuris 2018-08-16 06:07:07 UTC
fix verified on OSP13 2018-08-08.2

openvswitch-2.9.0-54.el7fdp.x86_64

The average time of creation port 0.425

added to the neutron server code:
diff --git a/networking_ovn/ml2/mech_driver.py b/networking_ovn/ml2/mech_driver.py
index 2fb7d35..0ebbc0e 100644
--- a/networking_ovn/ml2/mech_driver.py
+++ b/networking_ovn/ml2/mech_driver.py
@@ -27,6 +27,7 @@ from neutron_lib.services.qos import constants as qos_consts
 from oslo_config import cfg
 from oslo_db import exception as os_db_exc
 from oslo_log import log
+from oslo_utils import timeutils
 
 from neutron.common import utils as n_utils
 from neutron.db import provisioning_blocks
@@ -471,7 +472,11 @@ class OVNMechanismDriver(api.MechanismDriver):
         result in the deletion of the resource.
         """
         port = context.current
+        _timer = timeutils.StopWatch()
+        _timer.restart()
         self._ovn_client.create_port(port)
+        _timer.stop()
+        LOG.debug('[*] create_port = %.2f seconds', _timer.elapsed())
         self._notify_dhcp_updated(port['id'])

collected the logs from all controller node and ran the script:
import sys
import re

PATTERN = re.compile(r'create_port\ =\ (\d+\.\d+)\ seconds')

with open(sys.argv[1]) as logf:
    for line in logf:
        match = PATTERN.findall(line)
        if match:
            print(match[0])

Comment 23 errata-xmlrpc 2018-08-29 16:32:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2573


Note You need to log in before you can comment on or make changes to this bug.