RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1820009 - Improve the performance of nmstate when creating 300+ bridge with 1 VLAN each.
Summary: Improve the performance of nmstate when creating 300+ bridge with 1 VLAN each.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: nmstate
Version: 8.2
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 8.3
Assignee: Gris Ge
QA Contact: Mingyu Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-02 03:58 UTC by Gris Ge
Modified: 2023-01-09 14:02 UTC (History)
6 users (show)

Fixed In Version: nmstate-0.3.2-6.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 03:08:54 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)
terminal log (2.74 KB, text/plain)
2020-04-02 07:32 UTC, Dominik Holler 		no flags Details
verified.log (640.71 KB, text/plain)
2020-07-21 08:32 UTC, Mingyu Shi 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github nmstate nmstate pull 1016 0 None closed [Plugin design] nm: Replace MainLoop with MainContext 2021-01-14 12:15:04 UTC
Red Hat Product Errata RHBA-2020:4696 0 None None None 2020-11-04 03:09:08 UTC

Description Gris Ge 2020-04-02 03:58:16 UTC 
Description of problem:


Version-Release number of selected component (if applicable):
nmstate-0.2.6-9.el8

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Dominik Holler 2020-04-02 07:32:20 UTC 
Created attachment 1675625 [details]
terminal log

Please note that most of the computing time is currently consumed by firewalld, ovs-vswtichd and NetworkManager.


On RHEL 8.2.0, adding
the first 100 vlans and bridges takes around 30 seconds,
the next 100 ones takes around 50 seconds,
the next 100 ones takes around 80 seconds,
the next 100 ones takes around 120 seconds, and
the next 100 ones takes around 170 seconds.

Would be helpful, if nmstate could provide responsetimes < 150 seconds to modify up to 1000 vlan-bridge combinations.
Comment 2 Gris Ge 2020-04-08 09:12:37 UTC 
Hi Dominik,

I have posted a patch to upstream which improved the performance:

 * Create 500 VLANs + 500 bridges over each in a single transaction, nmstate takes 346 seconds.
 * Create 300 VLANs + 300 bridges over each in a single transaction, nmstate takes 114 seconds.
 * Create 100 VLANs + 100 bridges over each in a single transaction, nmstate takes 15 seconds.

OS details: RHEL 8.2 + NetworkManager master branch(1.23.2) + firewalld and ovs daemons running.

You may try the rpm via command `dnf copr enable packit/nmstate-nmstate-1016`.

Base on my test and repeat trying, nmstate can only support up to 1000 interfaces in a single desire state unless NetworkManager
make big performance improvements.
Comment 3 Dominik Holler 2020-04-08 15:42:40 UTC 
Same VM as in comment #1, with new software as described in comment #2 [1]:
the first 100 vlans and bridges takes 0m13.970s
the next 100 ones takes around 0m30.095s
the next 100 ones takes around 0m39.632s
the next 100 ones takes around 0m52.081s
the next 100 ones takes around 1m9.734s
the next 100 ones takes around 1m20.677s
the next 100 ones takes around 1m33.498s
the next 100 ones takes around 1m41.105s
the next 100 ones fails.

The means up to 800 networks works in an amazing time, great work Gris!
Let's check if we can agree that this is what we want to keep in the long run.


[1]
  nmstate-0.3.0-0.20200408115626576019.el8.noarch
  NetworkManager-1.23.2-25381.9a74d2e023.el8.x86_64
Comment 4 Gris Ge 2020-04-09 03:29:50 UTC 
> 
> The means up to 800 networks works in an amazing time, great work Gris!
> Let's check if we can agree that this is what we want to keep in the long
> run.
> 

The 500 networks(500 VLANs and 500 bridge over each) seems a reasonable amount.
If RHV agree on that, I can make that tested in our CI and officially documented.
Comment 5 Dominik Holler 2020-04-21 15:14:24 UTC 
(In reply to Gris Ge from comment #4)
> > 
> > The means up to 800 networks works in an amazing time, great work Gris!
> > Let's check if we can agree that this is what we want to keep in the long
> > run.
> > 
> 
> The 500 networks(500 VLANs and 500 bridge over each) seems a reasonable
> amount.
> If RHV agree on that, I can make that tested in our CI and officially
> documented.

Thanks, RHV agrees on 300 networks (300 VLANs and 300 bridge over each).
Comment 8 Mingyu Shi 2020-07-21 08:23:35 UTC 
Verified with versions:
nmstate-0.3.3-2.el8.noarch
NetworkManager-1.26.0-2.el8.x86_64
DISTRO=RHEL-8.3.0-20200721.n.0
Linux dell-r330-15.gsslab.brq.redhat.com 4.18.0-226.el8.x86_64 #1 SMP Wed Jul 15 07:40:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

To create with 300 ones, it took
real	0m52.557s
user	0m11.554s
sys	0m1.124s

On nmstate-0.2.6-6.el8.noarch, it took:
real	2m1.540s
user	0m21.550s
sys	0m2.760s
Comment 9 Mingyu Shi 2020-07-21 08:32:56 UTC 
Created attachment 1701856 [details]
verified.log
Comment 12 errata-xmlrpc 2020-11-04 03:08:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4696
Note You need to log in before you can comment on or make changes to this bug.