Created attachment 1038540 [details]
vlan interface creation script
Description of problem:
ifup becomes slow when large numbers of VLANs are created
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. create scripts for 254 VLANs
2. loop over the interface names doing ifup
In a VM on a laptop, the last "ifup" takes multiple seconds to complete.
The sequence as a whole appears to show quadratic behaviour
Strace of the "ifup" shows 3 slow "nmcli" operations and one slow "grep" -
each on order of 3 seconds. The first "nmcli" is a simple status inquiry:
"nmcli -t --fields running general status"... and it does over 20,000 write
syscalls (and equivalently large numbers of other syscalls). This is repeatable
manually (with the VLANs inplace). On a fresh boot without the VLANs, only 168 write syscalls are done.
Created attachment 1038552 [details]
vlan interface online script
I can reproduce the slowness by creating many devices (using the script from Description) and then do any operation with nmcli.
$ nmcli dev | wc -l
$ time nmcli general
STATE CONNECTIVITY WIFI-HW WIFI WWAN-HW WWAN
connected full enabled enabled enabled enabled
The problem is not syscalls per se, but rather in intensive usage of glib in libnm library. The nm_client_new() function itself takes about 50% of the instructions.
I profiled 'nmcli general' command with valgrind's callgrind. The log is attached in the next comment. Data can be displayed with:
$ callgrind_annotate callgrind.out.23830
$ kcachegrind callgrind.out.23830
which is very nice GUI tool to show the data in nice views with callgraphs, maps, etc.
Unfortunately, I don't see any simple culprit or a low hanging fruit there. It is just obvious that the most intensive functions are memory management and various glib functions, because libnm calls them too many times (1 - 3 million), which seems wrong.
Created attachment 1039130 [details]
Callgrind output for 'nmcli general' for NM 1.0.2
Callgrind output generated by:
valgrind --tool=callgrind nmcli general
on Fedora 22 with NetworkManager-1.0.2-1.fc22.x86_64
Dislpay data with:
Looks like dbus operations done by g_initable_init could be worth a look
(indeed, is that table needed for a plain status equiry? Not that other nmcli uses shouldn't be faster too, but...).
as libnm currently is, it fetches ~everything~ on initialization. There might be some places to optimize the fetching. But in the end loading everything will take some time on larger systems.
We should investigate fetch on-demand for libnm.
Lubomir suggested that porting libnm to use the GDBus ObjectManager interfaces to talk to NetworkManager (which NM 1.2 already implements service-side) is a possible fix here. We want to do that anyway to work around issues with D-Bus Policy on pending reply maximums.
This is already in upstream master, will make it into 7.4
All currently planned work to improve performance is already in upstream master, and hence part of upcoming rhel-7.4.
According to our tests, it significantly improves performance of nmcli/libnm.
I am marking this bug as fixed, although in the future we should find ways to improve performance further.
nmcli shouldn't be much affected by a lot of devices/connections now
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.