Bug 1271341 - virsh iface-dumpxml is extremely slow on a host with a large number of interfaces
virsh iface-dumpxml is extremely slow on a host with a large number of interf...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: netcf (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Laine Stump
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-13 13:06 EDT by Laine Stump
Modified: 2015-12-30 15:49 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1269613
Environment:
Last Closed: 2015-12-30 15:49:33 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Laine Stump 2015-10-13 13:06:39 EDT
+++ This bug was initially created as a clone of Bug #1269613 +++

On a test system with 514 bridges each having an attach vlan interface, "virsh iface-dumpxml" of all toplevel interfaces takes 13min46sec. This is a bad thing because that's essentially what virt-manager does every time it is started.

Investigation and upstream discussion on the netcf-devel and augeas-devel mailing lists:

  https://www.redhat.com/archives/augeas-devel/2015-September/msg00001.html
  https://www.redhat.com/archives/augeas-devel/2015-October/msg00000.html

led to the following:

1) Several commits have recently been pushed to upstream augeas (post 1.4.0) which end up having a dramatic effect on the execution time of "virsh iface-dumpxml" for all interfaces on a system with 514 bridges + 514 vlans. Here are the augeas upstream commit IDs:

  commit a659f09afa1d6383d17a724da9c2ddbb77f844a7
  Author: David Lutterkort <lutter@watzmann.net>
  Date:   Mon Sep 28 14:37:03 2015 -0700

    Speed up the duplicate check when constructing nodesets
    
  commit 41e989ca1f9abf7a13e0f333e90857f2ccafbf7d
  Author: David Lutterkort <lutter@watzmann.net>
  Date:   Mon Sep 28 14:51:42 2015 -0700

    * src/pathx.c (ns_filter): speed up by removing nodes in batches
    
  commit 23d5e480cab312ed3f47f9f6e0777b152d6df9df
  Author: David Lutterkort <lutter@watzmann.net>
  Date:   Mon Sep 28 15:40:31 2015 -0700

    * src/pathx.c (ns_from_locpath): special-case 'name[42]' path expressions

These reduce the time from 13m46s to 7m34s! (The netcf patch given in Bug 1268382 further reduces this time to 6m40s).

2) There is an aug_match() query in netcf that takes a *very* long time to complete (nearly 1 second for each query!). It is in drv_redhat.c:

    ndevs = aug_fmt_match(ncf, &devs,
              "%s[ DEVICE = '%s' or BRIDGE = '%s' or MASTER = '%s'"
              "    or MASTER = ../*[BRIDGE = '%s']/DEVICE" ]/DEVICE",
              ifcfg_path, nif->name, nif->name, nif->name, nif->name);

So the query expression is, e.g.:

     "/files/etc/sysconfig/network-scripts/*[ "
     "DEVICE = 'br1' or BRIDGE = 'br1' or MASTER = 'br1' or MASTER = "
     "../*[BRIDGE = 'br1']/DEVICE ]/DEVICE"

The time-consuming part is the final term:

     MASTER = ../*[BRIDGE='br1']DEVICE

This searches for all ifcfg files that have a MASTER that is equal to the DEVICE name of any other ifcfg file that has BRIDGE='br1'. David Lutterkort has offered some advice on how to rewrite that query to make it much more efficient:

  https://www.redhat.com/archives/augeas-devel/2015-October/msg00003.html

  "(/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = '%s']"
  "|/files/etc/sysconfig/network-scripts/*[MASTER]"
  "[MASTER = ../*[BRIDGE = '%s']/DEVICE ])/DEVICE

I've implemented that in a netcf patch, and it reduces the time for virsh iface-dumpxml of all interfaces from 6m40s down to 17 seconds.

--- Additional comment from Laine Stump on 2015-10-13 13:04:58 EDT ---

This is the upstream patch that changes the query and reduces time (in the specific test case) from 6m40s to 17s:

commit 396e4e0698d9fb542f2eb8b32790a069e1c0df61
Author: Laine Stump <laine@laine.org>
Date:   Wed Oct 7 13:49:45 2015 -0400

    optimize aug_match() query for all ifcfg files related to an interface
Comment 1 Fedora Update System 2015-10-14 11:20:08 EDT
netcf-0.2.8-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-96dfd04b66
Comment 2 Fedora Update System 2015-10-14 11:22:15 EDT
netcf-0.2.8-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-86a85c887d
Comment 3 Fedora Update System 2015-10-14 18:52:49 EDT
netcf-0.2.8-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update netcf'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-96dfd04b66
Comment 4 Fedora Update System 2015-10-15 01:52:10 EDT
netcf-0.2.8-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update netcf'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-86a85c887d
Comment 5 Fedora Update System 2015-11-09 16:54:14 EST
netcf-0.2.8-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 6 Fedora Update System 2015-11-09 19:24:19 EST
netcf-0.2.8-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.