RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1720482 - virNWFilterUndefine takes significant amount of time
Summary: virNWFilterUndefine takes significant amount of time
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-14 06:47 UTC by Roman Hodain
Modified: 2023-03-24 14:55 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-05 15:54:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Roman Hodain 2019-06-14 06:47:03 UTC
Description of problem:
virNWFilterUndefine takes more than 4 minutes to finish for 60 VMs. Each VM has just one interface with 

    <filterref filter='vdsm-no-mac-spoofing'/>

The virNWFilterUndefine triggers modifications to the firewall [1]. One modification per virExec call. This means that virExec is triggered about 1500 times.
One virtExec takes about 0.04 sec, but in the number of calls, it takes a long time.  The entire operation takes 4m34.725s. All delay is spent directly in libvirt.

The same operation takes about 16 sec after the libvirtd restart.


Version-Release number of selected component (if applicable):
libvirt-4.5.0-10.el7_6.2.x86_64

How reproducible:
100% in the affected environment. We were not able to reproduce it a test environment.

Actual results:
The operation takes a long time and stacked products (RHV) are affected by this.

Expected results:
FInish the operation in a reasonable amount of time. In a number of seconds.

Additional info:

[1]:
/usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/iptables -w -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1}
/usr/sbin/iptables -w -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1}
/usr/sbin/iptables -w -F FP-vnet${1}
/usr/sbin/iptables -w -X FP-vnet${1}
/usr/sbin/iptables -w -F FJ-vnet${1}
/usr/sbin/iptables -w -X FJ-vnet${1}
/usr/sbin/iptables -w -F HJ-vnet${1}
/usr/sbin/iptables -w -X HJ-vnet${1}
/usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/ip6tables -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1}
/usr/sbin/ip6tables -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1}
/usr/sbin/ip6tables -F FP-vnet${1}
/usr/sbin/ip6tables -X FP-vnet${1}
/usr/sbin/ip6tables -F FJ-vnet${1}
/usr/sbin/ip6tables -X FJ-vnet${1}
/usr/sbin/ip6tables -F HJ-vnet${1}
/usr/sbin/ip6tables -X HJ-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet${1} -j libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet${1} -j libvirt-P-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet${1}

Comment 5 Daniel Berrangé 2019-06-21 12:30:22 UTC
Defining of nwfilters is known to be slow, largely due to the overhead of running many individual iptables commands. This is discussed in quite some detail here

  https://www.redhat.com/archives/libvir-list/2018-September/msg01206.html

Unfortunately undefine is also slow because it ultimately triggers a rebuild of filters too.

It is not actually necessary to undefine first if you just want to update the existing filter. In fact it will be blocked if any running VM is using the filter. Instead apps can just define over the top of an existing filter causing automatic rebuild as needed. 

Overall

 - VDSM shouldn't call undefine at all
 - If it expects that rules have changed, then call define, but it will still be slow. Ideally don't call define if you don't have changes
 - Libvirt could be improved to optimize speed of undefine without much trouble, but 
 - Optimizing speed of define is very difficult

Comment 6 Roman Hodain 2019-06-26 11:18:52 UTC
Based on our discussion I have created Bug 1722854. So if that is OK with RHV ENG we may close this one.

Comment 7 Laine Stump 2019-08-05 15:54:09 UTC
Bug 1722854 is in POST and it sounds like it has solved the problem for RHV, so I'm closing this BZ (although if the nwfilter driver is ever refactored, we should keep this inefficiency in mind)


Note You need to log in before you can comment on or make changes to this bug.