Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
virNWFilterUndefine takes more than 4 minutes to finish for 60 VMs. Each VM has just one interface with
<filterref filter='vdsm-no-mac-spoofing'/>
The virNWFilterUndefine triggers modifications to the firewall [1]. One modification per virExec call. This means that virExec is triggered about 1500 times.
One virtExec takes about 0.04 sec, but in the number of calls, it takes a long time. The entire operation takes 4m34.725s. All delay is spent directly in libvirt.
The same operation takes about 16 sec after the libvirtd restart.
Version-Release number of selected component (if applicable):
libvirt-4.5.0-10.el7_6.2.x86_64
How reproducible:
100% in the affected environment. We were not able to reproduce it a test environment.
Actual results:
The operation takes a long time and stacked products (RHV) are affected by this.
Expected results:
FInish the operation in a reasonable amount of time. In a number of seconds.
Additional info:
[1]:
/usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/iptables -w -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1}
/usr/sbin/iptables -w -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1}
/usr/sbin/iptables -w -F FP-vnet${1}
/usr/sbin/iptables -w -X FP-vnet${1}
/usr/sbin/iptables -w -F FJ-vnet${1}
/usr/sbin/iptables -w -X FJ-vnet${1}
/usr/sbin/iptables -w -F HJ-vnet${1}
/usr/sbin/iptables -w -X HJ-vnet${1}
/usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1}
/usr/sbin/ip6tables -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1}
/usr/sbin/ip6tables -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1}
/usr/sbin/ip6tables -F FP-vnet${1}
/usr/sbin/ip6tables -X FP-vnet${1}
/usr/sbin/ip6tables -F FJ-vnet${1}
/usr/sbin/ip6tables -X FJ-vnet${1}
/usr/sbin/ip6tables -F HJ-vnet${1}
/usr/sbin/ip6tables -X HJ-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet${1} -j libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet${1} -j libvirt-P-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet${1}
/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet${1}
Defining of nwfilters is known to be slow, largely due to the overhead of running many individual iptables commands. This is discussed in quite some detail here
https://www.redhat.com/archives/libvir-list/2018-September/msg01206.html
Unfortunately undefine is also slow because it ultimately triggers a rebuild of filters too.
It is not actually necessary to undefine first if you just want to update the existing filter. In fact it will be blocked if any running VM is using the filter. Instead apps can just define over the top of an existing filter causing automatic rebuild as needed.
Overall
- VDSM shouldn't call undefine at all
- If it expects that rules have changed, then call define, but it will still be slow. Ideally don't call define if you don't have changes
- Libvirt could be improved to optimize speed of undefine without much trouble, but
- Optimizing speed of define is very difficult
Bug 1722854 is in POST and it sounds like it has solved the problem for RHV, so I'm closing this BZ (although if the nwfilter driver is ever refactored, we should keep this inefficiency in mind)
Description of problem: virNWFilterUndefine takes more than 4 minutes to finish for 60 VMs. Each VM has just one interface with <filterref filter='vdsm-no-mac-spoofing'/> The virNWFilterUndefine triggers modifications to the firewall [1]. One modification per virExec call. This means that virExec is triggered about 1500 times. One virtExec takes about 0.04 sec, but in the number of calls, it takes a long time. The entire operation takes 4m34.725s. All delay is spent directly in libvirt. The same operation takes about 16 sec after the libvirtd restart. Version-Release number of selected component (if applicable): libvirt-4.5.0-10.el7_6.2.x86_64 How reproducible: 100% in the affected environment. We were not able to reproduce it a test environment. Actual results: The operation takes a long time and stacked products (RHV) are affected by this. Expected results: FInish the operation in a reasonable amount of time. In a number of seconds. Additional info: [1]: /usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/iptables -w -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1} /usr/sbin/iptables -w -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1} /usr/sbin/iptables -w -F FP-vnet${1} /usr/sbin/iptables -w -X FP-vnet${1} /usr/sbin/iptables -w -F FJ-vnet${1} /usr/sbin/iptables -w -X FJ-vnet${1} /usr/sbin/iptables -w -F HJ-vnet${1} /usr/sbin/iptables -w -X HJ-vnet${1} /usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/ip6tables -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1} /usr/sbin/ip6tables -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1} /usr/sbin/ip6tables -F FP-vnet${1} /usr/sbin/ip6tables -X FP-vnet${1} /usr/sbin/ip6tables -F FJ-vnet${1} /usr/sbin/ip6tables -X FJ-vnet${1} /usr/sbin/ip6tables -F HJ-vnet${1} /usr/sbin/ip6tables -X HJ-vnet${1} /usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet${1} -j libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet${1} -j libvirt-P-vnet${1} /usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet${1} /usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet${1} /usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet${1}