Bug 1720482
| Summary: | virNWFilterUndefine takes significant amount of time | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Roman Hodain <rhodain> |
| Component: | libvirt | Assignee: | Laine Stump <laine> |
| Status: | CLOSED DEFERRED | QA Contact: | yalzhang <yalzhang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.6 | CC: | berrange, dyuan, gwatson, jsuchane, lmen, sirao, xuzhang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-05 15:54:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Defining of nwfilters is known to be slow, largely due to the overhead of running many individual iptables commands. This is discussed in quite some detail here https://www.redhat.com/archives/libvir-list/2018-September/msg01206.html Unfortunately undefine is also slow because it ultimately triggers a rebuild of filters too. It is not actually necessary to undefine first if you just want to update the existing filter. In fact it will be blocked if any running VM is using the filter. Instead apps can just define over the top of an existing filter causing automatic rebuild as needed. Overall - VDSM shouldn't call undefine at all - If it expects that rules have changed, then call define, but it will still be slow. Ideally don't call define if you don't have changes - Libvirt could be improved to optimize speed of undefine without much trouble, but - Optimizing speed of define is very difficult Based on our discussion I have created Bug 1722854. So if that is OK with RHV ENG we may close this one. Bug 1722854 is in POST and it sounds like it has solved the problem for RHV, so I'm closing this BZ (although if the nwfilter driver is ever refactored, we should keep this inefficiency in mind) |
Description of problem: virNWFilterUndefine takes more than 4 minutes to finish for 60 VMs. Each VM has just one interface with <filterref filter='vdsm-no-mac-spoofing'/> The virNWFilterUndefine triggers modifications to the firewall [1]. One modification per virExec call. This means that virExec is triggered about 1500 times. One virtExec takes about 0.04 sec, but in the number of calls, it takes a long time. The entire operation takes 4m34.725s. All delay is spent directly in libvirt. The same operation takes about 16 sec after the libvirtd restart. Version-Release number of selected component (if applicable): libvirt-4.5.0-10.el7_6.2.x86_64 How reproducible: 100% in the affected environment. We were not able to reproduce it a test environment. Actual results: The operation takes a long time and stacked products (RHV) are affected by this. Expected results: FInish the operation in a reasonable amount of time. In a number of seconds. Additional info: [1]: /usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/iptables -w -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/iptables -w -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1} /usr/sbin/iptables -w -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1} /usr/sbin/iptables -w -F FP-vnet${1} /usr/sbin/iptables -w -X FP-vnet${1} /usr/sbin/iptables -w -F FJ-vnet${1} /usr/sbin/iptables -w -X FJ-vnet${1} /usr/sbin/iptables -w -F HJ-vnet${1} /usr/sbin/iptables -w -X HJ-vnet${1} /usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/ip6tables -D libvirt-out -m physdev --physdev-out vnet${1} -g FP-vnet${1} /usr/sbin/ip6tables -D libvirt-in -m physdev --physdev-in vnet${1} -g FJ-vnet${1} /usr/sbin/ip6tables -D libvirt-host-in -m physdev --physdev-in vnet${1} -g HJ-vnet${1} /usr/sbin/ip6tables -F FP-vnet${1} /usr/sbin/ip6tables -X FP-vnet${1} /usr/sbin/ip6tables -F FJ-vnet${1} /usr/sbin/ip6tables -X FJ-vnet${1} /usr/sbin/ip6tables -F HJ-vnet${1} /usr/sbin/ip6tables -X HJ-vnet${1} /usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet${1} -j libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet${1} -j libvirt-P-vnet${1} /usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet${1} /usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet${1} /usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet${1} /usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet${1}