Bug 1035966
Summary: | Start autostarted virtual networks in background | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Hu Jianwei <jiahu> | ||||
Component: | libvirt | Assignee: | Laine Stump <laine> | ||||
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
Severity: | low | Docs Contact: | |||||
Priority: | low | ||||||
Version: | 7.0 | CC: | ajia, berrange, dyuan, honzhang, jdenemar, mzhan, rbalakri | ||||
Target Milestone: | rc | Keywords: | TestOnly | ||||
Target Release: | --- | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | libvirt-1.2.7-1.el7 | Doc Type: | Bug Fix | ||||
Doc Text: |
Restarting libvirtd with a large number of virtual networks configured was taking a very long time on systems with firewalld enabled, with libvirtd's services unavailable in the interim. This was due to libvirt repeatedly exec'ing firewall-cmd to add the relevant rules to the host's iptables chains, which was extremely slow. libvirt now manipulates firewalld via the dbus interface, which tests show to be 10x faster, resulting in greatly reduced downtime during libvirts restarts.
|
Story Points: | --- | ||||
Clone Of: | Environment: | ||||||
Last Closed: | 2015-03-05 07:28:41 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Hu Jianwei
2013-11-29 03:37:37 UTC
More questions on this issue: 1. Virsh commands(such as "virsh list --all", "virsh net-list --all", etc) will be blocked by libvirtd while its daemon is starting a lot of virtual networks in background. When libvirtd handled all background events(here is virtual network), all virsh commands responded to customer quickly. I think libvirtd should deal with foreground input with higher priority at anytime. What do you think of my opinion? 2. Are there some risk conditions between user input and libvirt daemon or among libvitd's child threads? Thanks. (In reply to Hu Jianwei from comment #1) > 2. Are there some risk conditions between user input and libvirt daemon or s/risk/race/, Jianwei, please attach relevant log files and make sure you haven't touch any limitation. Created attachment 831471 [details]
libvirtd.log and system messages log.
I can't find any useful log from attached logs, no error info recorded.
This is basically a request to start virtual networks in background so that a starting libvirtd is not blocked until all networks are set up. While this is a nice idea, I'm not convinced it's worth implementing. Is there any real reason one would need to deploy lots of autostarted virtual networks on a single host? Before we even look at doing clever things in autostarting networks, we should get clear data on what is slow about starting them serially. If we can make starting of individual networks faster then that is a better win overall than focusing on playing tricks at libvirtd startup. (In reply to Jiri Denemark from comment #4) > This is basically a request to start virtual networks in background so that > a starting libvirtd is not blocked until all networks are set up. While this > is a nice idea, I'm not convinced it's worth implementing. Is there any real > reason one would need to deploy lots of autostarted virtual networks on a > single host? No customer requirement for real deployment, only a testing scenario, but I think we could do some optimization to get better user experience, if possible. Dan Berrange has just pushed patches upstream that have a very significant effect on network start time when firewalld is enabled on the system - if the testing done above was on a system with firewalld enabled, it would be useful to try it again with a build of the latest upstream source to see if the situation is now acceptable. Yes, speed of starting virtual networks when firewalld is running has improved dramatically https://www.redhat.com/archives/libvir-list/2014-April/msg00335.html "timing the network driver $ for i in `seq 1 10` ; do virsh net-start default; virsh net-destroy default ; done Direct iptables: 3 seconds Via firewall-cmd: 42 seconds" So on a firewalld enabled host we cut time to start 10 networks from 42 seconds down to 3 seconds, so it is on a par with non-firewalld based hosts. As Daniel said, this should be dramatically improved by not using firewalld-cmd anymore. Compared with former results, libvirt has a great improvement. For 249 virtual networks, virsh will take 2m4.517s For 504 virtual networks, virsh will take 5m10.606s For 759 virtual networks, virsh will take 8m45.235s (On old libvirt: 256 vnet will spend 19m5.009s 765 vnet will spend 79m39.460s ) Detail steps: [root@localhost 1035966]# rpm -q libvirt kernel libvirt-1.2.7-1.el7.x86_64 kernel-3.10.0-138.el7.x86_64 For 249 virtual networks [root@localhost 1035966]# virsh net-list | grep active | wc -l 249 [root@localhost 1035966]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 2m4.517s user 0m0.007s sys 0m0.004s [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 0m0.015s user 0m0.007s sys 0m0.003s For 504 virtual networks [root@localhost 1035966]# virsh net-list | grep active | wc -l 504 [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 0m0.016s user 0m0.005s sys 0m0.005s [root@localhost 1035966]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 5m10.606s user 0m0.006s sys 0m0.006s [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 0m0.017s user 0m0.007s sys 0m0.005s For 759 virtual networks [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 0m0.020s user 0m0.005s sys 0m0.007s [root@localhost 1035966]# virsh net-list | grep active | wc -l 759 [root@localhost 1035966]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 8m45.235s user 0m0.005s sys 0m0.006s [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 0m0.018s user 0m0.004s sys 0m0.008s According to comment 11, it's acceptable, changed to Verified. I found a machine, its libvirtd has a quicker reply than other machines I used before. 251 networks(real 0m21.877s) 510 networks(real 1m10.441s) 765 networks(real 2m25.811s) Software environment: [root@intel-e31225-8-3 network]# rpm -q libvirt kernel libvirt-1.2.8-12.el7.x86_64 kernel-3.10.0-221.el7.x86_64 [root@intel-e31225-8-3 network]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 42 Model name: Intel(R) Xeon(R) CPU E31225 @ 3.10GHz Stepping: 7 CPU MHz: 2928.410 BogoMIPS: 6184.19 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 6144K NUMA node0 CPU(s): 0-3 [root@intel-e31225-8-3 network]# free -g total used free shared buff/cache available Mem: 7 2 0 0 4 4 Swap: 7 0 7 [root@intel-e31225-8-3 network]# uptime 16:25:46 up 5:45, 2 users, load average: 0.01, 0.06, 10.21 [root@intel-e31225-8-3 network]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@intel-e31225-8-3 network]# time virsh -q net-list |wc -l 765 real 2m20.285s user 0m0.069s sys 0m0.050s Same software environment on another machine: [root@hp-dl385g7-08 network]# rpm -q libvirt kernel libvirt-1.2.8-12.el7.x86_64 kernel-3.10.0-221.el7.x86_64 [root@hp-dl385g7-08 network]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 4 Vendor ID: AuthenticAMD CPU family: 21 Model: 1 Model name: AMD Opteron(TM) Processor 6272 Stepping: 2 CPU MHz: 2100.010 BogoMIPS: 4199.77 Virtualization: AMD-V L1d cache: 16K L1i cache: 64K L2 cache: 2048K L3 cache: 6144K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 NUMA node1 CPU(s): 16,18,20,22,24,26,28,30 NUMA node2 CPU(s): 1,3,5,7,9,11,13,15 NUMA node3 CPU(s): 17,19,21,23,25,27,29,31 [root@hp-dl385g7-08 network]# free -g total used free shared buff/cache available Mem: 62 3 47 0 11 58 Swap: 31 0 31 [root@hp-dl385g7-08 network]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@hp-dl385g7-08 network]# time virsh net-list | wc -l 768 real 20m54.478s user 0m0.178s sys 0m0.308s Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |