Red Hat Bugzilla – Bug 877244
Virsh command will delay a long time if restart libvirtd with many virtual networks running
Last modified: 2015-03-05 02:19:52 EST
Description of problem: This bug find during testing bug 869650. When define and start more than 500 networks, and restart libvirtd service, then execute any virsh command will take a long time(only the first time). Version-Release number of selected component (if applicable): Both libvirt-0.9.10-21.el6_3.6.x86_64 libvirt-0.10.2-8.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1. Define and start 500 virtual networks 2. Restart libvirtd service # service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] 3. Check the time of virsh list take # time virsh list --all Id Name State ---------------------------------------------------- 4 test running real 3m16.719s user 0m0.010s sys 0m0.010s run it again: # time virsh list --all Id Name State ---------------------------------------------------- 4 test running real 0m0.063s user 0m0.009s sys 0m0.022s 4. Destroy 250 networks, and keep others still running 5. Restart libvirtd service 6. Check the time virsh net-list take # time virsh net-list |wc -l 254 real 1m0.596s user 0m0.400s sys 0m0.078s run it again: # time virsh net-list |wc -l 254 real 0m1.524s user 0m0.419s sys 0m0.041s 7. Restart libvirtd service 8. Check the time virsh list take # time virsh list Id Name State ---------------------------------------------------- 4 test running real 0m59.872s user 0m0.009s sys 0m0.013s run it again: # time virsh list Id Name State ---------------------------------------------------- 4 test running real 0m0.044s user 0m0.006s sys 0m0.010s Actual results: When restart libvirtd, more running virtual networks will lead virsh command delay more time Expected results: virsh command can delay, but should not last so long. Additional info:
I believe this is because when libvirt is starting, it spawn multiple commands (e.g. iptables to insert firewall rules for network, dnsmasq to be DHCP server for network, and so on). If there are many networks to start, it can take some time. However, to tell for sure, yanbing can you please attach debug logs of libvirtd starting up? Thanks.
Created attachment 647477 [details] libvirtd-debug.log Yes, you are right. When libvirtd is starting, it takes long time to insert rules to iptables. Attach the detail debug logs of libvirtd.
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.
My conclusions as below: 1. On rhel7, libvirt has a great improvement on response time after reboot libvirtd, testing results have been compared with former results, you can refer to https://bugzilla.redhat.com/show_bug.cgi?id=1035966#c11 2. But compared with rhel6's results in comment 0, libvirt still takes long time. Version: libvirt-1.2.7-1.el7.x86_64 kernel-3.10.0-138.el7.x86_64 Setps: 1. Set Autostart=no for all virtual networks [root@localhost 1035966]# virsh net-list | grep active | wc -l 504 [root@localhost 1035966]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 5m13.404s user 0m0.005s sys 0m0.006s 2. After destroy 255 virtual networks [root@localhost 1035966]# virsh net-list | grep active | wc -l 249 [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 0m0.014s user 0m0.004s sys 0m0.005s [root@localhost 1035966]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@localhost 1035966]# time virsh list --all Id Name State ---------------------------------------------------- 20 r7_latest running - r7 shut off - r7_1 shut off - r7_n shut off real 2m16.312s user 0m0.007s sys 0m0.005s
According to comment 9, it's acceptable, changed to Verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html