Bug 877244 - Virsh command will delay a long time if restart libvirtd with many virtual networks running
Summary: Virsh command will delay a long time if restart libvirtd with many virtual ne...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 878359
TreeView+ depends on / blocked
 
Reported: 2012-11-16 02:53 UTC by yanbing du
Modified: 2015-03-05 07:19 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 878359 (view as bug list)
Environment:
Last Closed: 2015-03-05 07:19:52 UTC


Attachments (Terms of Use)
libvirtd-debug.log (14.30 MB, text/plain)
2012-11-19 03:08 UTC, yanbing du
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:0323 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2015-03-05 12:10:54 UTC

Description yanbing du 2012-11-16 02:53:38 UTC
Description of problem:
This bug find during testing bug 869650.
When define and start more than 500 networks, and restart libvirtd service, then execute any virsh command will take a long time(only the first time). 

Version-Release number of selected component (if applicable):
Both 
libvirt-0.9.10-21.el6_3.6.x86_64
libvirt-0.10.2-8.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Define and start 500 virtual networks
2. Restart libvirtd service
# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]
3. Check the time of virsh list take
# time virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	3m16.719s
user	0m0.010s
sys	0m0.010s

run it again:

# time virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	0m0.063s
user	0m0.009s
sys	0m0.022s

4. Destroy 250 networks, and keep others still running
5. Restart libvirtd service
6. Check the time virsh net-list take
# time virsh net-list |wc -l
254

real	1m0.596s
user	0m0.400s
sys	0m0.078s

run it again:

# time virsh net-list |wc -l
254

real	0m1.524s
user	0m0.419s
sys	0m0.041s

7. Restart libvirtd service
8. Check the time virsh list take
# time virsh list
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	0m59.872s
user	0m0.009s
sys	0m0.013s

run it again:
  
# time virsh list
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	0m0.044s
user	0m0.006s
sys	0m0.010s


Actual results:
When restart libvirtd, more running virtual networks will lead virsh command delay more time 

Expected results:
virsh command can delay, but should not last so long.


Additional info:

Comment 2 Michal Privoznik 2012-11-16 07:53:07 UTC
I believe this is because when libvirt is starting, it spawn multiple commands (e.g. iptables to insert firewall rules for network, dnsmasq to be DHCP server for network, and so on). If there are many networks to start, it can take some time. However, to tell for sure, yanbing can you please attach debug logs of libvirtd starting up? Thanks.

Comment 3 yanbing du 2012-11-19 03:08:12 UTC
Created attachment 647477 [details]
libvirtd-debug.log

Yes, you are right. When libvirtd is starting, it takes long time to insert rules to iptables.
Attach the detail debug logs of libvirtd.

Comment 8 Jiri Denemark 2014-04-04 21:37:12 UTC
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.

Comment 9 Hu Jianwei 2014-08-08 07:54:49 UTC
My conclusions as below:

1. On rhel7, libvirt has a great improvement on response time after reboot libvirtd, testing results have been compared with former results,
you can refer to 
https://bugzilla.redhat.com/show_bug.cgi?id=1035966#c11

2. But compared with rhel6's results in comment 0, libvirt still takes long time.

Version:
libvirt-1.2.7-1.el7.x86_64
kernel-3.10.0-138.el7.x86_64

Setps:
1. Set Autostart=no for all virtual networks
[root@localhost 1035966]# virsh net-list | grep active | wc -l
504
[root@localhost 1035966]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	5m13.404s
user	0m0.005s
sys	0m0.006s

2. After destroy 255 virtual networks
[root@localhost 1035966]# virsh net-list | grep active | wc -l
249
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	0m0.014s
user	0m0.004s
sys	0m0.005s
[root@localhost 1035966]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	2m16.312s
user	0m0.007s
sys	0m0.005s

Comment 10 Hu Jianwei 2014-11-18 13:02:35 UTC
According to comment 9, it's acceptable, changed to Verified.

Comment 12 errata-xmlrpc 2015-03-05 07:19:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html


Note You need to log in before you can comment on or make changes to this bug.