Bug 877244

Summary: Virsh command will delay a long time if restart libvirtd with many virtual networks running
Product: Red Hat Enterprise Linux 7 Reporter: yanbing du <ydu>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: cwei, dyuan, jiahu, mprivozn, mzhan, rbalakri
Target Milestone: rcKeywords: TestOnly
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 878359 (view as bug list) Environment:
Last Closed: 2015-03-05 07:19:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 878359    
Attachments:
Description Flags
libvirtd-debug.log none

Description yanbing du 2012-11-16 02:53:38 UTC 
Description of problem:
This bug find during testing bug 869650.
When define and start more than 500 networks, and restart libvirtd service, then execute any virsh command will take a long time(only the first time). 

Version-Release number of selected component (if applicable):
Both 
libvirt-0.9.10-21.el6_3.6.x86_64
libvirt-0.10.2-8.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Define and start 500 virtual networks
2. Restart libvirtd service
# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]
3. Check the time of virsh list take
# time virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	3m16.719s
user	0m0.010s
sys	0m0.010s

run it again:

# time virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	0m0.063s
user	0m0.009s
sys	0m0.022s

4. Destroy 250 networks, and keep others still running
5. Restart libvirtd service
6. Check the time virsh net-list take
# time virsh net-list |wc -l
254

real	1m0.596s
user	0m0.400s
sys	0m0.078s

run it again:

# time virsh net-list |wc -l
254

real	0m1.524s
user	0m0.419s
sys	0m0.041s

7. Restart libvirtd service
8. Check the time virsh list take
# time virsh list
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	0m59.872s
user	0m0.009s
sys	0m0.013s

run it again:
  
# time virsh list
 Id    Name                           State
----------------------------------------------------
 4     test                           running


real	0m0.044s
user	0m0.006s
sys	0m0.010s


Actual results:
When restart libvirtd, more running virtual networks will lead virsh command delay more time 

Expected results:
virsh command can delay, but should not last so long.


Additional info:
Comment 2 Michal Privoznik 2012-11-16 07:53:07 UTC 
I believe this is because when libvirt is starting, it spawn multiple commands (e.g. iptables to insert firewall rules for network, dnsmasq to be DHCP server for network, and so on). If there are many networks to start, it can take some time. However, to tell for sure, yanbing can you please attach debug logs of libvirtd starting up? Thanks.
Comment 3 yanbing du 2012-11-19 03:08:12 UTC 
Created attachment 647477 [details]
libvirtd-debug.log

Yes, you are right. When libvirtd is starting, it takes long time to insert rules to iptables.
Attach the detail debug logs of libvirtd.
Comment 8 Jiri Denemark 2014-04-04 21:37:12 UTC 
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.
Comment 9 Hu Jianwei 2014-08-08 07:54:49 UTC 
My conclusions as below:

1. On rhel7, libvirt has a great improvement on response time after reboot libvirtd, testing results have been compared with former results,
you can refer to 
https://bugzilla.redhat.com/show_bug.cgi?id=1035966#c11

2. But compared with rhel6's results in comment 0, libvirt still takes long time.

Version:
libvirt-1.2.7-1.el7.x86_64
kernel-3.10.0-138.el7.x86_64

Setps:
1. Set Autostart=no for all virtual networks
[root@localhost 1035966]# virsh net-list | grep active | wc -l
504
[root@localhost 1035966]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	5m13.404s
user	0m0.005s
sys	0m0.006s

2. After destroy 255 virtual networks
[root@localhost 1035966]# virsh net-list | grep active | wc -l
249
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	0m0.014s
user	0m0.004s
sys	0m0.005s
[root@localhost 1035966]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	2m16.312s
user	0m0.007s
sys	0m0.005s
Comment 10 Hu Jianwei 2014-11-18 13:02:35 UTC 
According to comment 9, it's acceptable, changed to Verified.
Comment 12 errata-xmlrpc 2015-03-05 07:19:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html