877244 – Virsh command will delay a long time if restart libvirtd with many virtual networks running

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 877244 - Virsh command will delay a long time if restart libvirtd with many virtual networks running

Summary: Virsh command will delay a long time if restart libvirtd with many virtual ne...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Michal Privoznik
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	878359
TreeView+	depends on / blocked

Reported:	2012-11-16 02:53 UTC by yanbing du
Modified:	2015-03-05 07:19 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	878359 (view as bug list)
Environment:
Last Closed:	2015-03-05 07:19:52 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
libvirtd-debug.log (14.30 MB, text/plain) 2012-11-19 03:08 UTC, yanbing du	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:0323	0	normal	SHIPPED_LIVE	Low: libvirt security, bug fix, and enhancement update	2015-03-05 12:10:54 UTC

Description yanbing du 2012-11-16 02:53:38 UTC

Description of problem:
This bug find during testing bug 869650.
When define and start more than 500 networks, and restart libvirtd service, then execute any virsh command will take a long time(only the first time).

Version-Release number of selected component (if applicable):
Both
libvirt-0.9.10-21.el6_3.6.x86_64
libvirt-0.10.2-8.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Define and start 500 virtual networks
2. Restart libvirtd service
# service libvirtd restart
Stopping libvirtd daemon: [ OK ]
Starting libvirtd daemon: [ OK ]
3. Check the time of virsh list take
# time virsh list --all
Id Name State
----------------------------------------------------
4 test running

real 3m16.719s
user 0m0.010s
sys 0m0.010s

run it again:

# time virsh list --all
Id Name State
----------------------------------------------------
4 test running

real 0m0.063s
user 0m0.009s
sys 0m0.022s

4. Destroy 250 networks, and keep others still running
5. Restart libvirtd service
6. Check the time virsh net-list take
# time virsh net-list |wc -l
254

real 1m0.596s
user 0m0.400s
sys 0m0.078s

run it again:

# time virsh net-list |wc -l
254

real 0m1.524s
user 0m0.419s
sys 0m0.041s

7. Restart libvirtd service
8. Check the time virsh list take
# time virsh list
Id Name State
----------------------------------------------------
4 test running

real 0m59.872s
user 0m0.009s
sys 0m0.013s

run it again:

# time virsh list
Id Name State
----------------------------------------------------
4 test running

real 0m0.044s
user 0m0.006s
sys 0m0.010s

Actual results:
When restart libvirtd, more running virtual networks will lead virsh command delay more time

Expected results:
virsh command can delay, but should not last so long.

Additional info:

Comment 2 Michal Privoznik 2012-11-16 07:53:07 UTC

I believe this is because when libvirt is starting, it spawn multiple commands (e.g. iptables to insert firewall rules for network, dnsmasq to be DHCP server for network, and so on). If there are many networks to start, it can take some time. However, to tell for sure, yanbing can you please attach debug logs of libvirtd starting up? Thanks.

Comment 3 yanbing du 2012-11-19 03:08:12 UTC

Created attachment 647477 [details]
libvirtd-debug.log

Yes, you are right. When libvirtd is starting, it takes long time to insert rules to iptables.
Attach the detail debug logs of libvirtd.

Comment 8 Jiri Denemark 2014-04-04 21:37:12 UTC

This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.

Comment 9 Hu Jianwei 2014-08-08 07:54:49 UTC

My conclusions as below:

1. On rhel7, libvirt has a great improvement on response time after reboot libvirtd, testing results have been compared with former results,
you can refer to 
https://bugzilla.redhat.com/show_bug.cgi?id=1035966#c11

2. But compared with rhel6's results in comment 0, libvirt still takes long time.

Version:
libvirt-1.2.7-1.el7.x86_64
kernel-3.10.0-138.el7.x86_64

Setps:
1. Set Autostart=no for all virtual networks
[root@localhost 1035966]# virsh net-list | grep active | wc -l
504
[root@localhost 1035966]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	5m13.404s
user	0m0.005s
sys	0m0.006s

2. After destroy 255 virtual networks
[root@localhost 1035966]# virsh net-list | grep active | wc -l
249
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	0m0.014s
user	0m0.004s
sys	0m0.005s
[root@localhost 1035966]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost 1035966]#  time virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    r7_latest                      running
 -     r7                             shut off
 -     r7_1                           shut off
 -     r7_n                           shut off


real	2m16.312s
user	0m0.007s
sys	0m0.005s

Comment 10 Hu Jianwei 2014-11-18 13:02:35 UTC

According to comment 9, it's acceptable, changed to Verified.

Comment 12 errata-xmlrpc 2015-03-05 07:19:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html

Note You need to log in before you can comment on or make changes to this bug.