Bug 511231 - libvirtd slows down
libvirtd slows down
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt (Show other bugs)
5.4
All Linux
low Severity medium
: rc
: ---
Assigned To: Daniel Veillard
Virtualization Bugs
hp:dl785solblk
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-07-14 07:04 EDT by Aron Griffis
Modified: 2010-11-09 08:32 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-09-24 06:00:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Aron Griffis 2009-07-14 07:04:31 EDT
Description of problem:
$ time virsh list --all
real	0m48.353s
user	0m0.043s
sys	0m0.025s

$ service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]

$ time virsh list --all
real	0m0.593s
user	0m0.041s
sys	0m0.022s

Version-Release number of selected component (if applicable):
RHEL 5.4 snapshot 1
libvirt-0.6.3-13.el5.x86_64

How reproducible:
I've seen this happen numerous times, just haven't taken the time to collect the timing info until now.

Steps to Reproduce:
1. use virt-install --import to define 500 guests
2. run virsh list --all
3. I'm also intermittently running the guests, but I took the timing info while no guest was running. I don't know if running the guests is required to reproduce the problem.  If not, you can probably reproduce it without a big box.
Comment 1 Aron Griffis 2009-07-14 07:07:55 EDT
For debugging a different problem, I had libvirtd configured with:

log_level = 1
log_outputs = "1:file:/var/log/libvirt/libvirtd.log"

The log appears to have been lost when I restarted libvirtd, but I wonder if the slowness could be related to debugging.  I'll disable debugging and keep you posted.
Comment 2 Daniel Berrange 2009-07-14 07:15:18 EDT
The logging calls could well have a non-trivial impact on performance since they are very very verbose at times, and if your host OS filesystem can't keep up that might in theory slow down the whole daemon. 

If you need specific logging it is better to not use 'log_level' and instead set  log_filters to refer to the filename which you require debugging for. eg to get of public API calls made (eg src/libvirt.c) you would set

log_filters="1:libvirt.c"
log_outsputs="1:file:/var/log/libvirt/libvirtd.log"

that should significantly restrict the amount of data logged to the point where it wont/shouldn't impact performance

Another possibility is that something or other in libvirtd is holding a mutex lock on one of the virtual machines, which would slow down calls.
Comment 3 Hugh Brock 2009-10-30 12:07:56 EDT
Aron, what happened when you disabled debugging? Did the problem go away?
Comment 4 Daniel Veillard 2010-01-14 10:25:37 EST
No new for a couple of months, so too late for 5.5, retargetting it for 5.6

Daniel
Comment 7 Daniel Veillard 2010-09-24 06:00:46 EDT
I have retried on 5.5 with libvirt-0.8.2-6.el5 expected to be pushed in 5.6 and
could not reproduce this with either Xen or KVM domain (based on context I think
the testing was done with KVM):


Xen:

50
real    0m0.482s
user    0m0.008s
sys 0m0.000s

100
real    0m0.898s
user    0m0.000s
sys 0m0.008s

200
real    0m1.724s
user    0m0.000s
sys 0m0.032s

300
real    0m2.558s
user    0m0.028s
sys 0m0.008s

400
real    0m3.392s
user    0m0.032s
sys 0m0.020s

500
real    0m4.215s
user    0m0.016s
sys 0m0.040s


KVM:
50
real    0m0.020s
user    0m0.005s
sys 0m0.004s

100
real    0m0.040s
user    0m0.011s
sys 0m0.009s

200
real    0m0.083s
user    0m0.026s
sys 0m0.006s

300
real    0m0.127s
user    0m0.030s
sys 0m0.011s

400
real    0m0.169s
user    0m0.043s
sys 0m0.019s

500
real    0m0.195s
user    0m0.048s
sys 0m0.024s

 it's way faster with KVM but in any case this seems completely linear
w.r.t. the number of domain being defined.

Daniel

Note You need to log in before you can comment on or make changes to this bug.