Description of problem: $ time virsh list --all real 0m48.353s user 0m0.043s sys 0m0.025s $ service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] $ time virsh list --all real 0m0.593s user 0m0.041s sys 0m0.022s Version-Release number of selected component (if applicable): RHEL 5.4 snapshot 1 libvirt-0.6.3-13.el5.x86_64 How reproducible: I've seen this happen numerous times, just haven't taken the time to collect the timing info until now. Steps to Reproduce: 1. use virt-install --import to define 500 guests 2. run virsh list --all 3. I'm also intermittently running the guests, but I took the timing info while no guest was running. I don't know if running the guests is required to reproduce the problem. If not, you can probably reproduce it without a big box.
For debugging a different problem, I had libvirtd configured with: log_level = 1 log_outputs = "1:file:/var/log/libvirt/libvirtd.log" The log appears to have been lost when I restarted libvirtd, but I wonder if the slowness could be related to debugging. I'll disable debugging and keep you posted.
The logging calls could well have a non-trivial impact on performance since they are very very verbose at times, and if your host OS filesystem can't keep up that might in theory slow down the whole daemon. If you need specific logging it is better to not use 'log_level' and instead set log_filters to refer to the filename which you require debugging for. eg to get of public API calls made (eg src/libvirt.c) you would set log_filters="1:libvirt.c" log_outsputs="1:file:/var/log/libvirt/libvirtd.log" that should significantly restrict the amount of data logged to the point where it wont/shouldn't impact performance Another possibility is that something or other in libvirtd is holding a mutex lock on one of the virtual machines, which would slow down calls.
Aron, what happened when you disabled debugging? Did the problem go away?
No new for a couple of months, so too late for 5.5, retargetting it for 5.6 Daniel
I have retried on 5.5 with libvirt-0.8.2-6.el5 expected to be pushed in 5.6 and could not reproduce this with either Xen or KVM domain (based on context I think the testing was done with KVM): Xen: 50 real 0m0.482s user 0m0.008s sys 0m0.000s 100 real 0m0.898s user 0m0.000s sys 0m0.008s 200 real 0m1.724s user 0m0.000s sys 0m0.032s 300 real 0m2.558s user 0m0.028s sys 0m0.008s 400 real 0m3.392s user 0m0.032s sys 0m0.020s 500 real 0m4.215s user 0m0.016s sys 0m0.040s KVM: 50 real 0m0.020s user 0m0.005s sys 0m0.004s 100 real 0m0.040s user 0m0.011s sys 0m0.009s 200 real 0m0.083s user 0m0.026s sys 0m0.006s 300 real 0m0.127s user 0m0.030s sys 0m0.011s 400 real 0m0.169s user 0m0.043s sys 0m0.019s 500 real 0m0.195s user 0m0.048s sys 0m0.024s it's way faster with KVM but in any case this seems completely linear w.r.t. the number of domain being defined. Daniel