Bug 860053 - Quadratic function-like call delay
Quadratic function-like call delay
Status: CLOSED WORKSFORME
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
unspecified
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Libvirt Maintainers
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-24 15:09 EDT by Andrey Korolyov
Modified: 2014-05-07 15:45 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-05-07 15:45:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Debug log from 30 and 68 guests (2.09 MB, application/x-tar)
2012-09-24 15:09 EDT, Andrey Korolyov
no flags Details

  None (edit)
Description Andrey Korolyov 2012-09-24 15:09:58 EDT
Created attachment 616686 [details]
Debug log from 30 and 68 guests

Description of problem:


Version-Release number of selected component (if applicable): 0.9.x and 0.10.x


How reproducible:


- start a bunch of VMs,
- stop libvirt,
- start libvirt and immediately issue any call, say, 'virsh version',
- delay until call completion may be fitted nearly as quadratic
function from number of running VMs,

  
Actual results: slower than linear initial timeout


Expected results: linear or sublinear time


Additional info: 

zgrep  virEventPollDispatchHandles libvirtd-68.log.gz | wc -l
92603
zgrep  virEventPollDispatchHandles libvirtd-30.log.gz | wc -l
21713
Comment 1 Doug Goldstein 2012-09-24 18:02:48 EDT
Could be completely unrelated but one of the slowdowns I've discovered is for storage pools that are marked autostart. libvirt restarting attempts to start them back up, but they're already started so there's just a bad interaction which takes some time to sort itself out. iSCSI is one of the pool types that experiences this issue and provides a very long delay. LVM also causes a delay but it very much depends on the version of LVM as to how libvirt reacts.
Comment 2 Daniel Berrange 2012-09-25 16:33:59 EDT
I don't think that dealing with storage pool autostart would cause a quadratic slow down in API handling at startup. In fact, storage pool autostart should block all libvirt API access completely. So I think there's some other explanation here.

I think something related to the event loop could well be relevant here. Each VM started adds to the number of file handles managed in the event loop, and these are dealt with using a number of linear array searches.

It might be interesting to try and collect some profile data using oprofile.
Comment 3 Andrey Korolyov 2014-05-07 15:45:10 EDT
Closing this as problem seems to be solved by greatly decreased multiplier, though bigger-than-linear dependency stays.

Note You need to log in before you can comment on or make changes to this bug.