Bug 446089 - xend / xenstored performance & scalability issues
xend / xenstored performance & scalability issues
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.2
All Linux
medium Severity medium
: rc
: ---
Assigned To: Daniel Berrange
Michal Marciniszyn
:
: 434146 (view as bug list)
Depends On:
Blocks: 449772
  Show dependency treegraph
 
Reported: 2008-05-12 13:32 EDT by Bryan Mason
Modified: 2014-02-10 18:03 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 16:14:23 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Allow putting xenstored on tmpfs (1.05 KB, patch)
2008-07-11 09:51 EDT, Daniel Berrange
no flags Details | Diff

  None (edit)
Description Bryan Mason 2008-05-12 13:32:23 EDT
Description of problem:

    Running 'virsh list' or 'xm list' consumes too much CPU time. When
    the number pf VMs is relatively high (10) and they are loaded with
    network activity the answer to the vmlist command may become very
    long (minutes) or may sometimes never complete.

    This problem existed to some extent in RHEL 5.1, but has gotten
    worse in RHEL 5.2.

    It appears xenstored is taking up most of the CPU time.

Version-Release number of selected component (if applicable):

    xen-3.0.3-41.el5

How reproducible:

    Can be reproduced after a fashion by running 'xm list' or 'virsh
    list' in a while loop in a script:

       #!/bin/bash
       trap "exit;" INT
       while true; do xm list &> /dev/null; usleep 5000;done

    If you taskset the script to a single CPU, it will consume >75% of
    that CPU.

Steps to Reproduce:

    1.  See above.
  
Actual results:

    'xm list'/xenstored takes up lots of CPU time and/or runs slowly
    when there are lots of VMs on the system.

Expected results:

    'xm list'/xenstored shouldn't consume as much CPU time.

Additional info:

    http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00487.html

    A workaround that we've come up with is to mount
    /var/lib/xenstored on tmpfs.  This reduces the overall system load
    when 'xm list' is run.
Comment 1 RHEL Product and Program Management 2008-06-02 16:00:21 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 2 Daniel Berrange 2008-07-11 09:51:59 EDT
Created attachment 311576 [details]
Allow putting xenstored on tmpfs

NB, with this patch applied, the user still needs to opt-in by setting

XENSTORED_TMPFS=yes  

in /etc/sysconfig/xend
Comment 4 Daniel Berrange 2008-07-11 10:00:19 EDT
*** Bug 434146 has been marked as a duplicate of this bug. ***
Comment 5 Daniel Berrange 2008-07-21 06:48:21 EDT
Built into xen-3.0.3-67.el5
Comment 8 Michal Marciniszyn 2008-12-16 11:19:45 EST
The performance is increased by a huge factor. Tested with 10-15 virt machines and it did not consume more than 20% of processor with xm list running all the time.
Comment 10 errata-xmlrpc 2009-01-20 16:14:23 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0118.html

Note You need to log in before you can comment on or make changes to this bug.