Bug 446089 - xend / xenstored performance & scalability issues
Summary: xend / xenstored performance & scalability issues
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen
Version: 5.2
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Daniel Berrangé
QA Contact: Michal Marciniszyn
URL:
Whiteboard:
: 434146 (view as bug list)
Depends On:
Blocks: 449772
TreeView+ depends on / blocked
 
Reported: 2008-05-12 17:32 UTC by Bryan Mason
Modified: 2018-10-20 01:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-01-20 21:14:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Allow putting xenstored on tmpfs (1.05 KB, patch)
2008-07-11 13:51 UTC, Daniel Berrangé
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2009:0118 0 normal SHIPPED_LIVE xen bug fix and enhancement update 2009-01-20 16:04:49 UTC

Description Bryan Mason 2008-05-12 17:32:23 UTC
Description of problem:

    Running 'virsh list' or 'xm list' consumes too much CPU time. When
    the number pf VMs is relatively high (10) and they are loaded with
    network activity the answer to the vmlist command may become very
    long (minutes) or may sometimes never complete.

    This problem existed to some extent in RHEL 5.1, but has gotten
    worse in RHEL 5.2.

    It appears xenstored is taking up most of the CPU time.

Version-Release number of selected component (if applicable):

    xen-3.0.3-41.el5

How reproducible:

    Can be reproduced after a fashion by running 'xm list' or 'virsh
    list' in a while loop in a script:

       #!/bin/bash
       trap "exit;" INT
       while true; do xm list &> /dev/null; usleep 5000;done

    If you taskset the script to a single CPU, it will consume >75% of
    that CPU.

Steps to Reproduce:

    1.  See above.
  
Actual results:

    'xm list'/xenstored takes up lots of CPU time and/or runs slowly
    when there are lots of VMs on the system.

Expected results:

    'xm list'/xenstored shouldn't consume as much CPU time.

Additional info:

    http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00487.html

    A workaround that we've come up with is to mount
    /var/lib/xenstored on tmpfs.  This reduces the overall system load
    when 'xm list' is run.

Comment 1 RHEL Program Management 2008-06-02 20:00:21 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 2 Daniel Berrangé 2008-07-11 13:51:59 UTC
Created attachment 311576 [details]
Allow putting xenstored on tmpfs

NB, with this patch applied, the user still needs to opt-in by setting

XENSTORED_TMPFS=yes  

in /etc/sysconfig/xend

Comment 4 Daniel Berrangé 2008-07-11 14:00:19 UTC
*** Bug 434146 has been marked as a duplicate of this bug. ***

Comment 5 Daniel Berrangé 2008-07-21 10:48:21 UTC
Built into xen-3.0.3-67.el5

Comment 8 Michal Marciniszyn 2008-12-16 16:19:45 UTC
The performance is increased by a huge factor. Tested with 10-15 virt machines and it did not consume more than 20% of processor with xm list running all the time.

Comment 10 errata-xmlrpc 2009-01-20 21:14:23 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0118.html


Note You need to log in before you can comment on or make changes to this bug.