Bug 446089

Summary: xend / xenstored performance & scalability issues
Product: Red Hat Enterprise Linux 5 Reporter: Bryan Mason <nobody+bjmason>
Component: xenAssignee: Daniel Berrangé <berrange>
Status: CLOSED ERRATA QA Contact: Michal Marciniszyn <mmarcini>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.2CC: duck, michel.monbernard, mmarcini, sputhenp, tao, tvujec, xen-maint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 21:14:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 449772    
Attachments:
Description Flags
Allow putting xenstored on tmpfs none

Description Bryan Mason 2008-05-12 17:32:23 UTC
Description of problem:

    Running 'virsh list' or 'xm list' consumes too much CPU time. When
    the number pf VMs is relatively high (10) and they are loaded with
    network activity the answer to the vmlist command may become very
    long (minutes) or may sometimes never complete.

    This problem existed to some extent in RHEL 5.1, but has gotten
    worse in RHEL 5.2.

    It appears xenstored is taking up most of the CPU time.

Version-Release number of selected component (if applicable):

    xen-3.0.3-41.el5

How reproducible:

    Can be reproduced after a fashion by running 'xm list' or 'virsh
    list' in a while loop in a script:

       #!/bin/bash
       trap "exit;" INT
       while true; do xm list &> /dev/null; usleep 5000;done

    If you taskset the script to a single CPU, it will consume >75% of
    that CPU.

Steps to Reproduce:

    1.  See above.
  
Actual results:

    'xm list'/xenstored takes up lots of CPU time and/or runs slowly
    when there are lots of VMs on the system.

Expected results:

    'xm list'/xenstored shouldn't consume as much CPU time.

Additional info:

    http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00487.html

    A workaround that we've come up with is to mount
    /var/lib/xenstored on tmpfs.  This reduces the overall system load
    when 'xm list' is run.

Comment 1 RHEL Program Management 2008-06-02 20:00:21 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 2 Daniel Berrangé 2008-07-11 13:51:59 UTC
Created attachment 311576 [details]
Allow putting xenstored on tmpfs

NB, with this patch applied, the user still needs to opt-in by setting

XENSTORED_TMPFS=yes  

in /etc/sysconfig/xend

Comment 4 Daniel Berrangé 2008-07-11 14:00:19 UTC
*** Bug 434146 has been marked as a duplicate of this bug. ***

Comment 5 Daniel Berrangé 2008-07-21 10:48:21 UTC
Built into xen-3.0.3-67.el5

Comment 8 Michal Marciniszyn 2008-12-16 16:19:45 UTC
The performance is increased by a huge factor. Tested with 10-15 virt machines and it did not consume more than 20% of processor with xm list running all the time.

Comment 10 errata-xmlrpc 2009-01-20 21:14:23 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0118.html