Bug 219951 - Xen performance on NUMA systems is fairly poor
Xen performance on NUMA systems is fairly poor
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel-xen (Show other bugs)
5.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Xen Maintainance List
:
Depends On:
Blocks: 234654 197865 225424
  Show dependency treegraph
 
Reported: 2006-12-17 12:04 EST by Chris Lalancette
Modified: 2008-08-13 23:38 EDT (History)
3 users (show)

See Also:
Fixed In Version: 5.0.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-01-19 08:21:08 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Spec diff to add NUMA support to RHEL-5 (1.24 KB, patch)
2006-12-17 12:09 EST, Chris Lalancette
no flags Details | Diff

  None (edit)
Description Chris Lalancette 2006-12-17 12:04:05 EST
Description of problem:
Because Xen is not currently NUMA aware, it's performance on NUMA machines is
definitely sub-optimal.  There are some patches available for upstream Xen 3.0.4
that address this; these should certainly be considered for 5.1.  Additionally,
IBM would like a release note laying out the reasons for the poor performance.
Comment 1 Chris Lalancette 2006-12-17 12:07:57 EST
From IBM:

I've finished the port of the 4 changesets from xen-unstable needed to enable
NUMA support in the Xen hypervisor.

The work involved the following:

Acquiring the relavent patches from xen-unstable.hg:

wget -O xen-numa-01-11893.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=f312c2d01d8b;style=raw"
wget -O xen-numa-02-11894.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=cf95c3218a70;style=raw"
wget -O xen-numa-03-11895.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=041507e2754c;style=raw"
wget -O xen-numa-04-11920.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=3db344ea8070;style=raw"

Installing the kernel source rpm for 12/07 drop:

rpm -ivh kernel-2.6.18-1.2839.el5.src.rpm

Putting the patches in /usr/src/redhat/SOURCES

Modifying the spec file to patch in the numa patches

Rebuilding the kernel package:  
 rpmbuild --target x86_64 --define='dist el5' -ba kernel-2.6.spec

The resultant kernel-xen rpm was installed:
 rpm -Uvh  kernel-xen-2.6.18-1.2840.el5.x86_64.rpm
on my test machine (2-way opteron which has SRAT tables) and I've test booted it
and it appears to function.  

I'll be attaching the binary kernel which can be used for testing, the
modifications to the specfile which were needed for building xen with the numa
patches, the source rpm which has all of the changes contained with in it, and
the xm dmesg output which shows the NUMA code functioning.
Comment 2 Chris Lalancette 2006-12-17 12:09:01 EST
Created attachment 143869 [details]
Spec diff to add NUMA support to RHEL-5
Comment 3 Don Domingo 2006-12-17 22:14:28 EST
Chris,

kindly post the text that IBM needs to be included in the release notes. note
that the deadline for Final Release Notes is on 19 December 2006.

making this bug block release notes, for tracking purposes.

thanks!


Cheers,
Don
Comment 4 Chris Lalancette 2006-12-18 18:56:07 EST
Don,
     Something like the following should work for the release notes:

-------------------------------------------------------------------------------
The hypervisor shipped in RHEL-5 GA is not NUMA aware; therefore, it's
performance on NUMA machines may be sub-optimal.  This issue will be addressed
in a future RHEL-5 update.
--------------------------------------------------------------------------------

Yeah, yeah, I know it's kind of lame but I don't know what the workaround is
right now; maybe setting node interleaving would help, but I haven't tested so I
can't say for sure.  I'll ask IBM if this is a valid workaround, but I probably
won't hear before the deadline.  Anyway, the above is at least something.

Chris Lalancette
Comment 5 Daniel Berrange 2006-12-18 22:00:46 EST
We should probably actually recommend that the admins turn on 'memory node
interleaving'  in the BIOS for NUMA systems. This will ensure there is more
consistent performance, by making the memory round-robin interleave in 4k blocks.
Comment 6 Don Domingo 2006-12-18 22:33:38 EST
added to release notes:

<quote>

At present, the Virtualization feature has the following limitations:

<snip>

* The hypervisor included in this release of Virtualization is not NUMA-aware;
as such, its performance on NUMA machines may be sub-optimal. This will be
addressed in a future update of Red Hat Enterprise Linux 4.92.

To work around this, you can enable memory node interleaving in the NUMA
machine's BIOS. This ensures a more consistent performance.
</quote>
Comment 9 Jay Turner 2007-01-19 08:21:08 EST
Closing out . . . please ensure that a RHEL5.1 bug is entered to track
resolution of the original issue.
Comment 14 Konrad Rzeszutek 2007-03-02 17:58:28 EST
This BZ is closed. Is that the correct state it should be in?

Note You need to log in before you can comment on or make changes to this bug.