Bug 219951

From IBM:

I've finished the port of the 4 changesets from xen-unstable needed to enable
NUMA support in the Xen hypervisor.

The work involved the following:

Acquiring the relavent patches from xen-unstable.hg:

wget -O xen-numa-01-11893.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=f312c2d01d8b;style=raw"
wget -O xen-numa-02-11894.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=cf95c3218a70;style=raw"
wget -O xen-numa-03-11895.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=041507e2754c;style=raw"
wget -O xen-numa-04-11920.patch
"http://xenbits.xensource.com/xen-unstable.hg?cs=3db344ea8070;style=raw"

Installing the kernel source rpm for 12/07 drop:

rpm -ivh kernel-2.6.18-1.2839.el5.src.rpm

Putting the patches in /usr/src/redhat/SOURCES

Modifying the spec file to patch in the numa patches

Rebuilding the kernel package:  
 rpmbuild --target x86_64 --define='dist el5' -ba kernel-2.6.spec

The resultant kernel-xen rpm was installed:
 rpm -Uvh  kernel-xen-2.6.18-1.2840.el5.x86_64.rpm
on my test machine (2-way opteron which has SRAT tables) and I've test booted it
and it appears to function.  

I'll be attaching the binary kernel which can be used for testing, the
modifications to the specfile which were needed for building xen with the numa
patches, the source rpm which has all of the changes contained with in it, and
the xm dmesg output which shows the NUMA code functioning.

Created attachment 143869 [details]
Spec diff to add NUMA support to RHEL-5

Chris,

kindly post the text that IBM needs to be included in the release notes. note
that the deadline for Final Release Notes is on 19 December 2006.

making this bug block release notes, for tracking purposes.

thanks!


Cheers,
Don

Don,
     Something like the following should work for the release notes:

-------------------------------------------------------------------------------
The hypervisor shipped in RHEL-5 GA is not NUMA aware; therefore, it's
performance on NUMA machines may be sub-optimal.  This issue will be addressed
in a future RHEL-5 update.
--------------------------------------------------------------------------------

Yeah, yeah, I know it's kind of lame but I don't know what the workaround is
right now; maybe setting node interleaving would help, but I haven't tested so I
can't say for sure.  I'll ask IBM if this is a valid workaround, but I probably
won't hear before the deadline.  Anyway, the above is at least something.

Chris Lalancette

We should probably actually recommend that the admins turn on 'memory node
interleaving'  in the BIOS for NUMA systems. This will ensure there is more
consistent performance, by making the memory round-robin interleave in 4k blocks.

added to release notes:

<quote>

At present, the Virtualization feature has the following limitations:

<snip>

* The hypervisor included in this release of Virtualization is not NUMA-aware;
as such, its performance on NUMA machines may be sub-optimal. This will be
addressed in a future update of Red Hat Enterprise Linux 4.92.

To work around this, you can enable memory node interleaving in the NUMA
machine's BIOS. This ensures a more consistent performance.
</quote>

Closing out . . . please ensure that a RHEL5.1 bug is entered to track
resolution of the original issue.

This BZ is closed. Is that the correct state it should be in?