Bug 220132

I am putting just IA64, but actually the problems are largely NUMA related.

OK, please update the bug if there's anything concrete you need to propose being
merged at any stage.  Thanks!

change QA contact

Just confirming that Xen does not boot on Altix, even a small 4cpu system.

*** Bug 446594 has been marked as a duplicate of this bug. ***

George,
   Can you please give specific details of the issue that surround the Altix platform and Xen code base in RHEL5

Thanks,
Jeff

This issue also shows up on  hp-rx8640-03.rhts.bos.redhat.com. hp-rx8640-02 does not have this problem. My understanding is -03 is configured for numa while -02 is not.

failure log from hp-rx8640-03:

http://rhts.redhat.com/cgi-bin/rhts/test_log.cgi?id=4691724

Jeff,

    Its been some time since I talked with Jes about Xen as SGI
decided that KVM was in our best interest.

    My understanding is that Xen doesn't understand NUMA and this
leads to unacceptable performance degradation even for relatively
small machines, e.g., 4 nodes w/ 16 cores.  The other part of the
puzzle was handling gaps in physical memory; on Altix each node
starts at a fixed address and so there are holes between nodes
in the memory map.

    If you need more information, I will need to get Jes's help.

George

George,
    Thanks for the information. Could you please speak with Jes again. This seems to be more of a performance issue. These systems panic when they try and boot. Also there is conflicting information on why SGI systems can't run the xen kernel.

    It would be good for SGI to give specific reasons on what any/all the issues are.

Thanks,
Jeff

George,
   Any update?

Thanks,
Jeff

Jeff,
The following Q/As are from an internal SGI document --- please treat
them as such.


 Q: Red Hat have announced Xen on ia64 support, what about Altix?
 A: Xen/ia64 runs on DIG compliant platforms and platforms that are
    almost DIG compliant, such as the HP systems (DIG is the Intel
    reference platform). Altix is _not_ DIG compliant.

 Q: When will Xen be ready for Altix?
 A: Xen will not be ported to Altix - the long term solution /
    replacement is KVM.

 Q: Why are we not porting or providing Xen for Altix?
 A: Xen is based on a micro-kernel design. The architecture of Xen
    means that it is running on the bare hardware, ie. it's effectively
    a separate operating system, which costs almost as much to port and
    support as it costs to support Linux. In addition, Xen's designers
    have decided to re-implement most core operating system components,
    including memory management, the scheduler and hardware device
    management. It has taken years to make these components scalable
    and NUMA aware in Linux - Xen is starting from scratch.

 Q: How does KVM differ from Xen?
 A: The fundamental difference between KVM and Xen is that while Xen is
    a micro-kernel based hypervisor, KVM is a Linux kernel module which
    runs it's virtual machines as tasks under Linux. KVM therefore
    benefits from the NUMA and scalability features already available
    in Linux. The IO model is also more efficient as a KVM virtual
    machine will call straight into the host Linux kernel to have IO
    served. A Xen virtual machine will call into Xen which will then
    hand off the IO request to a privileged guest which has access to
    the physical hardware - this significantly increases IO overhead.

Not mentioned above, but my understanding is that there were political
issues with Xen's designers in that they were simply not interested in
128 processor and larger systems that SGI ships.

Ultimately the issues are scalability and resources.  A determination
was made that KVM offered far better scalability and, because it was
being introduced into the mainline kernel, would require fewer resources
to maintain in the long run.

Based on Comment #13, I'm going to close this out.  If you disagree, feel free to re-open.

Chris Lalancette