Can you boot the hypervisor and add "loglvl=all" to the hypervisor command-line as well?  That should give us a bit more information about why it failed, although I have a guess.

Also, can you add the machine details so that someone can reserve the machine and take a look?

Thanks,
Chris Lalancette

kernel-2.6.18-175 has a Xen NUMA fix (BZ 526051), please try the latest kernel and report if issue persists.

The machine is intel-s3e36-01.rhts.eng.nay.redhat.com.
First, I retested this bug with kernel-2.6.18-175 on the same machine as Bhavna said, but it still failed and log was the same.
BTW, I also tested the 5.4 z-stream bug #539684(BZ 526051) these days and that one was for MC.
From then test result, I guess problem is different as for these two bugs.

Created attachment 375287 [details]
boot msg with loglvl set to all

Yes. With loglvl, probelm emerged.

That's more or less what I expected, but it is good to get confirmation.  To be clear, even if you get this message, the system still boots and works, right?

It's still a bug, I just want to make sure the hypervisor is falling back and doing the right thing in the case it can't properly enable NUMA.

Chris Lalancette

Yes, it boots and works. But numa doesnt work judged from the boot msgs and command line shows.

I suppose the same happens for a bare-metal kernel?

kernel-2.6.18-205.el5 would be fine.

reassigned

The configuration here is strange:

(XEN) SRAT: Node 0 PXM 0 0-5d00000000
(XEN) SRAT: Node 2 PXM 1 880000000-7d00000000

Note that the nodes overlap due to the hotplug areas.  Needs to see what Linux does in this case.

(In reply to comment #3)
> The machine is intel-s3e36-01.rhts.eng.nay.redhat.com.

(I don't have permission to provision the machine, so I can't retest.)

Probably irrelevant, but:
- the Beaker data sheet (the Details tab) doesn't mention "NUMA Nodes" at all for this machine,
- according to attachment 375287 [details], "xenheap_megabytes" was not passed to Xen, even though Beaker shows 110 GB RAM.

(In reply to comment #12)
> The configuration here is strange:
> 
> (XEN) SRAT: Node 0 PXM 0 0-5d00000000
> (XEN) SRAT: Node 2 PXM 1 880000000-7d00000000
> 
> Note that the nodes overlap due to the hotplug areas.  Needs to see what Linux
> does in this case.

Perhaps...

(
http://xenbits.xensource.com/hg/xen-unstable.hg/rev/ebb07c5934c8
http://xenbits.xensource.com/hg/xen-unstable.hg/rev/1f5f1674e53f
)

http://xenbits.xensource.com/hg/xen-unstable.hg/rev/e5a757ce7845

?

Created attachment 519278 [details]
fix nodes' memory parsing when SRAT table includes future-hotplug memory range

backport xen-unstable c/s 20053:ebb07c5934c8
cannibalize page_is_ram_type() into maddr_is_conv_ram()

Confirming with -281:

  (XEN) Command line: loglvl=all guest_loglvl=all bootscrub=0 numa=on
        com1=115200,8n1

This is the RAM map:

  (XEN) Xen-e820 RAM map:
  (XEN)  0000000000000000 - 000000000009b000 (usable)
  [...]
  (XEN)  0000000100000000 - 0000002080000000 (usable)

Then the hot-plug zones:

  (XEN) SRAT: hot plug zone found 2100000000 - 4100000000 
  (XEN) SRAT: hot plug zone found 4100000000 - 6100000000 
  (XEN) SRAT: hot plug zone found 6100000000 - 8100000000 
  (XEN) SRAT: hot plug zone found 8100000000 - a100000000 
  (XEN) SRAT: hot plug zone found a100000000 - c100000000 
  (XEN) SRAT: hot plug zone found c100000000 - e100000000 
  (XEN) SRAT: hot plug zone found e100000000 - 10100000000 
  (XEN) SRAT: hot plug zone found 10100000000 - 12100000000 

Verdict:

  (XEN) NUMA: Using 25 for the hash shift.
  (XEN) Your memory is not aligned you need to rebuild your kernel with a
        bigger NODEMAPSIZE shift=25
  (XEN) SRAT: No NUMA node hash function found. Contact maintainer
  (XEN) SRAT: SRAT not used.
  (XEN) No NUMA configuration found
  (XEN) Faking a node at 0000000000000000-0000002080000000
  (XEN) Xen heap: 13MB (13932kB)
  (XEN) Domain heap initialised: DMA width 32 bits

Notice how the lowest hot-plug zone (2100000000) is above the highest usable (or any other kind of) area provided in the RAM map (2080000000).

The patched hypervisor (-281 + attachment 519278 [details]) logs this instead (after displaying the same e820 map):

  (XEN) SRAT: unsupported future hotplug zone found 2100000000 - 4100000000 
  (XEN) SRAT: unsupported future hotplug zone found 4100000000 - 6100000000 
  (XEN) SRAT: unsupported future hotplug zone found 6100000000 - 8100000000 
  (XEN) SRAT: unsupported future hotplug zone found 8100000000 - a100000000 
  (XEN) SRAT: unsupported future hotplug zone found a100000000 - c100000000 
  (XEN) SRAT: unsupported future hotplug zone found c100000000 - e100000000 
  (XEN) SRAT: unsupported future hotplug zone found e100000000 - 10100000000 
  (XEN) SRAT: unsupported future hotplug zone found 10100000000 - 12100000000 
  (XEN) NUMA: Using 31 for the hash shift.
  (XEN) Xen heap: 13MB (13932kB)
  (XEN) Reserving non-aligned node boundary @ mfn 8912896
  (XEN) Reserving non-aligned node boundary @ mfn 17301504
  (XEN) Reserving non-aligned node boundary @ mfn 25690112
  (XEN) Domain heap initialised: DMA width 32 bits

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Patch(es) available in kernel-2.6.18-285.el5
You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5
Detailed testing feedback is always welcomed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0150.html