Bug 507033

This issue is somewhat similar to BZ 506805.  This is with respect to
Fedora11.

The issue is that, on multi-socket 5500 series / Nehalem systems like 
XE270, all of the cores are showing up on the first node.  Zero cores 
are showing up on the 2nd node.

Problem system:
SGI XE270, Supermicro X8DTN v 1.1 mainboard
Memory: 8 GB total, made up of 2GB DDR3 1066 MHz DIMMs, part number 
18JSF25672PY-1G1D1
2 4-core sockets, hyperthreading turned off.
cpu info: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
(cpuinfo will be attached).

For example, for the first node:

[root@cct201 ~]# ls /sys/devices/system/node/node0
cpu0  cpu2  cpu4  cpu6  cpulist  distance  numastat
cpu1  cpu3  cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages
[root@cct201 ~]# cat /sys/devices/system/node/node0/cpulist
0-7

For the 2nd node: 

[root@cct201 ~]# ls /sys/devices/system/node/node2
cpulist  cpumap  distance  meminfo  numastat  scan_unevictable_pages
[root@cct201 ~]# cat /sys/devices/system/node/node2/cpulist

[root@cct201 ~]# 

/proc/cpuinfo confirms there are indeed two cores and that hyperthreading is
off.  cpu cores is 4, siblings is 4, and the total core count is 8.

kernel version is: 2.6.29.4-167.fc11.x86_64

It is reported by Kannan Somangili, who also observed this problem on a
different multi-socket Nehalem system, that RHEL 5.3 and SLES 10 SP2 do 
not suffer from this problem.

dmesg is being attached.  Here are some interesting dmesg segments:

[root@cct201 ~]# dmesg|grep SRAT
ACPI: SRAT BF79A4C0, 0150 (r1 052809 OEMSRAT         1 INTL        1)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 0 -> APIC 2 -> Node 0
SRAT: PXM 0 -> APIC 4 -> Node 0
SRAT: PXM 0 -> APIC 6 -> Node 0
SRAT: PXM 1 -> APIC 16 -> Node 1
SRAT: PXM 1 -> APIC 18 -> Node 1
SRAT: PXM 1 -> APIC 20 -> Node 1
SRAT: PXM 1 -> APIC 22 -> Node 1
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 100000-c0000000
SRAT: Node 0 PXM 0 100000000-140000000
SRAT: Node 2 PXM 257 140000000-240000000


Here is a segment with tracebacks:

NUMA: Allocated memnodemap from 18000 - 1c880
NUMA: Using 20 for the hash shift.
Bootmem setup node 0 0000000000000000-0000000140000000
  NODE_DATA [000000000001c880 - 000000000003187f]
  bootmap [0000000000032000 -  0000000000059fff] pages 28
(8 early reservations) ==> bootmem [0000000000 - 0140000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0000200000 - 0000ac700c]    TEXT DATA BSS ==> [0000200000 - 0000ac700c]
  #3 [0037c97000 - 0037fefdc0]          RAMDISK ==> [0037c97000 - 0037fefdc0]
  #4 [000009a800 - 0000100000]    BIOS reserved ==> [000009a800 - 0000100000]
  #5 [0000010000 - 0000013000]          PGTABLE ==> [0000010000 - 0000013000]
  #6 [0000013000 - 0000018000]          PGTABLE ==> [0000013000 - 0000018000]
  #7 [0000018000 - 000001c880]       MEMNODEMAP ==> [0000018000 - 000001c880]
Bootmem setup node 2 0000000140000000-0000000240000000
  NODE_DATA [0000000140000000 - 0000000140014fff]
  bootmap [0000000140015000 -  0000000140034fff] pages 20
(8 early reservations) ==> bootmem [0140000000 - 0240000000]
  #0 [0000000000 - 0000001000]   BIOS data page
  #1 [0000006000 - 0000008000]       TRAMPOLINE
  #2 [0000200000 - 0000ac700c]    TEXT DATA BSS
  #3 [0037c97000 - 0037fefdc0]          RAMDISK
  #4 [000009a800 - 0000100000]    BIOS reserved
  #5 [0000010000 - 0000013000]          PGTABLE
  #6 [0000013000 - 0000018000]          PGTABLE
  #7 [0000018000 - 000001c880]       MEMNODEMAP
found SMP MP-table at [ffff8800000ff780] 000ff780
 [ffffe20000000000-ffffe200045fffff] PMD -> [ffff880028200000-ffff88002b9fffff] on node 0
 [ffffe20004600000-ffffe20007dfffff] PMD -> [ffff880140200000-ffff8801439fffff] on node 2
Zone PFN ranges: