The RHEL5.1 installation went fine. only after installing the RT kernel did this
issue start. Some additional data about the hardware.

********** System Information **********
Hostname                = hp-dl585-02.rhts.boston.redhat.com
Kernel Version          = 2.6.18-48.el5
Machine Hardware Name   = i686
Processor Type          = athlon
uname -a output         = Linux hp-dl585-02.rhts.boston.redhat.com 2.6.18-48.el5
#1 SMP Mon Sep 17 17:26:31 EDT 2007 i686 athlon i386 GNU/Linux
Swap Size               = 8191 MB
Mem Size                = 3890 MB
Number of Processors    = 8
System Release          = Red Hat Enterprise Linux Server release 5.1 Beta (Tikanga)
Command Line            = ro root=/dev/VolGroup00/LogVol00 console=ttyS0,115200
System NMI Interrupts   = NMI:          0          0          0          0     
    0          0          0          0 
********** LSPCI **********
00:03.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
00:04.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
00:04.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
00:04.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:07.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:07.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:08.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:08.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
00:1a.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:1a.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:1a.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:1a.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
00:1b.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:1b.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:1b.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:1b.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
01:02.0 System peripheral: Compaq Computer Corporation Integrated Lights Out
Controller (rev 01)
01:02.2 System peripheral: Compaq Computer Corporation Integrated Lights Out 
Processor (rev 01)
01:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
02:04.0 RAID bus controller: Compaq Computer Corporation Smart Array 5i/532 (rev 01)
02:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit
Ethernet (rev 10)
02:06.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit
Ethernet (rev 10)
04:09.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
04:09.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
04:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
04:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
04:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
04:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
04:0c.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
04:0c.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
********** Modprob **********
alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter cciss
********** Module Information **********
Checking module information autofs4:
Checking module information hidp:
Bluetooth HIDP ver 1.1
Checking module information rfcomm:
Bluetooth RFCOMM ver 1.8
Checking module information l2cap:
Bluetooth L2CAP ver 2.8
Checking module information bluetooth:
Bluetooth Core ver 2.10
Checking module information sunrpc:
Checking module information ipv6:
IPv6 protocol stack for Linux
Checking module information dm_multipath:
device-mapper multipath target
Checking module information video:
ACPI Video Driver
Checking module information sbs:
Smart Battery System ACPI interface driver
Checking module information backlight:
Backlight Lowlevel Control Abstraction
Checking module information i2c_ec:
ACPI EC SMBus driver
Checking module information button:
ACPI Button Driver
Checking module information battery:
ACPI Battery Driver
Checking module information asus_acpi:
Asus Laptop ACPI Extras Driver
Checking module information ac:
ACPI AC Adapter Driver
Checking module information parport_pc:
PC-style parallel port driver
Checking module information lp:
Checking module information parport:
Checking module information i2c_amd756:
AMD756/766/768/8111 and nVidia nForce SMBus driver
Checking module information i2c_core:
I2C-Bus main module
Checking module information serio_raw:
Raw serio driver
Checking module information floppy:
Checking module information ide_cd:
ATAPI CD-ROM Driver
Checking module information amd_rng:
H/W RNG driver for AMD chipsets
Checking module information cdrom:
Checking module information k8temp:
AMD K8 core temperature monitor
Checking module information hwmon:
hardware monitoring sysfs/class support
Checking module information tg3:
Broadcom Tigon3 ethernet driver
Checking module information k8_edac:
MC support for AMD K8 memory controllers -  Ver: 2.0.2 Sep 17 2007
Checking module information pcspkr:
PC Speaker beeper driver
Checking module information edac_mc:
Core library routines for MC reporting
Checking module information dm_snapshot:
device-mapper snapshot target
Checking module information dm_zero:
device-mapper dummy target returning zeros
Checking module information dm_mirror:
device-mapper mirror target
Checking module information dm_mod:
device-mapper driver
Checking module information cciss:
Driver for HP Controller SA5xxx SA6xxx version 3.6.16-RH1
Checking module information sd_mod:
SCSI disk (sd) driver
Checking module information scsi_mod:
SCSI core
Checking module information ext3:
Second Extended Filesystem with journaling extensions
Checking module information jbd:
Checking module information ehci_hcd:
10 Dec 2004 USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Checking module information ohci_hcd:
2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver
Checking module information uhci_hcd:
USB Universal Host Controller Interface driver

******** End System Information ********

Here is a link to the successful install and reboot with the standard RHEL5.1

http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=854730
You will need ot use your Bugzilla login to view the above information.

The original kernel boots fine with this in the log:

Linux version 2.6.18-48.el5 (brewbuilder.redhat.com) (gcc
version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Mon Sep 17 17:26:31 EDT 2007
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
 BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000f57f6800 (usable)
 BIOS-e820: 00000000f57f6800 - 00000000f5800000 (ACPI data)
 BIOS-e820: 00000000fdc00000 - 00000000fdc01000 (reserved)
 BIOS-e820: 00000000fdc10000 - 00000000fdc11000 (reserved)
 BIOS-e820: 00000000fdc20000 - 00000000fdc21000 (reserved)
 BIOS-e820: 00000000fdc30000 - 00000000fdc31000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fec10000 - 00000000fec11000 (reserved)
 BIOS-e820: 00000000fec20000 - 00000000fec21000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000017fffff000 (usable)
Warning only 4GB will be used.
Use a PAE enabled kernel.
3200MB HIGHMEM available.
896MB LOWMEM available.
...

The machine has 96 GB RAM, but this kernel does not have PAE enabled, so it uses
only 4 GB. Works OK.

On the other hand, with the -rt kernel there is:
...
97407MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000f4fa0
NX (Execute Disable) protection: active
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 -> 25165823
early_node_map[1] active PFN ranges
    0:        0 -> 25165823
bootmem alloc of 1409286144 bytes failed!
...

The -rt kernel has PAE enabled, so it tries to use all of RAM. 96 GB is a bit
too much for a 32-bit kernel. I think mem_map is so big that it doesn't even fit
into lowmem. According to http://www.redhat.com/rhel/compare/ the maximum memory
supported by RHEL on i386 is 16 GB. For 96 GB RAM x86_64 is necessary.

It should not crash like that, but it's not a supported configuration => setting
low priority of the bug.

2.6.21-39.el5rtvanilla boots fine on the machine. However, /proc/meminfo shows:
LowTotal:       125196 kB
LowFree:         57704 kB
So it has only 122 MB of low memory.

The same kernel when booted with mem=16G:
LowTotal:       780556 kB
LowFree:        738916 kB
i.e. much healthier 762 MB of low memory.

2.6.21-39.el5rt is also able to boot with mem=16G and has:
LowTotal:       677680 kB
LowFree:        626748 kB
So it has 661 MB of low memory. That's less than non-rt kernel, but usable.

Why is there a difference? The memory map takes more space on -rt, because
struct page is bigger (56 bytes vs. 32 bytes). The difference comes from a
spinlock contained in struct page:
#if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
            spinlock_t ptl;
#endif

On non-rt, spinlock_t is a raw_spinlock_t which is 8 bytes. On -rt, it is a
mutex and takes 32 bytes.

Are you suggesting that we should have a documented maximum memory configuration
for x86?  If so, what max limit do you recommend?

We already have the maximum documented. The limit for RHEL-RT should be the same
as RHEL5's. For RHEL5 we officially support up to 16GB RAM on x86. RHEL3 and
RHEL4 supported up to 64GB, but only with the -hugemem kernel which was droppped
in RHEL5. Anyone with more than 16GB RAM should be running a 64-bit system now.

That said, I believe the kernel should protect itself from the crash and
artificially limit the amount of RAM it detects. I'll make a patch.

Reiterated x86 max mem config in a release note.  closing as notabug.

Fixed by limit-i386-ram-to-16gb.patch in 2.6.21-46.el5rt.