Bug 311411
Summary: | [RHEL5 RT] Kernel panic - not syncing: Out of memory | ||
---|---|---|---|
Product: | Red Hat Enterprise MRG | Reporter: | Jeff Burke <jburke> |
Component: | realtime-kernel | Assignee: | Michal Schmidt <mschmidt> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | |
Severity: | high | Docs Contact: | |
Priority: | low | ||
Version: | 1.0 | CC: | lwoodman, srostedt, tburke |
Target Milestone: | --- | Keywords: | Reopened |
Target Release: | --- | ||
Hardware: | i386 | ||
OS: | Linux | ||
URL: | http://rhts.lab.boston.redhat.com/cgi-bin/rhts/jobs.cgi?id=7856 | ||
Whiteboard: | |||
Fixed In Version: | 2.6.21-46.el5rt | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2007-10-26 09:22:38 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Jeff Burke
2007-09-28 17:41:34 UTC
The RHEL5.1 installation went fine. only after installing the RT kernel did this issue start. Some additional data about the hardware. ********** System Information ********** Hostname = hp-dl585-02.rhts.boston.redhat.com Kernel Version = 2.6.18-48.el5 Machine Hardware Name = i686 Processor Type = athlon uname -a output = Linux hp-dl585-02.rhts.boston.redhat.com 2.6.18-48.el5 #1 SMP Mon Sep 17 17:26:31 EDT 2007 i686 athlon i386 GNU/Linux Swap Size = 8191 MB Mem Size = 3890 MB Number of Processors = 8 System Release = Red Hat Enterprise Linux Server release 5.1 Beta (Tikanga) Command Line = ro root=/dev/VolGroup00/LogVol00 console=ttyS0,115200 System NMI Interrupts = NMI: 0 0 0 0 0 0 0 0 ********** LSPCI ********** 00:03.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:04.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:04.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:04.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 00:07.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:07.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 00:08.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:08.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:1a.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:1a.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:1a.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:1a.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:1b.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:1b.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:1b.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:1b.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 01:02.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 01) 01:02.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 01) 01:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 02:04.0 RAID bus controller: Compaq Computer Corporation Smart Array 5i/532 (rev 01) 02:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 02:06.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 04:09.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 04:09.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 04:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 04:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 04:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 04:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 04:0c.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 04:0c.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) ********** Modprob ********** alias eth0 tg3 alias eth1 tg3 alias scsi_hostadapter cciss ********** Module Information ********** Checking module information autofs4: Checking module information hidp: Bluetooth HIDP ver 1.1 Checking module information rfcomm: Bluetooth RFCOMM ver 1.8 Checking module information l2cap: Bluetooth L2CAP ver 2.8 Checking module information bluetooth: Bluetooth Core ver 2.10 Checking module information sunrpc: Checking module information ipv6: IPv6 protocol stack for Linux Checking module information dm_multipath: device-mapper multipath target Checking module information video: ACPI Video Driver Checking module information sbs: Smart Battery System ACPI interface driver Checking module information backlight: Backlight Lowlevel Control Abstraction Checking module information i2c_ec: ACPI EC SMBus driver Checking module information button: ACPI Button Driver Checking module information battery: ACPI Battery Driver Checking module information asus_acpi: Asus Laptop ACPI Extras Driver Checking module information ac: ACPI AC Adapter Driver Checking module information parport_pc: PC-style parallel port driver Checking module information lp: Checking module information parport: Checking module information i2c_amd756: AMD756/766/768/8111 and nVidia nForce SMBus driver Checking module information i2c_core: I2C-Bus main module Checking module information serio_raw: Raw serio driver Checking module information floppy: Checking module information ide_cd: ATAPI CD-ROM Driver Checking module information amd_rng: H/W RNG driver for AMD chipsets Checking module information cdrom: Checking module information k8temp: AMD K8 core temperature monitor Checking module information hwmon: hardware monitoring sysfs/class support Checking module information tg3: Broadcom Tigon3 ethernet driver Checking module information k8_edac: MC support for AMD K8 memory controllers - Ver: 2.0.2 Sep 17 2007 Checking module information pcspkr: PC Speaker beeper driver Checking module information edac_mc: Core library routines for MC reporting Checking module information dm_snapshot: device-mapper snapshot target Checking module information dm_zero: device-mapper dummy target returning zeros Checking module information dm_mirror: device-mapper mirror target Checking module information dm_mod: device-mapper driver Checking module information cciss: Driver for HP Controller SA5xxx SA6xxx version 3.6.16-RH1 Checking module information sd_mod: SCSI disk (sd) driver Checking module information scsi_mod: SCSI core Checking module information ext3: Second Extended Filesystem with journaling extensions Checking module information jbd: Checking module information ehci_hcd: 10 Dec 2004 USB 2.0 'Enhanced' Host Controller (EHCI) Driver Checking module information ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver Checking module information uhci_hcd: USB Universal Host Controller Interface driver ******** End System Information ******** Here is a link to the successful install and reboot with the standard RHEL5.1 http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=854730 You will need ot use your Bugzilla login to view the above information. The original kernel boots fine with this in the log: Linux version 2.6.18-48.el5 (brewbuilder.redhat.com) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Mon Sep 17 17:26:31 EDT 2007 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009f400 (usable) BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 00000000f57f6800 (usable) BIOS-e820: 00000000f57f6800 - 00000000f5800000 (ACPI data) BIOS-e820: 00000000fdc00000 - 00000000fdc01000 (reserved) BIOS-e820: 00000000fdc10000 - 00000000fdc11000 (reserved) BIOS-e820: 00000000fdc20000 - 00000000fdc21000 (reserved) BIOS-e820: 00000000fdc30000 - 00000000fdc31000 (reserved) BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) BIOS-e820: 00000000fec10000 - 00000000fec11000 (reserved) BIOS-e820: 00000000fec20000 - 00000000fec21000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved) BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 00000017fffff000 (usable) Warning only 4GB will be used. Use a PAE enabled kernel. 3200MB HIGHMEM available. 896MB LOWMEM available. ... The machine has 96 GB RAM, but this kernel does not have PAE enabled, so it uses only 4 GB. Works OK. On the other hand, with the -rt kernel there is: ... 97407MB HIGHMEM available. 896MB LOWMEM available. found SMP MP-table at 000f4fa0 NX (Execute Disable) protection: active Zone PFN ranges: DMA 0 -> 4096 Normal 4096 -> 229376 HighMem 229376 -> 25165823 early_node_map[1] active PFN ranges 0: 0 -> 25165823 bootmem alloc of 1409286144 bytes failed! ... The -rt kernel has PAE enabled, so it tries to use all of RAM. 96 GB is a bit too much for a 32-bit kernel. I think mem_map is so big that it doesn't even fit into lowmem. According to http://www.redhat.com/rhel/compare/ the maximum memory supported by RHEL on i386 is 16 GB. For 96 GB RAM x86_64 is necessary. It should not crash like that, but it's not a supported configuration => setting low priority of the bug. 2.6.21-39.el5rtvanilla boots fine on the machine. However, /proc/meminfo shows: LowTotal: 125196 kB LowFree: 57704 kB So it has only 122 MB of low memory. The same kernel when booted with mem=16G: LowTotal: 780556 kB LowFree: 738916 kB i.e. much healthier 762 MB of low memory. 2.6.21-39.el5rt is also able to boot with mem=16G and has: LowTotal: 677680 kB LowFree: 626748 kB So it has 661 MB of low memory. That's less than non-rt kernel, but usable. Why is there a difference? The memory map takes more space on -rt, because struct page is bigger (56 bytes vs. 32 bytes). The difference comes from a spinlock contained in struct page: #if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS spinlock_t ptl; #endif On non-rt, spinlock_t is a raw_spinlock_t which is 8 bytes. On -rt, it is a mutex and takes 32 bytes. Are you suggesting that we should have a documented maximum memory configuration for x86? If so, what max limit do you recommend? We already have the maximum documented. The limit for RHEL-RT should be the same as RHEL5's. For RHEL5 we officially support up to 16GB RAM on x86. RHEL3 and RHEL4 supported up to 64GB, but only with the -hugemem kernel which was droppped in RHEL5. Anyone with more than 16GB RAM should be running a 64-bit system now. That said, I believe the kernel should protect itself from the crash and artificially limit the amount of RAM it detects. I'll make a patch. Reiterated x86 max mem config in a release note. closing as notabug. Fixed by limit-i386-ram-to-16gb.patch in 2.6.21-46.el5rt. |