Bug 503048

Summary: LS21 do not boot RT enabled kernels (not APIC issue) - ibm-ls21-7972-01.rhts.bos.redhat.com
Product: Red Hat Enterprise MRG Reporter: David Sommerseth <davids>
Component: realtime-kernelAssignee: Clark Williams <williams>
Status: CLOSED ERRATA QA Contact: David Sommerseth <davids>
Severity: high Docs Contact:
Priority: low    
Version: 1.1CC: bhu, lgoncalv, ovasik, pzijlstr, sassmann
Target Milestone: 1.1.5   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-07-14 19:12:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Attempt to fix free_bootmem for ls21
none
Fixed version of the last patch
none
Force free_bootmem to use the current node none

Description David Sommerseth 2009-05-28 14:44:02 UTC
Noticed that this host (ibm-ls21-7972-01.rhts.bos.redhat.com) in particular do not boot kernel-rt-2.6.24.7-117.  It boots stock RHEL5.3 kernel.  I've also tried -107 kernel as well, without success.

The -107 kernel solved an issue regarding to HPET, but this seems to be something completely different.

Log from one boot attempt:
-------------------------------------------------------------------------------
Linux version 2.6.24.7-117.el5rt (mockbuild.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP PREEMPT RT Wed May 20 13:39:07 EDT 2009
Command line: ro root=/dev/VolGroup00/LogVol00 console=ttyS1,19200 debug ignore_loglevel earlyprintk=serial,ttyS1,19200 noapic
BIOS-provided physical RAM map:                                                
 BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
 BIOS-e820: 000000000009d000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ffa5980 (usable)
 BIOS-e820: 000000007ffa5980 - 000000007ffa8800 (ACPI data)
 BIOS-e820: 000000007ffa8800 - 0000000080000000 (reserved)
 BIOS-e820: 00000000f4000000 - 00000000fc000000 (reserved)
 BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
console [earlyser0] enabled
end_pfn_map = 1048576
kernel direct mapping tables up to 100000000 @ 8000-d000
DMI 2.3 present.
ACPI: RSDP 000FDFC0, 0014 (r0 IBM   )
ACPI: RSDT 7FFA8780, 0030 (r1 IBM    SERLEWIS     1000 IBM  45444F43)
ACPI: FACP 7FFA86C0, 0084 (r2 IBM    SERLEWIS     1000 IBM  45444F43)
ACPI: DSDT 7FFA5980, 2B02 (r1 IBM    SERLEWIS     1000 INTL  2002025)
ACPI: FACS 7FFA8500, 0040
ACPI: APIC 7FFA8600, 0090 (r1 IBM    SERLEWIS     1000 IBM  45444F43)
ACPI: SRAT 7FFA8540, 00C0 (r1 AMD    HAMMER          1 AMD         1)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 0 -> APIC 1 -> Node 0
SRAT: PXM 1 -> APIC 2 -> Node 1
SRAT: PXM 1 -> APIC 3 -> Node 1
SRAT: Node 1 PXM 1 0-a0000
SRAT: Node 1 PXM 1 0-80000000
Bootmem setup node 1 0000000000000000-000000007ffa5000
PANIC: early exception rip ffffffff814ba27d error 0 cr2 73c8
Pid: 0, comm: swapper Not tainted 2.6.24.7-117.el5rt #1

Call Trace:
 [<ffffffff814ba27d>] ? free_bootmem+0x14/0x22
 [<ffffffff814bc584>] ? sparse_init+0x11d/0x196
 [<ffffffff814b6178>] ? paging_init+0x41/0x94
 [<ffffffff814acdb9>] ? setup_arch+0x471/0x4e4
 [<ffffffff814a68c0>] ? start_kernel+0x76/0x329
 [<ffffffff814a6119>] ? _sinittext+0x119/0x120

RIP free_bootmem+0x14/0x22
-------------------------------------------------------------------------------
$ addr2line -f -e  /usr/lib/debug/lib/modules/2.6.24.7-117.el5rt/vmlinux ffffffff814ba27d
free_bootmem
/usr/src/debug/kernel-rt-2.6.24.7-117.el5rt/linux-2.6.24.7.x86_64/mm/bootmem.c:409
-------------------------------------------------------------------------------

This seems to be only an x86_64 issue, as we have seen it working on i686.  It also boots the rt-vanilla kernel.  Ref: http://rhts.redhat.com/cgi-bin/rhts/jobs.cgi?id=60504

Comment 1 Luis Claudio R. Goncalves 2009-05-28 15:34:37 UTC
Created attachment 345780 [details]
Attempt to fix free_bootmem for ls21

void __init setup_node_bootmem()
...
       printk(KERN_INFO "Bootmem setup node %d %016lx-%016lx\n", nodeid, start, end);
...
       memset(NODE_DATA(nodeid), 0, sizeof(pg_data_t));void __init 
...
}

And:

free_bootmem(unsigned long addr, unsigned long size)
{
       free_bootmem_core(NODE_DATA(0)->bdata, addr, size);
}

As the log has the message:

       Bootmem setup node 1 0000000000000000-000000007ffa5000

And given the code we try to act on NODE_DATA(0) even though what has been set was NODE_DATA(1), this code shoul fix the issue.

Comment 2 Luis Claudio R. Goncalves 2009-05-28 17:27:41 UTC
Created attachment 345803 [details]
Fixed version of the last patch

There was a missing hunk in the earlier patch. I am right now compiling a kernel to test this patch.

Comment 3 Luis Claudio R. Goncalves 2009-05-28 20:14:28 UTC
Created attachment 345835 [details]
Force free_bootmem to use the current node

Force free_bootmem to use the current node

David Sommerseth has noticed that a ls21 (7972) machine was not booting with
MRG RT V1 kernel 2.6.24.7-117.el5rt. He was able to get a kernel backtrace:

        Bootmem setup node 1 0000000000000000-000000007ffa5000
        PANIC: early exception rip ffffffff814ba27d error 0 cr2 73c8
        Pid: 0, comm: swapper Not tainted 2.6.24.7-117.el5rt #1

        Call Trace:
         [<ffffffff814ba27d>] ? free_bootmem+0x14/0x22
         [<ffffffff814bc584>] ? sparse_init+0x11d/0x196
         [<ffffffff814b6178>] ? paging_init+0x41/0x94
         [<ffffffff814acdb9>] ? setup_arch+0x471/0x4e4
         [<ffffffff814a68c0>] ? start_kernel+0x76/0x329
         [<ffffffff814a6119>] ? _sinittext+0x119/0x120

        RIP free_bootmem+0x14/0x22

According to setup_node_bootmem(), the message "Bootmem setup node 1 ..." tells
us that the process is happening on node 1:

       printk(KERN_INFO "Bootmem setup node %d %016lx-%016lx\n", nodeid, ...);
...
       memset(NODE_DATA(nodeid), 0, sizeof(pg_data_t));void __init

But when free_bootmem is called it has node 0 hardcoded:

       free_bootmem_core(NODE_DATA(0)->bdata, addr, size);

Leading to the early panic scenario observed by David.

This is a simple fix, forcing free_bootmem() to act over the current node.
---

Comment 4 Luis Claudio R. Goncalves 2009-05-28 20:28:00 UTC
Added to kernel -118 for testing.

Comment 6 David Sommerseth 2009-07-09 14:15:15 UTC
In RHTS job 70101 (http://rhts.redhat.com/cgi-bin/rhts/jobs.cgi?id=70101) the problematic LS21 blade (ibm-ls21-7972-01.rhts.bos.redhat.com) has booted and ran 3 different -122 based kernels.

RHTS job 70720 (http://rhts.redhat.com/cgi-bin/rhts/jobs.cgi?id=70720) the same box booted and tested 2 kernels of the -126 based kernels.

Found attached patch as mrg-rt.git commit 16314b176e5ba7d2a8769c4a0ed8b89e4bd6813d in the 2.6.24.7-126 SRPM.

Comment 8 errata-xmlrpc 2009-07-14 19:12:08 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1157.html