Bug 362371 - x86_64 largesmp kernel panics at boot time on DL585
x86_64 largesmp kernel panics at boot time on DL585
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.6
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Larry Woodman
Martin Jenner
: ZStream
Depends On:
Blocks: 248673 368411
  Show dependency treegraph
 
Reported: 2007-11-01 14:59 EDT by Larry Woodman
Modified: 2010-10-22 15:59 EDT (History)
5 users (show)

See Also:
Fixed In Version: RHBA-2007-0791
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-11-15 11:33:41 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Larry Woodman 2007-11-01 14:59:57 EDT
Description of problem:

x86_64 largesmp kernel panics at boot time on DL585-04 and DL585-03 but boots OK
on DL585-02

Version-Release number of selected component (if applicable):

RHEL4-U6


How reproduciblkernel-largesmp-2.6.9-65.test.EL.x86_64.rpme:

Panics on every boot


Steps to Reproduce:
1. Install kernel-largesmp-2.6.9-65.test.EL.x86_64.rpm
2. reboot
3.
  
Actual results:

Unable to handle kernel paging request at 0000000000006a80 RIP:
<ffffffff8015eb2d>{__alloc_pages+117}
PML4 f57b7067 PGD 3f0c02067 PMD 0
Oops: 0000 [1] SMP
CPU 2
Modules linked in: mptsas mptscsi mptbase qla2xxx scsi_transport_fc sd_mod scsi_mod
Pid: 559, comm: insmod Not tainted 2.6.9-65.ELlargesmp
RIP: 0010:[<ffffffff8015eb2d>] <ffffffff8015eb2d>{__alloc_pages+117}
RSP: 0000:00000103f0ef5888  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000040 RCX: 0000000000000220
RDX: 0000000000006a80 RSI: 0000000000000006 RDI: 0000000000000220
RBP: ffffffffffffffff R08: 0000000000000028 R09: 00000000000025a7
R10: 0000000000000430 R11: 0000ffff804061c0 R12: 0000000000000220
R13: 0000000000006a80 R14: 0000000000000006 R15: 00000100f57127f0
FS:  0000000000000000(0000) GS:ffffffff80506100(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000006a80 CR3: 00000000f579e000 CR4: 00000000000006e0
Process insmod (pid: 559, threadinfo 00000103f0ef4000, task 00000100f57127f0)
Stack: 0000000000000001 0000000100000246 0000000000000000 0000000000000001
       0000000000000014 0000000000000040 ffffffffffffffff 0000010037c7aa30
       000000000002c600 0000000000000220
Call Trace:<ffffffff801210b5>{dma_alloc_pages+125}
<ffffffff801212dc>{dma_alloc_coherent+97}
       <ffffffffa006e474>{:mptbase:mpt_do_ioc_recovery+1529}
       <ffffffff801e68c0>{mpi_tdiv_qr+1026} <ffffffff801e503a>{DSA_verify+530}
       <ffffffff801fb967>{vgacon_cursor+0}
<ffffffff8024afb7>{serial8250_console_write+553}
       <ffffffff801edc9b>{vsnprintf+1406} <ffffffff801edeba>{sprintf+136}
       <ffffffff801eb0b5>{idr_get_new+11} <ffffffff801afc33>{proc_register+157}
       <ffffffff802a9019>{pci_conf1_read+197}
<ffffffff801f25ad>{pci_bus_read_config_byte+97}
       <ffffffffa007036c>{:mptbase:mpt_attach+1891}
<ffffffffa008d81c>{:mptsas:mptsas_probe+30}
       <ffffffff801b515a>{init_dir+0} <ffffffff801b45ca>{sysfs_create+162}
       <ffffffff8019049d>{dput+56} <ffffffff801b5337>{create_dir+413}
       <ffffffff801f6660>{pci_device_probe+110} <ffffffff8024c9c9>{bus_match+57}
       <ffffffff8024cac7>{driver_attach+68} <ffffffff8024cde3>{bus_add_driver+143}
       <ffffffff801f63d0>{pci_register_driver+119}
<ffffffff801505d2>{sys_init_module+278}
       <ffffffff8011026a>{system_call+126}
                                                                               
                     
Code: 49 8b 7d 00 31 c0 48 85 ff 0f 84 04 03 00 00 48 89 f8 48 2b

Expected results:

Normal boot

Additional info:
Comment 1 Larry Woodman 2007-11-01 17:12:14 EDT
I verified that the cause of this panic is linux-2.6.9-x8664-physflat.patch.  If
I remove this patch the system boots and runs fine, with this patch the system
panics.  This patch was introduced in 2.6.9-42.18 on Wed Oct 11 2006.

Larry Woodman
Comment 2 Larry Woodman 2007-11-01 17:15:56 EDT
This could be a BIOS version issue:

-------------------------------------------------------------------------------
This patch addresses Bug 192760.  The bug as reported was that the largesmp
x86_64 kernel did not support more than 8 AMD processors.  The real issue
is that AMD supports a "physical flat" APIC mode for >8 processors.
                                                                               
                     
We had not initially included this patch because I thought that clustered
APIC mode would be sufficient for >8 processor support, and tests on
a 12-way iWill box confirmed this.  What I did not know is that AMD has
deprecated clustered APIC mode in favor of physflat, and newer BIOSes
only support the latter.
AMD has verified this patch on their in-house largesmp systems, and I
tested it successfully on the iWill box.
-------------------------------------------------------------------------------
Comment 3 Larry Woodman 2007-11-01 17:31:45 EDT
The failing systems have really old BIOS versions:

# dmidecode 2.2
SMBIOS 2.3 present.
152 structures occupying 3941 bytes.
Table at 0x000EC000.
Handle 0x0000
        DMI type 0, 20 bytes.
        BIOS Information
                Vendor: HP
                Version: A01
                Release Date: 08/26/2005
                Address: 0xF0000
                Runtime Size: 64 kB
                ROM Size: 2048 kB
Comment 4 Johnray Fuller 2007-11-01 19:15:38 EDT
I updated the BIOs (in stages) on dl585-03 from 2006.01.20 to the latest
(2007.02.14). The system continues to panic on boot with the largesmp kernel.

I also checked another dl585 with 8G of RAM and it booted fine. How much RAM is
in dl585-02?

J
Comment 5 Larry Woodman 2007-11-01 19:24:05 EDT
dl585-02 has 96GB.

Larry
Comment 6 Larry Woodman 2007-11-01 19:26:48 EDT
Johnray, the kernel that does boot is
dl585-04:/usr/src/redhat/RPMS/x86_64/kernel-largesmp-2.6.9-65.test.EL.x86_64.rpm

All I did was remove the linux-2.6.9-x8664-physflat.patch


Larry
Comment 7 Johnray Fuller 2007-11-01 20:52:49 EDT
I got a full capture of the error off of DL585-03

*****

ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
ttyS2 at I/O 0x3e8 (irq = 5) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:04.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:04.1 (rev 03) UDMA133 controller
    ide0: BM-DMA at 0x2000-0x2007, BIOS settings: hda:pio, hdb:pio
hda: TSSTcorpCD-ROM TS-L162C, ATAPI CD/DVD-ROM drive
Using cfq io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide2: I/O resource 0x3EE-0x3EE not free.
ide2: ports already in use, skipping probe
hda: ATAPI 24X CD-ROM drive, 96kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
input: PS/2 Generic Mouse on isa0060/serio1
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 262144 (order: 10, 4194304 bytes)
TCP: Hash tables configured (established 262144 bind 262144)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
powernow-k8: Found 4 AMD Opteron (tm) Processor 885 processors (8 cpu cores)
(version 2.00.00-rhel4)
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
ACPI wakeup devices: 

ACPI: (supports S0 S4 S5)
Freeing unused kernel memory: 212k freed
Red Hat nash version 4.2.1.13 starting
Mounted /proc filesystem
Mounting sysfs
Creating /dev
Starting udev
Loading scsi_mod.ko module
SCSI subsystem initialized
Loading sd_mod.ko module
Loading ehci-hcd.ko module
Loading ohci-hcd.ko module
ACPI: PCI Interrupt 0000:01:00.0[D] -> GSI 19 (level, low) -> IRQ 169
ohci_hcd 0000:01:00.0: OHCI Host Controller
ohci_hcd 0000:01:00.0: irq 169, pci mem ffffff000002e000
ohci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
ACPI: PCI Interrupt 0000:01:00.1[D] -> GSI 19 (level, low) -> IRQ 169
ohci_hcd 0000:01:00.1: OHCI Host Controller
ohci_hcd 0000:01:00.1: irq 169, pci mem ffffff0000030000
ohci_hcd 0000:01:00.1: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
Loading uhci-hcd.ko module
USB Universal Host Controller Interface driver v2.2
Loading usb-storage.ko module
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
Waiting for driver initialization.
Loading cciss.ko module
HP CISS Driver (v 2.6.16.RH1)
ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 18 (level, low) -> IRQ 193
cciss: using DAC cycles
      blocks= 71122559 block_size= 512
      heads= 255, sectors= 32, cylinders= 8716

      blocks= 71122559 block_size= 512
      heads= 255, sectors= 32, cylinders= 8716

      blocks= 71122559 block_size= 512
      heads= 255, sectors= 32, cylinders= 8716

 cciss/c0d0: p1 p2 p3 p4 < p5 p6 >
      blocks= 71122559 block_size= 512
      heads= 255, sectors= 32, cylinders= 8716

 cciss/c0d1: p1 p2 p3
Loading qla4xxx.QLogic iSCSI HBA Driver (ffffffffa007b000)
ko module
ACPI: PCI Interrupt 0000:05:0d.1[B] -> GSI 33 (level, low) -> IRQ 225
qla4xxx 0000:05:0d.1: Found an ISP4022, irq 0, iobase 0xffffff0000036000
Unable to handle kernel paging request at 0000000000006a80 RIP: 
<ffffffff8015eb2d>{__alloc_pages+117}
PML4 7e2c9d067 PGD 37d8a067 PMD 0 
Oops: 0000 [1] SMP 
CPU 1 
Modules linked in: qla4xxx cciss usb_storage uhci_hcd ohci_hcd ehci_hcd sd_mod
scsi_mod
Pid: 580, comm: insmod Not tainted 2.6.9-65.ELlargesmp
RIP: 0010:[<ffffffff8015eb2d>] <ffffffff8015eb2d>{__alloc_pages+117}
RSP: 0000:00000107e2e2bd98  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000040 RCX: 0000000000000220
RDX: 0000000000006a80 RSI: 0000000000000006 RDI: 0000000000000220
RBP: ffffffffffffffff R08: 00000000000927bf R09: 00000107e2ed03c8
R10: ffffffff8031b1a0 R11: 0000ffff804061c0 R12: 0000000000000220
R13: 0000000000006a80 R14: 0000000000000006 R15: 00000107e3f327f0
FS:  0000000000000000(0000) GS:ffffffff80506080(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000006a80 CR3: 00000007e3f88000 CR4: 00000000000006e0
Process insmod (pid: 580, threadinfo 00000107e2e2a000, task 00000107e3f327f0)
Stack: 00000100f541aac8 0000000100004022 0000000000000000 ffffff0000036000 
       00000100f57d4008 0000000000000040 ffffffffffffffff 00000100f541aa30 
       0000000000022000 0000000000000220 
Call Trace:<ffffffff801210b5>{dma_alloc_pages+125}
<ffffffff801212dc>{dma_alloc_coherent+97} 
       <ffffffffa007f527>{:qla4xxx:qla4xxx_probe_adapter+1227} 
       <ffffffff801f6660>{pci_device_probe+110} <ffffffff8024c9c9>{bus_match+57} 
       <ffffffff8024cac7>{driver_attach+68} <ffffffff8024cde3>{bus_add_driver+143} 
       <ffffffff801f63d0>{pci_register_driver+119}
<ffffffffa0098022>{:qla4xxx:qla4xxx_module_init+34} 
       <ffffffff801505d2>{sys_init_module+278} <ffffffff8011026a>{system_call+126} 
       

Code: 49 8b 7d 00 31 c0 48 85 ff 0f 84 04 03 00 00 48 89 f8 48 2b 
RIP <ffffffff8015eb2d>{__alloc_pages+117} RSP <00000107e2e2bd98>
CR2: 0000000000006a80
 <0>Kernel panic - not syncing: Oops
Comment 8 Johnray Fuller 2007-11-01 21:12:16 EDT
I tested kernel-largesmp-2.6.9-65.test.EL.x86_64.rpm on dl585-03 and it does work.

J
Comment 9 Larry Woodman 2007-11-02 12:19:20 EDT
This is a problem with memory interleaving/numa=off.  Any DL585 with
interleaving enabled and/or numa=off on the bootline will panic when booting the
largesmp kernel because it expects NUMA.  When the system is NUMA it works OK.

Larry
Comment 18 Larry Woodman 2007-11-05 16:29:11 EST
This patch fixes the problem by fixing the linux-2.6.9-x8664-physflat.patch:

--- linux-2.6.9/arch/x86_64/pci/k8-bus.c.orig
+++ linux-2.6.9/arch/x86_64/pci/k8-bus.c
@@ -59,6 +59,10 @@ fill_mp_bus_to_cpumask(void)
                }
        }
                                                                               
                 
+       for (i = 0; i < 256; i++)
+               if (cpus_empty(pci_bus_to_cpumask[i]))
+                       pci_bus_to_cpumask[i] = CPU_MASK_ALL;
+
        return 0;
 }
                                                                               
                 


Larry Woodman

Comment 19 Kenneth Topp 2007-11-05 16:54:52 EST
so just to be clear, it seems that the bios is a red herring, and the only issue
is numa setting, and this only impacts largesmp after 4u4?

And largesmp never worked on opterons>8  before 4.5?
Comment 22 Jason Baron 2007-11-06 16:24:32 EST
committed in stream U6 build 66. A test kernel with this patch is available from
http://people.redhat.com/~jbaron/rhel4/
Comment 23 Larry Woodman 2007-11-07 11:52:09 EST
Martin tested this kernel on all conbinations of numa=on/numa=off, memory
ionterleaving enabled/disabled with the UP, SMP and LARGESMP kernel and reported
that everything worked fine.

Larry 
Comment 28 Johnray Fuller 2007-11-07 21:03:36 EST
I have tested this upatched and the patched kernels on another DL585 G1 Dual
core system. Here are the results:

Memory Interleaving ON in BIOS:

2.6.9-65.EL largesmp = FAILED
2.6.9-66.EL largesmp = Success

numa=off:

2.6.9-65.EL largesmp = FAILED
2.6.9-66.EL largesmp = Success

I also tested these two kernels on a DL385 G2 and *both* kernels successfully
booted.

J
Comment 34 errata-xmlrpc 2007-11-15 11:33:41 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0791.html

Note You need to log in before you can comment on or make changes to this bug.