Bug 1337705

Summary: RHELSA kernels starting with 4.5.0-0.35.el7 hang early in boot on Gigabyte MP30-AR0
Product: Red Hat Enterprise Linux 7 Reporter: Richard W.M. Jones <rjones>
Component: kernel-aarch64Assignee: arm-mgr
kernel-aarch64 sub component: Platform Enablement QA Contact: Jeff Bastian <jbastian>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: jcm, jshortt, mlangsdo
Version: 7.3   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-aarch64-4.5.0-0.46.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-20 12:38:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
lspci -vvvv output
none
dmidecode output none

Description Richard W.M. Jones 2016-05-19 21:21:19 UTC
Description of problem:

Recent kernel-aarch64 fail to boot on the Gigabyte MP30-AR0.

All I see is:

  EFI stub: Booting Linux Kernel...
  EFI stub: Using DTB from configuration table
  EFI stub: Exiting boot services and installing virtual address map...
  L3c Cache: 8MB

and then it hangs and I have to power-cycle.

During ordinary boots the messages above only flash on the serial console
very briefly before seeing the following kernel messages:

  [    0.000000] Booting Linux on physical CPU 0x0
  [    0.000000] Linux version 4.5.0-0.34.el7.aarch64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Apr 26 15:38:40 EDT 2016

Version-Release number of selected component (if applicable):

  kernel-4.5.0-0.33.el7  -- boots OK
  kernel-4.5.0-0.34.el7  -- boots OK
  kernel-4.5.0-0.35.el7  -- hangs
  kernel-4.5.0-0.37.el7  -- hangs
  kernel-4.5.0-0.38.el7  -- hangs

How reproducible:

100%

Steps to Reproduce:
1. Install new kernel.
2. Boot.

Comment 1 Richard W.M. Jones 2016-05-19 21:22:00 UTC
Firmware version is
EFI v2.40 by X-Gene Mp30ar0 Board EFI Jan 26 2016 18:10:52

Comment 3 Richard W.M. Jones 2016-05-23 19:35:31 UTC
This Fedora bug was reported which does look rather similar:

https://bugzilla.redhat.com/show_bug.cgi?id=1323239

I will see if I can use the technique described in

https://bugzilla.redhat.com/show_bug.cgi?id=1323239#c6

to capture a panic trace (but not right now).

Comment 4 Richard W.M. Jones 2016-05-23 21:27:02 UTC
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
L3c Cache: 8MB
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.5.0-0.37.el7.aarch64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue May 17 13:12:34 EDT 2016
[    0.000000] Boot CPU: AArch64 Processor [500f0001]
[    0.000000] earlycon: Early serial console at MMIO32 0x1c020000 (options '')
[    0.000000] bootconsole [uart0] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] EFI v2.40 by X-Gene Mp30ar0 Board EFI Jan 26 2016 18:10:52
[    0.000000] efi:  ACPI=0xfffff000  ACPI 2.0=0xfffff014  SMBIOS 3.0=0x807ae10000 
[    0.000000] cma: Reserved 512 MiB at 0x00000000c0000000
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000FFFFF014 000024 (v02 APM   )
[    0.000000] ACPI: XSDT 0x00000000FFFFE0E8 00006C (v01 APM    XGENE    00000003      01000013)
[    0.000000] ACPI: FACP 0x00000000FFFF6000 00010C (v05 APM    XGENE    00000003 INTL 20140724)
[    0.000000] ACPI: DSDT 0x00000000FFFF7000 0051A6 (v05 APM    APM88xxx 00000001 INTL 20140724)
[    0.000000] ACPI: DBG2 0x00000000FFFFD000 0000AA (v00 APMC0D XGENEDBG 00000000 INTL 20140724)
[    0.000000] ACPI: GTDT 0x00000000FFFF4000 0000E0 (v02 APM    XGENE    00000001 INTL 20140724)
[    0.000000] ACPI: MCFG 0x00000000FFFF3000 00007C (v01 APM    XGENE    00000002 INTL 20140724)
[    0.000000] ACPI: SPCR 0x00000000FFFF2000 000050 (v01 APMC0D XGENESPC 00000000 INTL 20140724)
[    0.000000] ACPI: SSDT 0x00000000FFFF1000 00002D (v02 APM    XGENE    00000001 INTL 20140724)
[    0.000000] ACPI: APIC 0x00000000FFFF0000 0002A4 (v03 APM    XGENE    00000003      01000013)
[    0.000000] ACPI: SSDT 0x00000000FFFEF000 000078 (v02 REDHAT MACADDRS 00000001      01000013)
[    0.000000] ACPI: SSDT 0x00000000FFFEE000 000032 (v02 REDHAT UARTCLKS 00000001      01000013)
[    0.000000] psci: is not implemented in ACPI.
[    0.000000] ACPI-PMU: Assign CPU 0 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 1 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 2 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 3 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 4 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 5 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 6 girq 28 level 0
[    0.000000] ACPI-PMU: Assign CPU 7 girq 28 level 0
[    0.000000] PERCPU: Embedded 3 pages/cpu @ffff807fffdd0000 s107520 r8192 d80896 u196608
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 523776
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.5.0-0.37.el7.aarch64 root=/dev/mapper/rhel_harm-root ro rd.lvm.lv=rhel_harm/root rd.lvm.lv=rhel_harm/swap console=ttyS0,115200 LANG=en_GB.UTF-8 earlycon=uart8250,mmio32,0x1c020000
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 4194304 (order: 9, 33554432 bytes)
[    0.000000] Inode-cache hash table entries: 2097152 (order: 8, 16777216 bytes)
[    0.000000] software IO TLB [mem 0xfbfd0000-0xfffd0000] (64MB) mapped at [ffff80007bfd0000-ffff80007ffcffff]
[    0.000000] Memory: 32839040K/33554432K available (7232K kernel code, 1320K rwdata, 3392K rodata, 1024K init, 6672K bss, 191104K reserved, 524288K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vmalloc : 0xffff000000000000 - 0xffff77ffffff0000   (122879 GB)
[    0.000000]     vmemmap : 0xffff780000000000 - 0xffff7c0000000000   (  4096 GB maximum)
[    0.000000]               0xffff780000000000 - 0xffff780020000000   (   512 MB actual)
[    0.000000]     fixed   : 0xffff7ffffa7d0000 - 0xffff7ffffac00000   (  4288 KB)
[    0.000000]     PCI I/O : 0xffff7ffffae00000 - 0xffff7ffffbe00000   (    16 MB)
[    0.000000]     modules : 0xffff7ffffc000000 - 0xffff800000000000   (    64 MB)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff808000000000   (524288 MB)
[    0.000000]       .init : 0xffff800000af0000 - 0xffff800000bf0000   (  1024 KB)
[    0.000000]       .text : 0xffff800000080000 - 0xffff800000af0000   ( 10688 KB)
[    0.000000]       .data : 0xffff800000bf0000 - 0xffff800000d3a000   (  1320 KB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=4096 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GIC: Adjusting CPU interface base to 0x780af000
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] No distributor detected at @ffff000000080000, giving up
[    0.000000] Architected cp15 timer(s) running at 50.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000002] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.008544] Console: colour dummy device 80x25
[    0.013253] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=50000)
[    0.024209] pid_max: default: 32768 minimum: 301
[    0.029109] ACPI: Core revision 20160108
[    0.038179] ACPI: 4 ACPI AML tables successfully acquired and loaded
[    0.044903] 
[    0.046493] Security Framework initialized
[    0.050830] SELinux:  Initializing.
[    0.054582] Mount-cache hash table entries: 65536 (order: 3, 524288 bytes)
[    0.061856] Mountpoint-cache hash table entries: 65536 (order: 3, 524288 bytes)
[    0.071372] ftrace: allocating 26787 entries in 7 pages
[    0.096948] ASID allocator initialised with 65536 entries
[    0.103307] Remapping and enabling EFI services.
[    0.108222]   EFI remap 0x0000000010510000 => 0000000020000000
[    0.114380]   EFI remap 0x0000000010548000 => 0000000020018000
[    0.120545]   EFI remap 0x0000000017000000 => 0000000020020000
[    0.126727]   EFI remap 0x000000001c025000 => 0000000020035000
[    0.132900]   EFI remap 0x000000807ae0d000 => 000000002004d000
[    0.139083]   EFI remap 0x000000807ae11000 => 0000000020051000
[    0.145247]   EFI remap 0x000000807ffb9000 => 0000000020249000
[    0.151412]   EFI remap 0x000000807ffcd000 => 000000002025d000
[    0.157843] Detected PIPT I-cache on CPU1
[    0.157864] CPU1: Booted secondary processor [500f0001]
[    0.158010] Detected PIPT I-cache on CPU2
[    0.158030] CPU2: Booted secondary processor [500f0001]
[    0.158164] Detected PIPT I-cache on CPU3
[    0.158175] CPU3: Booted secondary processor [500f0001]
[    0.158309] Detected PIPT I-cache on CPU4
[    0.158329] CPU4: Booted secondary processor [500f0001]
[    0.158456] Detected PIPT I-cache on CPU5
[    0.158468] CPU5: Booted secondary processor [500f0001]
[    0.158601] Detected PIPT I-cache on CPU6
[    0.158620] CPU6: Booted secondary processor [500f0001]
[    0.158750] Detected PIPT I-cache on CPU7
[    0.158762] CPU7: Booted secondary processor [500f0001]
[    0.158787] Brought up 8 CPUs
[    0.230037] SMP: Total of 8 processors activated.
[    0.235019] CPU: All CPU(s) started at EL2
[    0.239855] devtmpfs: initialized
[    0.243681] SMBIOS 3.0.0 present.
[    0.247269] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.258055] atomic64_test: passed
[    0.261784] NET: Registered protocol family 16
[    0.266688] PCCT header not found.
[    0.270298] vdso: 2 pages (1 code @ ffff800000c10000, 1 data @ ffff800000c00000)
[    0.278145] hw-breakpoint: found 4 breakpoint and 4 watchpoint registers.
[    0.285607] DMA: preallocated 256 KiB pool for atomic allocations
[    0.292041] ACPI: bus type PCI registered
[    0.296272] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.303089] ACPI: IORT: Failed to get table, AE_NOT_FOUND
[    0.308864] Serial: AMBA PL011 UART driver
[    0.313282] ACPI-PMU: Setting up 8 PMUs for CPU type 0
[    0.325856] HugeTLB registered 512 MB page size, pre-allocated 0 pages
[    0.333085] ACPI: Added _OSI(Module Device)
[    0.337504] ACPI: Added _OSI(Processor Device)
[    0.342184] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.347166] ACPI: Added _OSI(Processor Aggregator Device)
[    0.359870] ACPI: Interpreter enabled
[    0.363745] ACPI: Using GIC for interrupt routing
[    0.368680] ACPI: MCFG entry for domain 0000 [bus 00-ff] (base 0x000000e0d0000000)
[    0.376704] ACPI: MCFG entry for domain 0001 [bus 00-ff] (base 0x000000d0d0000000)
[    0.384693] ACPI: MCFG entry for domain 0002 [bus 00-ff] (base 0x00000090d0000000)
[    0.392681] ACPI: MCFG entry for domain 0003 [bus 00-ff] (base 0x000000a0d0000000)
[    0.400694] ACPI: MCFG entry for domain 0004 [bus 00-ff] (base 0x000000c0d0000000)
[    0.408708] ACPI: MCFG table loaded, 5 entries
[    0.415787] ACPI: Power Resource [SCVR] (off)
[    0.428114] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.434691] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.443547] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[    0.452082] XGene PCIe firmware set bridge bus range to 010100, expected ff0100
[    0.459839] Remapped I/O 0x000000e010000000 to [io  0x0000-0xffff window]
[    0.467133] acpi PNP0A08:00: PCI host bridge to bus 0000:00
[    0.473044] pci_bus 0000:00: root bus resource [mem 0x1f2b0000-0x1f2bffff]
[    0.480282] pci_bus 0000:00: root bus resource [io  0x0000-0xffff window]
[    0.487460] pci_bus 0000:00: root bus resource [mem 0xe140000000-0xe17fffffff window] (bus address [0x40000000-0x7fffffff])
[    0.499268] pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff window]
[    0.507525] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.513580] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.519167] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 00-ff])
[    0.525728] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.534622] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[    0.543202] Remapped I/O 0x0000009010000000 to [io  0x10000-0x1ffff window]
[    0.550708] acpi PNP0A08:01: PCI host bridge to bus 0002:00
[    0.556593] pci_bus 0002:00: root bus resource [mem 0x1f2d0000-0x1f2dffff]
[    0.563815] pci_bus 0002:00: root bus resource [io  0x10000-0x1ffff window] (bus address [0x10000000-0x1000ffff])
[    0.574664] pci_bus 0002:00: root bus resource [mem 0x9030000000-0x90afffffff window] (bus address [0x30000000-0xafffffff])
[    0.586402] pci_bus 0002:00: root bus resource [mem 0x9100000000-0x917fffffff window]
[    0.594685] pci_bus 0002:00: root bus resource [bus 00-ff]
[    0.600721] Unhandled fault: synchronous external abort (0x96000010) at 0xffff000005040000
[    0.609478] Internal error: : 96000010 [#1] SMP
[    0.614258] Modules linked in:
[    0.617478] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.5.0-0.37.el7.aarch64 #1
[    0.625213] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    0.632922] task: ffff807fe4020000 ti: ffff807fe0080000 task.ti: ffff807fe0080000
[    0.640833] PC is at pci_generic_config_read+0x5c/0xd0
[    0.646236] LR is at pci_generic_config_read+0x40/0xd0
[    0.651672] pc : [<ffff8000003ff7f4>] lr : [<ffff8000003ff7d8>] pstate: 200000c5
[    0.659485] sp : ffff807fe00838a0
[    0.662985] x29: ffff807fe00838a0 x28: ffff800000c43000 
[    0.668604] x27: ffff807ffac1da00 x26: ffff800000a5c065 
[    0.674214] x25: ffff807ffac1d618 x24: 0000000000000040 
[    0.679823] x23: 0000000000000040 x22: ffff800001396fc8 
[    0.685450] x21: 0000000000000040 x20: ffff807fe00839bc 
[    0.691042] x19: ffff807fe03fd800 x18: 000000000000001c 
[    0.696634] x17: 000000000000000a x16: 000000000000047f 
[    0.702244] x15: ffffffffffffffff x14: 7469623436206666 
[    0.707836] x13: 6666666666663038 x12: 0000000000000038 
[    0.713445] x11: 0101010101010101 x10: ffff807fffdec4e0 
[    0.719020] x9 : ff72646268756463 x8 : 0000007fff210000 
[    0.724647] x7 : 00000000000000ff x6 : 000000000000000c 
[    0.730230] x5 : 0000000000000000 x4 : ffff807fe008393c 
[    0.735831] x3 : 0000000000000004 x2 : 0000000000040000 
[    0.741440] x1 : 0000000000040000 x0 : ffff000005040000 
[    0.747041] 
[    0.748572] Process swapper/0 (pid: 1, stack limit = 0xffff807fe0080020)
[    0.755651] Stack: (0xffff807fe00838a0 to 0xffff807fe0084000)
[    0.761719] 38a0: ffff807fe00838e0 ffff8000003ffd10 ffff807fe03fd800 ffff8000003ffd28
[    0.769982] 38c0: ffff807fe008393c 0000000000000004 0000000000000000 0000000000000040
[    0.778262] 38e0: ffff807fe0083940 ffff800000401c68 0000000000000000 ffff807fe00839bc
[    0.786559] 3900: 000000000000ea60 ffff807ffac1d618 ffff807fe03fd800 ffff800000d2d000
[    0.794847] 3920: ffff807fe0083930 0000000000000000 ffff807fe0083950 000000000040d208
[    0.803135] 3940: ffff807fe0083980 ffff800000403740 0000000000000000 ffff807fe03fd800
[    0.811431] 3960: 0000000000000040 ffff807ffac1d618 0000000000000000 ffff800000d2d000
[    0.819702] 3980: ffff807fe00839c0 ffff800000403814 ffff807fe03fd800 ffff807fe03fd800
[    0.827972] 39a0: 0000000000000040 ffff807ffac1d580 ffff800000ce2ee8 00000000fac1d618
[    0.836269] 39c0: ffff807fe0083a00 ffff8000004047e8 0000000000000040 ffff807fe03fd800
[    0.844566] 39e0: ffff800000ce2ee8 ffff800000c43000 ffff807ffac1d680 ffff800000a14e48
[    0.852862] 3a00: ffff807fe0083a40 ffff800000453974 ffff807ffac1d600 ffff807fe03fd800
[    0.861167] 3a20: ffff800000ce2ee8 ffff807ffac1d618 0000000000000000 ffff807fe03f6000
[    0.869446] 3a40: ffff807fe0083ab0 ffff8000004778d4 ffff807ffac1d580 00000000000000ff
[    0.877734] 3a60: 0000000000000000 0000000000000000 0000000000000002 ffff807ffac1d600
[    0.886014] 3a80: ffff800000ce2eb0 ffff800000a85880 0000000000000000 0000000000000002
[    0.894302] 3aa0: ffff807ffac1dba0 ffff807ffac1d680 ffff807fe0083b20 ffff800000453524
[    0.902589] 3ac0: ffff807ffac1d580 ffff807fe03f6000 0000000000000000 ffff807fe03f6268
[    0.910886] 3ae0: ffff807fe03f6000 000000000000001d 0000000000000000 ffff800000ce1b10
[    0.919148] 3b00: ffff807fe8016f40 000000000000001f 000000000000001d ffff807fe03f6268
[    0.927445] 3b20: ffff807fe0083bb0 ffff80000044ee08 ffff807fe03f6000 ffff800000ce1c00
[    0.935742] 3b40: ffff807fe03ce600 ffff807fe03f6090 ffff800000ce1000 ffff800000ce1878
[    0.944021] 3b60: ffff807fe03ce800 00000000000000dc ffff800000d60000 0000000000000000
[    0.952309] 3b80: ffff807fe0083bb0 000000000044edc8 0000000000000002 ffff800000ce1c00
[    0.960580] 3ba0: 0000001f0000001d 0000000000000000 ffff807fe0083c10 ffff80000044ee8c
[    0.968868] 3bc0: ffff807ffbd1a800 ffff807fe03f6000 ffff807ffbd1a828 ffff807ffbd1a890
[    0.977147] 3be0: ffff800000ce1000 ffff800000ce1878 ffff800000a5b12b ffff80000044ee44
[    0.985435] 3c00: ffff807ffbd1a800 ffff800000803270 ffff807fe0083c70 ffff80000044ee8c
[    0.993731] 3c20: ffff807ffbd1a000 ffff807ffbd1a800 ffff807ffbd1a028 ffff807ffbd1a090
[    1.002011] 3c40: ffff800000ce1000 ffff800000ce1878 ffff800000a5b0c2 ffff80000044ee44
[    1.010299] 3c60: ffff807ffbd1a000 000000000000000f ffff807fe0083cd0 ffff80000044efdc
[    1.018595] 3c80: ffffffffffffffff ffff800000ce1000 ffff800001399000 ffff800000a62000
[    1.026891] 3ca0: ffff800000c3e000 ffff800000ad7928 ffff800000b40800 00000000000000dc
[    1.035180] 3cc0: ffff800000d60000 000000000000000f ffff807fe0083d10 ffff800000b24a58
[    1.043424] 3ce0: ffff800000ce1760 ffff800000776f14 ffff807fe0083d10 ffff807fe0083d08
[    1.051704] 3d00: ffff800000ce1760 ffff807ffbd1a000 ffff807fe0083d50 ffff800000b24814
[    1.059983] 3d20: 0000000000000000 ffff800001399d60 ffff800001399000 ffff800000a62000
[    1.068279] 3d40: ffff807fe0083d50 ffff800000b24810 ffff807fe0083dd0 ffff80000009233c
[    1.076559] 3d60: ffff800000c31ea0 ffff807fec12fc00 ffff800000c31ea0 ffff800000b2458c
[    1.084855] 3d80: 0000000000000000 ffff807fe0083dd0 0000000e00000001 ffff807fe8010050
[    1.093143] 3da0: ffff800000ce1700 0000000000000001 0000000000000008 ffff807fe0083d90
[    1.101431] 3dc0: ffffffffffffffff 0000000000000000 ffff807fe0083e50 ffff800000af0b88
[    1.109693] 3de0: ffff800000bb7a00 ffff800000b40750 0000000000000004 ffff800000d60000
[    1.117990] 3e00: ffff800000af0278 0000000000000000 ffff807ffff77200 ffff800000d2a050
[    1.126295] 3e20: 0000000000000000 0000000400000004 ffff800000d29fd8 0000000000000000
[    1.134591] 3e40: ffff800000af0278 ffff800000ad7928 ffff807fe0083eb0 ffff800000773b6c
[    1.142870] 3e60: ffff800000773b54 0000000000000000 0000000000000000 0000000000000000
[    1.151167] 3e80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.159464] 3ea0: 0000000000000000 fffffe7fe8060000 0000000000000000 ffff8000000919c0
[    1.167734] 3ec0: ffff800000773b54 0000000000000000 0000000000000000 0000000000000000
[    1.176014] 3ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.184284] 3f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.192581] 3f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.200877] 3f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.209130] 3f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.217427] 3f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.225724] 3fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.234012] 3fc0: 0000000000000000 0000000000000000 0000000000000000 0000000000000005
[    1.242309] 3fe0: 0000000000000000 0000000000000000 effca7d43941edfd ccf619bdf3537c1d
[    1.250605] Call trace:
[    1.253165] Exception stack(0xffff807fe00836e0 to 0xffff807fe0083800)
[    1.259993] 36e0: ffff807fe03fd800 ffff807fe00839bc ffff807fe00838a0 ffff8000003ff7f4
[    1.268281] 3700: 0000000000000007 ffff800000000000 ffff000005040000 ffffffffffffffff
[    1.276578] 3720: ffff807fe0083850 ffff807f30333635 ffff807fe0083740 ffff800000a0d3e2
[    1.284857] 3740: ffff807fe00837c0 ffff8000003be5d8 ffff807ff7d8b000 0000000000000800
[    1.293162] 3760: ffff807ff7d8a0a8 ffff800000cd9e68 ffff8000007f1000 0000000000000000
[    1.301450] 3780: ffff000005040000 0000000000040000 0000000000040000 0000000000000004
[    1.309755] 37a0: ffff807fe008393c 0000000000000000 000000000000000c 00000000000000ff
[    1.318035] 37c0: 0000007fff210000 ff72646268756463 ffff807fffdec4e0 0101010101010101
[    1.326323] 37e0: 0000000000000038 6666666666663038 7469623436206666 ffffffffffffffff
[    1.334611] [<ffff8000003ff7f4>] pci_generic_config_read+0x5c/0xd0
[    1.341164] [<ffff8000003ffd10>] pci_bus_read_config_dword+0x88/0xb8
[    1.347880] [<ffff800000401c68>] pci_bus_read_dev_vendor_id+0x40/0x108
[    1.354796] [<ffff800000403740>] pci_scan_single_device+0x60/0xd4
[    1.361244] [<ffff800000403814>] pci_scan_slot+0x60/0x118
[    1.366967] [<ffff8000004047e8>] pci_scan_child_bus+0x48/0x15c
[    1.373122] [<ffff800000453974>] acpi_pci_root_create+0x164/0x1c4
[    1.379554] [<ffff8000004778d4>] pci_acpi_scan_root+0x25c/0x2b0
[    1.385803] [<ffff800000453524>] acpi_pci_root_add+0x38c/0x454
[    1.391966] [<ffff80000044ee08>] acpi_bus_attach+0x130/0x1ec
[    1.397947] [<ffff80000044ee8c>] acpi_bus_attach+0x1b4/0x1ec
[    1.403937] [<ffff80000044ee8c>] acpi_bus_attach+0x1b4/0x1ec
[    1.409910] [<ffff80000044efdc>] acpi_bus_scan+0x68/0x80
[    1.415537] [<ffff800000b24a58>] acpi_scan_init+0x54/0x184
[    1.421328] [<ffff800000b24814>] acpi_init+0x288/0x2ac
[    1.426757] [<ffff80000009233c>] do_one_initcall+0xcc/0x1c0
[    1.432662] [<ffff800000af0b88>] kernel_init_freeable+0x1ac/0x24c
[    1.439094] [<ffff800000773b6c>] kernel_init+0x18/0xec
[    1.444522] [<ffff8000000919c0>] ret_from_fork+0x10/0x50
[    1.450132] Code: 7100047f 54000300 7100087f 540001c0 (b9400001) 
[    1.456596] ---[ end trace 64d0fd70ffcc4d63 ]---
[    1.461484] Kernel panic - not syncing: Fatal exception
[    1.467011] CPU2: stopping
[    1.469852] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    1.480144] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    1.487887] Call trace:
[    1.490450] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    1.496173] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    1.501526] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    1.506868] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    1.512391] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    1.518096] Exception stack(0xffff807fffe33fd0 to 0xffff807fffe340f0)
[    1.524924] 3fc0:                                   ffff807fe00afe10 ffff800000d26000
[    1.533221] 3fe0: ffff807fe00aff30 ffff80000009411c 0000000060000045 ffff807fe00ac000
[    1.541526] 4000: ffff807fe00aff30 ffff807fe00afe10 0000000000000000 0000000000000000
[    1.549797] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.558076] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.566355] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.574634] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.582940] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.591236] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.599533] 40e0: 0000000000000000 0000000000000000
[    1.604693] [<ffff800000091520>] el1_irq+0xa0/0x100
[    1.609837] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    1.615543] [<ffff800000108660>] default_idle_call+0x20/0x30
[    1.621524] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    1.627704] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    1.634325] [<000000008009218c>] 0x8009218c
[    1.638750] CPU1: stopping
[    1.641606] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    1.651908] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    1.659624] Call trace:
[    1.662204] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    1.667918] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    1.673252] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    1.678603] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    1.684135] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    1.689839] Exception stack(0xffff807fffe03fd0 to 0xffff807fffe040f0)
[    1.696641] 3fc0:                                   ffff807fe00abe10 ffff800000d26000
[    1.704930] 3fe0: ffff807fe00abf30 ffff80000009411c 0000000060000045 ffff807fe00a8000
[    1.713227] 4000: ffff807fe00abf30 ffff807fe00abe10 0000000000000000 0000000000000000
[    1.721497] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.729803] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.738090] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.746387] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.754666] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.762946] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.771216] 40e0: 0000000000000000 0000000000000000
[    1.776359] [<ffff800000091520>] el1_irq+0xa0/0x100
[    1.781511] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    1.787217] [<ffff800000108660>] default_idle_call+0x20/0x30
[    1.793206] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    1.799360] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    1.805972] [<000000008009218c>] 0x8009218c
[    1.810380] CPU4: stopping
[    1.813229] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    1.823521] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    1.831264] Call trace:
[    1.833853] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    1.839559] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    1.844910] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    1.850243] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    1.855776] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    1.861489] Exception stack(0xffff807fffe93fd0 to 0xffff807fffe940f0)
[    1.868291] 3fc0:                                   ffff807fe00b7e10 ffff800000d26000
[    1.876596] 3fe0: ffff807fe00b7f30 ffff80000009411c 0000000060000045 ffff807fe00b4000
[    1.884868] 4000: ffff807fe00b7f30 ffff807fe00b7e10 0000000000000000 0000000000000000
[    1.893139] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.901418] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.909697] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.917985] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.926290] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.934587] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.942883] 40e0: 0000000000000000 0000000000000000
[    1.948026] [<ffff800000091520>] el1_irq+0xa0/0x100
[    1.953170] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    1.958884] [<ffff800000108660>] default_idle_call+0x20/0x30
[    1.964865] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    1.971028] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    1.977648] [<000000008009218c>] 0x8009218c
[    1.982074] CPU5: stopping
[    1.984929] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    1.995239] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    2.002973] Call trace:
[    2.005560] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    2.011257] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    2.016600] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    2.021950] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    2.027465] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    2.033169] Exception stack(0xffff807fffec3fd0 to 0xffff807fffec40f0)
[    2.039980] 3fc0:                                   ffff807fe00bbe10 ffff800000d26000
[    2.048268] 3fe0: ffff807fe00bbf30 ffff80000009411c 0000000060000045 ffff807fe00b8000
[    2.056557] 4000: ffff807fe00bbf30 ffff807fe00bbe10 0000000000000000 0000000000000000
[    2.064853] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.073132] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.081420] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.089717] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.098005] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.106302] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.114590] 40e0: 0000000000000000 0000000000000000
[    2.119716] [<ffff800000091520>] el1_irq+0xa0/0x100
[    2.124868] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    2.130573] [<ffff800000108660>] default_idle_call+0x20/0x30
[    2.136546] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    2.142709] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    2.149346] [<000000008009218c>] 0x8009218c
[    2.153763] CPU6: stopping
[    2.156621] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    2.166939] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    2.174656] Call trace:
[    2.177236] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    2.182933] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    2.188268] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    2.193601] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    2.199116] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    2.204820] Exception stack(0xffff807fffef3fd0 to 0xffff807fffef40f0)
[    2.211648] 3fc0:                                   ffff807fe00bfe10 ffff800000d26000
[    2.219954] 3fe0: ffff807fe00bff30 ffff80000009411c 0000000060000045 ffff807fe00bc000
[    2.228251] 4000: ffff807fe00bff30 ffff807fe00bfe10 0000000000000000 0000000000000000
[    2.236539] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.244836] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.253115] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.261403] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.269682] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.277979] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.286258] 40e0: 0000000000000000 0000000000000000
[    2.291401] [<ffff800000091520>] el1_irq+0xa0/0x100
[    2.296545] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    2.302259] [<ffff800000108660>] default_idle_call+0x20/0x30
[    2.308241] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    2.314403] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    2.321024] [<000000008009218c>] 0x8009218c
[    2.325441] CPU7: stopping
[    2.328296] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    2.338597] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    2.346323] Call trace:
[    2.348902] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    2.354608] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    2.359950] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    2.365275] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    2.370806] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    2.376511] Exception stack(0xffff807ffff23fd0 to 0xffff807ffff240f0)
[    2.383339] 3fc0:                                   ffff807fe00c3e10 ffff800000d26000
[    2.391645] 3fe0: ffff807fe00c3f30 ffff80000009411c 0000000060000045 ffff807fe00c0000
[    2.399924] 4000: ffff807fe00c3f30 ffff807fe00c3e10 0000000000000000 0000000000000000
[    2.408204] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.416500] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.424780] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.433059] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.441338] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.449626] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.457905] 40e0: 0000000000000000 0000000000000000
[    2.463048] [<ffff800000091520>] el1_irq+0xa0/0x100
[    2.468200] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    2.473914] [<ffff800000108660>] default_idle_call+0x20/0x30
[    2.479895] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    2.486066] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    2.492687] [<000000008009218c>] 0x8009218c
[    2.497087] CPU3: stopping
[    2.499925] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D        ------------   4.5.0-0.37.el7.aarch64 #1
[    2.510261] Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Jan 26 2016
[    2.518004] Call trace:
[    2.520583] [<ffff800000096fc0>] dump_backtrace+0x0/0x200
[    2.526306] [<ffff8000000971e4>] show_stack+0x24/0x2c
[    2.531631] [<ffff8000003baa54>] dump_stack+0x8c/0xb0
[    2.536964] [<ffff80000009c9ac>] handle_IPI+0x274/0x288
[    2.542479] [<ffff8000000904c8>] gic_handle_irq+0xa0/0xb8
[    2.548192] Exception stack(0xffff807fffe63fd0 to 0xffff807fffe640f0)
[    2.555003] 3fc0:                                   ffff807fe00b3e10 ffff800000d26000
[    2.563300] 3fe0: ffff807fe00b3f30 ffff80000009411c 0000000060000045 ffff807fe00b0000
[    2.571579] 4000: ffff807fe00b3f30 ffff807fe00b3e10 0000000000000000 0000000000000000
[    2.579859] 4020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.588138] 4040: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.596408] 4060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.604697] 4080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.612967] 40a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.621264] 40c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    2.629543] 40e0: 0000000000000000 0000000000000000
[    2.634686] [<ffff800000091520>] el1_irq+0xa0/0x100
[    2.639829] [<ffff80000009411c>] arch_cpu_idle+0x3c/0x110
[    2.645543] [<ffff800000108660>] default_idle_call+0x20/0x30
[    2.651525] [<ffff8000001087b4>] cpu_startup_entry+0x144/0x238
[    2.657696] [<ffff80000009c34c>] secondary_start_kernel+0x128/0x134
[    2.664316] [<000000008009218c>] 0x8009218c
[    2.668709] ---[ end Kernel panic - not syncing: Fatal exception

Comment 5 Richard W.M. Jones 2016-06-13 20:52:40 UTC
Bad news: 4.5.0-0.40.el7.aarch64 is still affected by this bug.

Good news: You can use acpi=off as a workaround, so it's an ACPI
problem of some kind (likely a firmware problem).

Comment 6 Jon Masters 2016-07-12 18:58:29 UTC
I think it's likely one of two things:

1). Either the system isn't doing PCI config cycles correctly (can you post the system name as listed in dmidecode?) due to the way we were matching platform for PCIe quirks (which explains acpi=off maybe working).

2). Or there's something on the PCIe bus (there's a BMC, with a VGA) and we've turned on support for the AST VGA which isn't configuring correctly. Possibly also due to 1 above. But maybe we only turned that on at the point you saw failure.

Can you start by sending me dmidecode and lspci?

Jon.

Comment 7 Richard W.M. Jones 2016-07-12 19:55:20 UTC
Created attachment 1178986 [details]
lspci -vvvv output

Please note, this is when running the -34 kernel (last working kernel
without using acpi=off hack).

Comment 8 Richard W.M. Jones 2016-07-12 19:56:20 UTC
Created attachment 1178987 [details]
dmidecode output

Please note, this is when running the -34 kernel (last working kernel
without using acpi=off hack).

Comment 9 Mark Langsdorf 2016-07-19 19:07:37 UTC
The new -0.45 and -0.46 kernels enable the APM PCI quirk using a different DMI table that should be more friendly to secondary vendors like Gigabyte. Can we get those kernels tested on this box?

Comment 10 Richard W.M. Jones 2016-07-19 20:05:21 UTC
No problem at all.  I tested kernel-aarch64-4.5.0-0.46.el7
on the Gigabyte MP30-AR0, booting without any special workarounds
and I can confirm that it boots correctly, and there are
no other issues that I could see.

Comment 11 Mark Langsdorf 2016-07-20 12:32:54 UTC
Okay. I'm going to suggest the problem was a failure to properly enable the APM PCI quirk due to DMI differences between APM and Gigabyte, and that the issue was resolved with the new DMI quirk detection code in -0.45.

Do we want to move this to ON_QA or closed?

Comment 12 Richard W.M. Jones 2016-07-20 12:38:10 UTC
Let's close it.  We don't have the machine available for QA to test
in any case.  I'll keep an eye on things and reopen the bug if
anything regresses.