Bug 1898161

Summary: aarch64: kernel panic booting thunderx1 platforms with acpi
Product: [Fedora] Fedora Reporter: Mark Salter <msalter>
Component: kernelAssignee: Mark Salter <msalter>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: acaringi, adscvr, airlied, bskeggs, hdegoede, itamar, jarodwilson, jeremy, jglisse, jonathan, josef, kernel-maint, lgoncalv, linville, masami256, mchehab, mjg59, pbrobinson, ptalbert, pwhalen, steved
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-10 17:56:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418    

Description Mark Salter 2020-11-16 14:57:03 UTC 
1. Please describe the problem:

The following splat is seen while booting thunderx1 platforms using acpi:

[    7.927837] ACPI GTDT: [Firmware Bug]: failed to get the Watchdog base address. 
[    7.935411] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028 
[    7.944184] Mem abort info: 
[    7.946967]   ESR = 0x96000005 
[    7.950010]   EC = 0x25: DABT (current EL), IL = 32 bits 
[    7.955310]   SET = 0, FnV = 0 
[    7.958353]   EA = 0, S1PTW = 0 
[    7.961482] Data abort info: 
[    7.964352]   ISV = 0, ISS = 0x00000005 
[    7.968176]   CM = 0, WnR = 0 
[    7.971132] [0000000000000028] user address but active_mm is swapper 
[    7.977475] Internal error: Oops: 96000005 [#1] SMP 
[    7.982342] Modules linked in: 
[    7.985390] CPU: 15 PID: 1 Comm: swapper/0 Not tainted 5.10.0-0.rc3.68.eln105.aarch64 #1 
[    7.993468] Hardware name: GIGABYTE R120-T34-00/MT30-GS2-00, BIOS F02 08/06/2019 
[    8.000852] pstate: 20000085 (nzCv daIf -PAN -UAO -TCO BTYPE=--) 
[    8.006856] pc : __ipi_send_mask+0x60/0x114 
[    8.011032] lr : smp_cross_call+0x3c/0xd0 
[    8.015030] sp : ffff8000128efa00 
[    8.018333] x29: ffff8000128efa00 x28: 0000000000000000  
[    8.023638] x27: ffff8000110e0538 x26: ffff800011171960  
[    8.028942] x25: 0000000000000006 x24: 0000000000000000  
[    8.034245] x23: ffff8000115e1000 x22: ffff800011998188  
[    8.039550] x21: ffff800010c0e0f0 x20: ffff800010c0e0f0  
[    8.044854] x19: ffff000100022c20 x18: 00000000fffffffe  
[    8.050158] x17: 0000000071254848 x16: 000000001055a9cd  
[    8.055461] x15: 0000000000000020 x14: ffffffffffffffff  
[    8.060765] x13: ffff8000928efc68 x12: 0000000000000018  
[    8.066069] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f  
[    8.071372] x9 : ffff80001002761c x8 : 00000000000004e4  
[    8.076676] x7 : 000003f02abd11d0 x6 : 00000000ffffffff  
[    8.081980] x5 : ffff800010f9d800 x4 : ffff800010f9d8c0  
[    8.087284] x3 : ffff800010c0daf0 x2 : 0000000000000000  
[    8.092588] x1 : ffff800010c0e0f0 x0 : 0000000000000000  
[    8.097892] Call trace: 
[    8.100330]  __ipi_send_mask+0x60/0x114 
[    8.104155]  smp_cross_call+0x3c/0xd0 
[    8.107807]  smp_send_reschedule+0x3c/0x50 
[    8.111894]  resched_curr+0x70/0xb0 
[    8.115372]  check_preempt_curr+0x58/0x90 
[    8.119370]  ttwu_do_wakeup+0x2c/0x194 
[    8.123109]  try_to_wake_up+0x248/0x5b0 
[    8.126934]  wake_up_process+0x24/0x30 
[    8.130676]  devtmpfs_submit_req+0x90/0xe0 
[    8.134761]  devtmpfs_create_node+0xa8/0xec 
[    8.138934]  device_add+0x474/0x4a0 
[    8.142412]  device_create+0x134/0x16c 
[    8.146153]  unix98_pty_init+0x214/0x22c 
[    8.150065]  pty_init+0x1c/0x2c 
[    8.153196]  do_one_initcall+0x50/0x280 
[    8.157021]  do_initcalls+0x104/0x144 
[    8.160673]  kernel_init_freeable+0x168/0x1c0 
[    8.165019]  kernel_init+0x20/0x134 
[    8.168498]  ret_from_fork+0x10/0x18 
[    8.172066] Code: a90363f7 aa0103f5 d000a557 f9401260 (b9402800)  
[    8.178173] ---[ end trace e117d9bdf84db237 ]--- 
[    8.182780] Kernel panic - not syncing: Oops: Fatal exception 
[    8.188528] SMP: stopping secondary CPUs 
[    9.242445] SMP: failed to stop secondary CPUs 2,15 
[    9.247311] Kernel Offset: disabled 
[    9.250788] CPU features: 0x0040002,69101108 
[    9.255046] Memory Limit: none 
[    9.258106] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]--- 


2. What is the Version-Release number of the kernel:

Any Fedora kernel based on upstream v5.10. I bisected this to
commit 64b499d8df40 "irqchip/gic-v3: Configure SGIs as standard interrupts"

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :


4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

100% reproducible. Boot v5.10 based kernel with acpi=on

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:


6. Are you running any modules that not shipped with directly Fedora's kernel?:


7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.
Comment 1 Mark Salter 2020-11-16 15:01:01 UTC 
The problem seems to be rooted in the firmware. The watchdog timer interrupt is 0 which corresponds to an IPI after commit 64b499d8df40.  The zero GSI for the watchdog is clearly wrong but was harmless (except for non-functioning WDT) until now when it break IPI0.
Comment 2 Mark Salter 2021-03-10 17:56:04 UTC 
As noted in comment 1 this is a firmware issue. There is a workaround with a command line option, so I see no need to clutter up the kernel code with some sort of quirk for EOL platforms. So to work around this, add "initcall_blacklist=gtdt_sbsa_gwdt_init" to the kernel command line.