Bug 2134177

Summary: kernel crash on a number of rockchip rk3399 devices on boot
Product: [Fedora] Fedora Reporter: Peter Robinson <pbrobinson>
Component: kernelAssignee: Peter Robinson <pbrobinson>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 37CC: acaringi, adscvr, airlied, alciregi, awilliam, bskeggs, dan, hdegoede, hpa, jarodwilson, jglisse, jonathan, josef, jritter, kernel-maint, lgoncalv, linville, masami256, mchehab, ptalbert, pwhalen, robatino, steved, wcohen
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedBlocker
Fixed In Version: kernel-5.19.15-201.fc36 kernel-5.19.15-101.fc35 kernel-5.19.15-301.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-17 22:55:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 2009539    

Description Peter Robinson 2022-10-12 17:26:31 UTC 
I'm seeing a crash on one or two of my rockchip rk3399 devices and I believe others are possibly seeing it too but the reports aren't detailed [1] 

[   11.996724] Unable to handle kernel execute from non-executable memory at virtual address 0000000000000000
[   11.997604] Mem abort info:
[   11.997858]   ESR = 0x0000000086000004
[   11.998194]   EC = 0x21: IABT (current EL), IL = 32 bits
[   11.998670]   SET = 0, FnV = 0
[   11.998946]   EA = 0, S1PTW = 0
[   11.999230]   FSC = 0x04: level 0 translation fault
[   11.999666] user pgtable: 4k pages, 48-bit VAs, pgdp=00000000546b3000
[   12.000240] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[   12.000852] Internal error: Oops: 86000004 [#1] SMP
[   12.001292] Modules linked in: xhci_plat_hcd dwc3 udc_core ulpi mmc_block dw_hdmi_i2s_audio dw_hdmi_cec governor_simpleondemand panfrost crct10dif_ce ghash_ce dwc3_of_simple gpu_sched roc
kchipdrm phy_rockchip_emmc phy_rockchip_inno_usb2 dw_mipi_dsi dw_hdmi rtc_rk808 analogix_dp dw_wdt phy_rockchip_typec pl330 io_domain drm_display_helper pwm_rockchip dw_mmc_rockchip sdhci_of
_arasan dw_mmc_pltfm sdhci_pltfm spi_rockchip cqhci cec sdhci dw_mmc ohci_platform cpufreq_dt ohci_hcd drm_cma_helper ehci_platform scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[   12.005729] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.19.13-300.fc37.aarch64 #1
[   12.006398] Hardware name: vamrs,rock960 96boards Rock960/96boards Rock960, BIOS 2022.10 10/01/2022
[   12.007198] pstate: 004000c5 (nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   12.007820] pc : 0x0
[   12.008027] lr : call_timer_fn+0x3c/0x1c4
[   12.008401] sp : ffff800008213a10
[   12.008701] x29: ffff800008213a10 x28: 0000000000000000 x27: ffff800008213ab0
[   12.009345] x26: ffffd3ee6e534008 x25: 0000000000000000 x24: ffffd3ee6eadec48
[   12.009986] x23: 0000000000000101 x22: 0000000000000000 x21: ffff800008213ab0
[   12.010627] x20: ffff4026fdb7bb40 x19: ffff4026837e6f10 x18: 0000000000000000
[   12.011268] x17: ffff6c388f644000 x16: ffff80000802c000 x15: 0000989680013880
[   12.011909] x14: 000000000003a980 x13: 0000000000000000 x12: 000342c486a19e23
[   12.012059] systemd-journald[240]: Received SIGTERM from PID 1 (systemd).
[   12.012542] x11: ffff4026fdb7bba8 x10: 0000000000000001 x9 : ffffd3ee6c74f300
[   12.012557] x8 : 00000000000000be x7 : 0000000000000008 x6 : 0000000000000200
[   12.014431] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 00000000000001c0
[   12.015071] x2 : 00000000ffff8f80 x1 : 0000000000000000 x0 : ffff4026837e6f10
[   12.015712] Call trace:
[   12.015936]  0x0
[   12.016110]  __run_timers.part.0+0x19c/0x2a0
[   12.016501]  run_timer_softirq+0x50/0x8c
[   12.016860]  __do_softirq+0x168/0x410
[   12.017193]  __irq_exit_rcu+0xf8/0x15c
[   12.017534]  irq_exit_rcu+0x18/0x24
[   12.017850]  el1_interrupt+0x38/0x70
[   12.018180]  el1h_64_irq_handler+0x18/0x24
[   12.018553]  el1h_64_irq+0x68/0x6c
[   12.018861]  cpuidle_enter_state+0xbc/0x404
[   12.019240]  cpuidle_enter+0x40/0x5c
[   12.019562]  cpuidle_idle_call+0x10c/0x1a0
[   12.019933]  do_idle+0xa4/0xf4
[   12.020212]  cpu_startup_entry+0x2c/0x34
[   12.020567]  secondary_start_kernel+0xd8/0xf0
[   12.020961]  __secondary_switched+0xa0/0xa4
[   12.021344] Code: bad PC value
[   12.021622] ---[ end trace 0000000000000000 ]---
[   12.022034] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[   12.022642] SMP: stopping secondary CPUs
[   12.023205] Kernel Offset: 0x53ee645e0000 from 0xffff800008000000
[   12.023744] PHYS_OFFSET: 0xffffbfd980000000
[   12.024115] CPU features: 0x2000,04125811,00001086
[   12.024542] Memory Limit: none
[   12.024820] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

[1] https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org/thread/XPDSK3P2MTL5ST5NW5DUNNNF6UWMNQCR/
Comment 1 Peter Robinson 2022-10-12 18:16:47 UTC 
Looks like an issue with the USB2 module. Adding modprobe.blacklist=phy_rockchip_inno_usb2 works around the issue
Comment 2 Fedora Blocker Bugs Application 2022-10-12 18:22:22 UTC 
Proposed as a Blocker for 37-final by Fedora user pbrobinson using the blocker tracking app because:

 Affects key hardware supported by both Arm and IoT. This won't allow the affected devices boot to completion where the user could apply a zero update to fix it.
Comment 3 Adam Williamson 2022-10-13 08:44:37 UTC 
+4 in https://pagure.io/fedora-qa/blocker-review/issue/968 , marking accepted.
Comment 4 Peter Robinson 2022-10-13 12:58:57 UTC 
For 5.19/6.0:
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2098
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2101
Comment 5 Fedora Update System 2022-10-14 06:49:47 UTC 
FEDORA-2022-1a5b125ac6 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-1a5b125ac6
Comment 6 Fedora Update System 2022-10-14 11:38:21 UTC 
FEDORA-2022-2cfbe17910 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-2cfbe17910
Comment 7 Fedora Update System 2022-10-14 11:38:44 UTC 
FEDORA-2022-b948fc3cfb has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-b948fc3cfb
Comment 8 Fedora Update System 2022-10-14 12:59:49 UTC 
FEDORA-2022-2cfbe17910 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 9 Adam Williamson 2022-10-14 13:19:50 UTC 
Re-opening for F37, as this is an F37 blocker.
Comment 10 Fedora Update System 2022-10-14 14:38:01 UTC 
FEDORA-2022-b948fc3cfb has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-b948fc3cfb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-b948fc3cfb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 11 Fedora Update System 2022-10-15 21:27:00 UTC 
FEDORA-2022-b948fc3cfb has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 12 Fedora Update System 2022-10-17 22:55:18 UTC 
FEDORA-2022-1a5b125ac6 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.