Created attachment 2008833 [details]
UART log of kernel which crashed at boot stage.

Out of interest does the same crash happen if booted with a Fedora Minimal image?

Also what type of storage are you using for the rootfs and related bits?

So looking at the crash, included below for ease, the lines around truncate_inode_pages_range tells me this is some issue to do with filesystems, the mention of folios says it's memory too.


[   23.198693] ------------[ cut here ]------------
[   23.203312] kernel BUG at lib/list_debug.c:52!
[   23.207760] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
[   23.213857] Modules linked in: xfs sy8106a_regulator mmc_block dwmac_sun8i stmmac_platform crct10dif_ce stmmac polyval_ce polyval_generic i2c_mv64xxx ghash_ce pcs_xpcs phylink spi_sun6i mdio_mux sunxi governor_simpleondemand musb_hdrc sun8i_a33_mbus phy_generic des
_generic libdes sunxi_wdt ohci_platform ohci_hcd ehci_platform udc_core sun8i_ce nvmem_sunxi_sid phy_sun4i_usb sunxi_mmc sun6i_dma gpio_keys cpufreq_dt be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libis
csi_tcp libiscsi scsi_transport_iscsi scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables ip_tables dm_multipath fuse
[   23.270667] CPU: 3 PID: 1 Comm: systemd Not tainted 6.6.8-200.fc39.aarch64 #1
[   23.277805] Hardware name: xunlong,orangepi-pc2 Xunlong Orange Pi PC 2/Xunlong Orange Pi PC 2, BIOS 2023.07 07/01/2023
[   23.288497] pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   23.295460] pc : __list_del_entry_valid_or_report+0x6c/0xd0
[   23.301048] lr : __list_del_entry_valid_or_report+0x6c/0xd0
[   23.306625] sp : ffff80008002b9e0
[   23.309941] x29: ffff80008002b9e0 x28: ffff80008002bb70 x27: 0000000000000006
[   23.317086] x26: ffff00000012d000 x25: 00000000ffffffff x24: fffffc0000149e08
[   23.324230] x23: ffff80008002ba18 x22: ffff800082b71100 x21: ffff80008002bbb8
[   23.331375] x20: 0000000000000002 x19: fffffc0000149e00 x18: 0000000000000006
[   23.338518] x17: 0000000000000000 x16: 0000000000000000 x15: 0765076e073e072d
[   23.345662] x14: 0738073007650739 x13: 0765076e073e072d x12: 0738073007650739
[   23.352805] x11: 00000000ffffdfff x10: ffff800082c5e560 x9 : ffff80008017436c
[   23.359948] x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 : 00000000000affa8
[   23.367091] x5 : 0000000000001fff x4 : 0000000000000000 x3 : 0000000000000000
[   23.374234] x2 : 0000000000000000 x1 : ffff00000028a200 x0 : 0000000000000033
[   23.381378] Call trace:
[   23.383827]  __list_del_entry_valid_or_report+0x6c/0xd0
[   23.389059]  release_pages+0x310/0x560
[   23.392818]  __folio_batch_release+0x30/0x78
[   23.397096]  truncate_inode_pages_range+0x15c/0x488
[   23.401981]  truncate_inode_pages_final+0x58/0x90
[   23.406691]  evict+0x184/0x1a8
[   23.409755]  iput_final+0xb0/0x158
[   23.413164]  iput.part.0+0x110/0x130
[   23.416745]  iput+0x24/0x40
[   23.419546]  do_unlinkat+0x1d4/0x2d0
[   23.423128]  __arm64_sys_unlinkat+0x44/0x90
[   23.427318]  invoke_syscall+0x78/0x100
[   23.431073]  el0_svc_common.constprop.0+0x48/0xf0
[   23.435781]  do_el0_svc+0x24/0x38
[   23.439101]  el0_svc+0x3c/0x110
[   23.442252]  el0t_64_sync_handler+0x120/0x130
[   23.446615]  el0t_64_sync+0x194/0x198
[   23.450288] Code: d65f03c0 b00087a0 913c8000 97e4cd28 (d4210000) 
[   23.456385] ---[ end trace 0000000000000000 ]---

(In reply to Peter Robinson from comment #2)
> Out of interest does the same crash happen if booted with a Fedora Minimal
> image?

I flashed "Fedora-Minimal-39-1.5.aarch64.raw.xz" using "arm-image-installer --image Fedora-Minimal-39-1.5.aarch64.raw.xz --media /dev/sda --addconsole --addkey ../.ssh/id_ed25519.pub --norootpass --resizefs --target orangepi_pc2 -y" command.
Before that I zero-filled the uSD card using gnome-disks (no error messages appeared at dmesg output).

After running "dnf install fedora-repos-rawhide" and "dnf update --enablerepo=rawhide kernel" I was able boot with the 6.8 kernel (surprisingly I could get the kernel updated successfully)

I hit the wall again though. See the crash log please.

Created attachment 2008960 [details]
Fedora Minimal with Rawhide kernel.

(In reply to Peter Robinson from comment #3)
> Also what type of storage are you using for the rootfs and related bits?

I use a Sandisk Ultra 16GB Class 10 HC I uSD card as the storage. The board has uboot supplied by fedora package installed on the SPI nor flash but since the uSD card has its own uboot as well the SPI uboot is masked

I reflash the same card with Armbian or ArchlinuxARM, and works fine at both of them. 

I have more than 1 of these uSD cards (same model but bought at different times) and check their healths regularly (in need) by zero-filling.

(In reply to Peter Robinson from comment #4)
> So looking at the crash, included below for ease, the lines around
> truncate_inode_pages_range tells me this is some issue to do with
> filesystems, the mention of folios says it's memory too.
> 
> 
> [   23.198693] ------------[ cut here ]------------
> [   23.203312] kernel BUG at lib/list_debug.c:52!
> [   23.207760] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
> [   23.213857] Modules linked in: xfs sy8106a_regulator mmc_block
> dwmac_sun8i stmmac_platform crct10dif_ce stmmac polyval_ce polyval_generic
> i2c_mv64xxx ghash_ce pcs_xpcs phylink spi_sun6i mdio_mux sunxi
> governor_simpleondemand musb_hdrc sun8i_a33_mbus phy_generic des
> _generic libdes sunxi_wdt ohci_platform ohci_hcd ehci_platform udc_core
> sun8i_ce nvmem_sunxi_sid phy_sun4i_usb sunxi_mmc sun6i_dma gpio_keys
> cpufreq_dt be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio
> libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libis
> csi_tcp libiscsi scsi_transport_iscsi scsi_dh_rdac scsi_dh_emc scsi_dh_alua
> ip6_tables ip_tables dm_multipath fuse
> [   23.270667] CPU: 3 PID: 1 Comm: systemd Not tainted
> 6.6.8-200.fc39.aarch64 #1
> [   23.277805] Hardware name: xunlong,orangepi-pc2 Xunlong Orange Pi PC
> 2/Xunlong Orange Pi PC 2, BIOS 2023.07 07/01/2023
> [   23.288497] pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS
> BTYPE=--)
> [   23.295460] pc : __list_del_entry_valid_or_report+0x6c/0xd0
> [   23.301048] lr : __list_del_entry_valid_or_report+0x6c/0xd0
> [   23.306625] sp : ffff80008002b9e0
> [   23.309941] x29: ffff80008002b9e0 x28: ffff80008002bb70 x27:
> 0000000000000006
> [   23.317086] x26: ffff00000012d000 x25: 00000000ffffffff x24:
> fffffc0000149e08
> [   23.324230] x23: ffff80008002ba18 x22: ffff800082b71100 x21:
> ffff80008002bbb8
> [   23.331375] x20: 0000000000000002 x19: fffffc0000149e00 x18:
> 0000000000000006
> [   23.338518] x17: 0000000000000000 x16: 0000000000000000 x15:
> 0765076e073e072d
> [   23.345662] x14: 0738073007650739 x13: 0765076e073e072d x12:
> 0738073007650739
> [   23.352805] x11: 00000000ffffdfff x10: ffff800082c5e560 x9 :
> ffff80008017436c
> [   23.359948] x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 :
> 00000000000affa8
> [   23.367091] x5 : 0000000000001fff x4 : 0000000000000000 x3 :
> 0000000000000000
> [   23.374234] x2 : 0000000000000000 x1 : ffff00000028a200 x0 :
> 0000000000000033
> [   23.381378] Call trace:
> [   23.383827]  __list_del_entry_valid_or_report+0x6c/0xd0
> [   23.389059]  release_pages+0x310/0x560
> [   23.392818]  __folio_batch_release+0x30/0x78
> [   23.397096]  truncate_inode_pages_range+0x15c/0x488
> [   23.401981]  truncate_inode_pages_final+0x58/0x90
> [   23.406691]  evict+0x184/0x1a8
> [   23.409755]  iput_final+0xb0/0x158
> [   23.413164]  iput.part.0+0x110/0x130
> [   23.416745]  iput+0x24/0x40
> [   23.419546]  do_unlinkat+0x1d4/0x2d0
> [   23.423128]  __arm64_sys_unlinkat+0x44/0x90
> [   23.427318]  invoke_syscall+0x78/0x100
> [   23.431073]  el0_svc_common.constprop.0+0x48/0xf0
> [   23.435781]  do_el0_svc+0x24/0x38
> [   23.439101]  el0_svc+0x3c/0x110
> [   23.442252]  el0t_64_sync_handler+0x120/0x130
> [   23.446615]  el0t_64_sync+0x194/0x198
> [   23.450288] Code: d65f03c0 b00087a0 913c8000 97e4cd28 (d4210000) 
> [   23.456385] ---[ end trace 0000000000000000 ]---

As an average linux user my humble guess is that the problem may be related to crypto engine (sun8i_ce module). The problem mostly occurs during podman blob extraction or ssh login. It worths a try to disable it and depend on the CPU's NEON thingie
Can you instruct me about this?

> As an average linux user my humble guess is that the problem may be related
> to crypto engine (sun8i_ce module). The problem mostly occurs during podman
> blob extraction or ssh login. It worths a try to disable it and depend on
> the CPU's NEON thingie
> Can you instruct me about this?

It doesn't show up in the crash but it doesn't mean it's not coming into play so it's definitely worth giving it a try. The sunxi crypto drivers have caused issues in the past. To do this you add a file to block it by:

echo "blacklist sun8i_ce" > /etc/modprobe.d/sunxi.conf

I did 'echo "blacklist sun8i_ce" > /etc/modprobe.d/sunxi.conf' and rebooted. Kernel crashed at boot stage.
Then I followed these instructions (except kdump part) and rebuilt the initramfs with the command "dracut --regenerate-all --force". System ran prettyt stable for a long time (installed podman via dnf). While I reboot, kernel crashed again. See crash log please.

Created attachment 2009009 [details]
Shutdown crash.

Forgot to put the link:
https://access.redhat.com/solutions/41278

Oh? Did you find a fix?

Yes.I switched to Armbian.

The story is:
Starting with Fedora 37 (IoT) kernel, I see these crashes. Tried Fedora 35 and 36 (IoT) they are stable, of course all without USB and without CPU governor.
Initially I thought it was a kernel regression appeared after 6.1.x (latest Armbian and ArchlinuxARM kernels for Orangepi PC2)  but 6.0.x kernel of Fedora 37 hit the bug either.
So we have a SBC having more than 2 years old of a kernel bug and more than 4 years of a suboptimal CPU (running at ~2/3th of possible CPU freq) with subfunctional peripherals (no USB etc).
Even more, it's listed as "supported" by the Fedora team.

Assuming you don't possess the board and wanting to fix the bug(s), I'm here to help for sake of community: I test the pathecd kernels, new builds etc.
Otherwise please remove it from the "supported devices list".

> Assuming you don't possess the board and wanting to fix the bug(s), I'm here

I believe it's impossible for us to possess all possible combination of boards, it doesn't mean we don't want to fix the bugs. Isn't that what I was doing in this and your other bug?

We, unlike Armbian, consume pure upstream kernels, if we have fixes we work to get them upstream to improve the devices for all distros not just Fedora where Armbian just pulls patches in locally.

> to help for sake of community: I test the pathecd kernels, new builds etc.
> Otherwise please remove it from the "supported devices list".

Where is it listed as a supported device? I'll review it.

(In reply to Peter Robinson from comment #15)

> I believe it's impossible for us to possess all possible combination of
> boards, it doesn't mean we don't want to fix the bugs. Isn't that what I was
> doing in this and your other bug?

I appreciate that. Thank you very much.

> We, unlike Armbian, consume pure upstream kernels, if we have fixes we work
> to get them upstream to improve the devices for all distros not just Fedora
> where Armbian just pulls patches in locally.

I support the software development model you follow. You spearhead the advancement and don't rely on others work. That's why I use/perefer Fedora. 

> Where is it listed as a supported device? I'll review it.

See the attchment please.

This kernel crash is ~2 years old and apparently no one other then me even dared to open a bug report, but I did open one ^^
Excuse me for being rude but don't I have a point from a user perspective?
Still as you said, there's nothing to do at your side. That's why I do my best and continue to invest some more time ;D

Anyways... Good news that I narrowed down the bug to a uboot-image-armv8 package version.
As I mentioned in my previous post I found out that uboot version (2022.04) of Fedora 36 is the last working uboot.
So it's not a direct kernel bug but a uboot one. uboot 2022.10 and its sucessors with any other kernel just crashes.
I'm running the latest coreos build with uboot 2022.04 just fine, no crashes so far.

Do you want me to open another bug report for it or will you just remove the support?

Created attachment 2009261 [details]
Supported boards list of arm-image-installer.

> > Where is it listed as a supported device? I'll review it.
> 
> See the attchment please.

So being listed as supported by arm-image-installer it means that arm-image-installer supports writing out the firmware for that board and setting up the device, this is in fact the case and you used it to write a image out successfully.

It does not mean that the device is 100% supported in Fedora for all bits of HW attached to it and all use cases. It purely indicates whether that utility supports the device. I will add a sentence at the top of that output to be clearer about what supported means there.

(In reply to Peter Robinson from comment #18)
> > > Where is it listed as a supported device? I'll review it.
> > 
> > See the attchment please.
> 
> So being listed as supported by arm-image-installer it means that
> arm-image-installer supports writing out the firmware for that board and
> setting up the device, this is in fact the case and you used it to write a
> image out successfully.
> 
> It does not mean that the device is 100% supported in Fedora for all bits of
> HW attached to it and all use cases. It purely indicates whether that
> utility supports the device. I will add a sentence at the top of that output
> to be clearer about what supported means there.

You should check this too:
https://src.fedoraproject.org/rpms/uboot-tools/blob/f39/f/aarch64-boards

> > So being listed as supported by arm-image-installer it means that
> > arm-image-installer supports writing out the firmware for that board and
> > setting up the device, this is in fact the case and you used it to write a
> > image out successfully.
> > 
> > It does not mean that the device is 100% supported in Fedora for all bits of
> > HW attached to it and all use cases. It purely indicates whether that
> > utility supports the device. I will add a sentence at the top of that output
> > to be clearer about what supported means there.
> 
> You should check this too:
> https://src.fedoraproject.org/rpms/uboot-tools/blob/f39/f/aarch64-boards

That's the U-Boot firmwares we build, again whether we build the firmware for a device has no impact on whether USB or the diplay works in the kernel.

uboot firmware you distrubute for a specific board causes kernels stop working for ~2 years.
Are you even reading my posts?

> uboot firmware you distrubute for a specific board causes kernels stop
> working for ~2 years.

We ship pure upstream U-Boot

> Are you even reading my posts?

Where in your posts do you show that it's a problem with U-Boot, everything to me looks like a kernel problem.

> Anyways... Good news that I narrowed down the bug to a uboot-image-armv8
> package version.
> As I mentioned in my previous post I found out that uboot version (2022.04)
> of Fedora 36 is the last working uboot.
> So it's not a direct kernel bug but a uboot one. uboot 2022.10 and its
> sucessors with any other kernel just crashes.
> I'm running the latest coreos build with uboot 2022.04 just fine, no crashes
> so far.

I possess the board and tested it thoroughly and I definitely back my words, those are not based on assumptions, guesses...

> We ship pure upstream U-Boot

I wouldn't call this "pure":
https://src.fedoraproject.org/rpms/uboot-tools/tree/f39

It's either a packaging bug or upstream bug you have to an action againist:
1. Remove the support for the board, and do not distrubute knowingly a broken package binary (playing with words won't get us anywhere)
2. Fix the bug with the feedback from a user. Provide the user kernel/uboot packages and test. Get the feedback.

I sent bug outputs, feedbacks, even bisected the bug which is very time consuming at release level and you either ignore them or beahave like a commentator.

Please show me "your" code or "your" action.