Update: the crash doesn't always happen in the same code, so the event code likely has nothing to do with it. Here are two other traces:

[  663.835239] Call Trace:
[  663.835798]  <TASK>
[  663.836339]  ? __die_body.cold+0x19/0x27
[  663.837232]  ? die_addr+0x3c/0x60
[  663.837969]  ? exc_general_protection+0x17d/0x400
[  663.839060]  ? asm_exc_general_protection+0x26/0x30
[  663.840212]  ? event_init+0x1d/0x70
[  663.840965]  trace_module_notify+0x19a/0x240
[  663.841965]  notifier_call_chain+0x5d/0xd0
[  663.842885]  blocking_notifier_call_chain_robust+0x65/0xc0
[  663.844113]  load_module+0x1cce/0x23e0
[  663.845068]  ? __do_sys_init_module+0x17a/0x1b0
[  663.846355]  __do_sys_init_module+0x17a/0x1b0
[  663.847320]  do_syscall_64+0x82/0x160
[  663.848111]  ? __vm_munmap+0xb9/0x170
[  663.849155]  ? syscall_exit_to_user_mode_prepare+0x173/0x1b0
[  663.850608]  ? syscall_exit_to_user_mode+0x10/0x210
[  663.851758]  ? do_syscall_64+0x8e/0x160
[  663.852695]  ? __handle_mm_fault+0xb55/0xfd0
[  663.853712]  ? syscall_exit_to_user_mode+0x10/0x210
[  663.854862]  ? __count_memcg_events+0xc0/0x180
[  663.855878]  ? count_memcg_events.constprop.0+0x1a/0x30
[  663.857056]  ? handle_mm_fault+0x21b/0x330
[  663.858196]  ? do_user_addr_fault+0x55a/0x7b0
[  663.859222]  ? exc_page_fault+0x7e/0x180
[  663.860088]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  663.861371] RIP: 0033:0x7f07ac6d269e

[  650.995898] Call Trace:
[  650.996516]  <TASK>
[  650.996973]  ? __die_body.cold+0x19/0x27
[  650.997888]  ? die_addr+0x3c/0x60
[  650.998662]  ? exc_general_protection+0x17d/0x400
[  650.999749]  ? asm_exc_general_protection+0x26/0x30
[  651.000898]  ? avc_lookup+0x47/0x70
[  651.001721]  avc_has_perm_noaudit+0x3d/0xf0
[  651.002667]  selinux_inode_permission+0x13a/0x1e0
[  651.003823]  security_inode_permission+0x42/0xf0
[  651.004897]  link_path_walk.part.0.constprop.0+0xad/0x390
[  651.006164]  path_lookupat+0x3e/0x1a0
[  651.007003]  filename_lookup+0xf2/0x200
[  651.007904]  user_path_at+0x56/0x90
[  651.008717]  inotify_find_inode+0x21/0x80
[  651.009647]  __x64_sys_inotify_add_watch+0xbe/0x140
[  651.010770]  do_syscall_64+0x82/0x160
[  651.011666]  ? syscall_exit_to_user_mode_prepare+0x173/0x1b0
[  651.012928]  ? syscall_exit_to_user_mode+0x10/0x210
[  651.014083]  ? do_syscall_64+0x8e/0x160
[  651.014987]  ? syscall_exit_to_user_mode+0x1d5/0x210
[  651.016122]  ? do_syscall_64+0x8e/0x160
[  651.017104]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  651.018284] RIP: 0033:0x7fa0592366cb

Steven Rostedt suggests it looks like memory corruption since it crashes in different places each time.

Looks like this doesn't only affect live installs, https://openqa.fedoraproject.org/tests/3064100 is a non-live install to BIOS that looks like it was also affected. I really need to try and get the system/anaconda logs for a failure, though, so we can see exactly what's happening when the GPF hits. Will twiddle with that today.

Created attachment 2060849 [details]
video of a likely-related issue in UEFI installs

Huh. There is actually a bug happening on UEFI installs as well which may be the equivalent of this there, but it's even weirder.

What happens there is that, while the installer is at the "installing bootloader" stage, the system suddenly reboots, then gets stuck at shim/grub stage (not surprisingly, I guess, as it doesn't look like the installer actually completed cleanly). You can see this happen around 0:56 of the attached video - the installer is running...and suddenly we're rebooting. This is not normal, usually the installer would run until it showed a 'complete' screen, the test system is waiting to see that screen at this point.

I'm guessing we actually hit something similar to the BIOS path on the UEFI path too, but somehow rather than causing the system to hang, it causes it to *reboot*, and we attempt boot to an incomplete install that fails to boot properly.

Oh, one thing that may be relevant here - we set the qemu CPU model to Nehalem. We do this intentionally to make sure Fedora isn't inadvertently progressing beyond its intended CPU baseline.

ah, yeah, I think that *is* relevant. I hadn't been able to reproduce this locally before, but I just set my local VM's CPU to Nehalem, and reproduced the UEFI version of this first try. I noticed the VM did indeed seem to be stuck at the bootloader install stage - it was not responding to any kind of input, couldn't switch to a VT - then after it sat like that for a few seconds or minutes (don't know how long it was sitting like that before I saw it), it spontaneously rebooted and then failed to boot properly from the disk, just like the openQA test in the video.

It's good news that I can reproduce it locally now, I can fiddle about with settings to try and get the anaconda logs to the serial console without having to 'fly by wire' with openQA.

OK, here we go, managed to get combined kernel and anaconda logs when reproducing this locally. This is the UEFI case. Here's what we get:

Dec 02 18:55:26 localhost-live org.fedoraproject.Anaconda.Modules.Storage[5364]: INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
[ 1243.766337] SGI XFS with ACLs, security attributes, realtime, scrub, quota, no debug enabled
[ 1243.801236] JFS: nTxBlock = 8192, nTxLock = 65536
[ 1244.145085] list_del corruption. next->prev should be ffff9c39c4489298, but was 7b8d4840438d4840. (next=ffff9c3a0233d640)
[ 1244.145273] ------------[ cut here ]------------
[ 1244.145276] kernel BUG at lib/list_debug.c:65!
[ 1244.145340] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 1244.145352] CPU: 0 UID: 0 PID: 92 Comm: kworker/u9:4 Not tainted 6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
[ 1244.145360] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS edk2-20240813-2.fc42 08/13/2024
[ 1244.145371] Workqueue: btrfs-delayed-meta btrfs_work_helper
[ 1244.145423] RIP: 0010:__list_del_entry_valid_or_report.cold+0x23/0x6f
[ 1244.145446] Code: e8 a5 8a fb ff 0f 0b 48 89 fe 48 c7 c7 18 7a e3 87 e8 94 8a fb ff 0f 0b 48 89 d1 48 c7 c7 38 7b e3 87 48 89 c2 e8 80 8a fb ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 e8 7a e3 87 e8 6c 8a fb ff 0f 0b
[ 1244.145449] RSP: 0018:ffffbbf68030fd90 EFLAGS: 00010246
[ 1244.145460] RAX: 000000000000006d RBX: ffff9c38cdabc7a8 RCX: 0000000000000000
[ 1244.145464] RDX: 0000000000000000 RSI: ffff9c3a02e21900 RDI: ffff9c3a02e21900
[ 1244.145465] RBP: ffff9c39d61b08c0 R08: 0000000000000000 R09: 0000000000000000
[ 1244.145467] R10: 6e28202e30343834 R11: 666666663d747865 R12: ffff9c393e54f800
[ 1244.145468] R13: ffff9c39cb28bcc0 R14: ffff9c39c4489298 R15: ffff9c393c9b4c78
[ 1244.145469] FS:  0000000000000000(0000) GS:ffff9c3a02e00000(0000) knlGS:0000000000000000
[ 1244.145471] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1244.145472] CR2: 0000562d47c53900 CR3: 000000010218e000 CR4: 00000000000006f0
[ 1244.145477] Call Trace:
[ 1244.145497]  <TASK>
[ 1244.145502]  ? __die_body.cold+0x19/0x27
[ 1244.145514]  ? die+0x2e/0x50
[ 1244.145540]  ? do_trap+0xca/0x110
[ 1244.145546]  ? do_error_trap+0x6a/0x90
[ 1244.145548]  ? __list_del_entry_valid_or_report.cold+0x23/0x6f
[ 1244.145551]  ? exc_invalid_op+0x50/0x70
[ 1244.145565]  ? __list_del_entry_valid_or_report.cold+0x23/0x6f
[ 1244.145567]  ? asm_exc_invalid_op+0x1a/0x20
[ 1244.145591]  ? __list_del_entry_valid_or_report.cold+0x23/0x6f
[ 1244.145593]  btrfs_async_run_delayed_root+0x93/0x2c0
[ 1244.145605]  btrfs_work_helper+0xe8/0x380
[ 1244.145611]  ? drm_atomic_state_default_clear+0x1c3/0x2e0
[ 1244.145634]  process_one_work+0x179/0x330
[ 1244.145653]  worker_thread+0x252/0x390
[ 1244.145656]  ? __pfx_worker_thread+0x10/0x10
[ 1244.145657]  kthread+0xd2/0x100
[ 1244.145664]  ? __pfx_kthread+0x10/0x10
[ 1244.145667]  ret_from_fork+0x34/0x50
[ 1244.145674]  ? __pfx_kthread+0x10/0x10
[ 1244.145676]  ret_from_fork_asm+0x1a/0x30
[ 1244.145694]  </TASK>
[ 1244.145697] Modules linked in: ufs hfsplus hfs minix msdos jfs nls_ucs2_utils xfs libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 nvme_tcp nvme_fabrics uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel joydev snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore pktcdvd iTCO_wdt i2c_i801 intel_pmc_bxt iTCO_vendor_support virtio_balloon i2c_smbus pcspkr lpc_ich binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_scsi
[ 1244.145764]  net_failover virtio_console virtio_blk failover virtio_gpu virtio_dma_buf nvme_keyring nvme_core nvme_auth serio_raw sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse qemu_fw_cfg [last unloaded: nvme_fabrics]
[ 1244.145798] ---[ end trace 0000000000000000 ]---
[ 1244.145802] RIP: 0010:__list_del_entry_valid_or_report.cold+0x23/0x6f
[ 1244.145804] Code: e8 a5 8a fb ff 0f 0b 48 89 fe 48 c7 c7 18 7a e3 87 e8 94 8a fb ff 0f 0b 48 89 d1 48 c7 c7 38 7b e3 87 48 89 c2 e8 80 8a fb ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 e8 7a e3 87 e8 6c 8a fb ff 0f 0b
[ 1244.145806] RSP: 0018:ffffbbf68030fd90 EFLAGS: 00010246
[ 1244.145808] RAX: 000000000000006d RBX: ffff9c38cdabc7a8 RCX: 0000000000000000
[ 1244.145809] RDX: 0000000000000000 RSI: ffff9c3a02e21900 RDI: ffff9c3a02e21900
[ 1244.145810] RBP: ffff9c39d61b08c0 R08: 0000000000000000 R09: 0000000000000000
[ 1244.145811] R10: 6e28202e30343834 R11: 666666663d747865 R12: ffff9c393e54f800
[ 1244.145812] R13: ffff9c39cb28bcc0 R14: ffff9c39c4489298 R15: ffff9c393c9b4c78
[ 1244.145814] FS:  0000000000000000(0000) GS:ffff9c3a02e00000(0000) knlGS:0000000000000000
[ 1244.145815] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1244.145846] CR2: 0000562d47c53900 CR3: 000000010218e000 CR4: 00000000000006f0
[ 1244.145859] note: kworker/u9:4[92] exited with preempt_count 1
[ 1245.822413] show_signal_msg: 17 callbacks suppressed
[ 1245.822421] Isolated Web Co[5650]: segfault at 0 ip 00007fcb20e99fee sp 00007fff2b2f21e8 error 6 in libxul.so[51e9fee,7fcb1bcb0000+51f7000] likely on CPU 1 (core 0, socket 1)
[ 1245.822446] Code: ff e8 66 c7 e1 fa 66 0f ef ed 0f 28 f5 e9 a3 fb ff ff 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 8b 05 ed c7 de 02 48 89 10 <c7> 04 25 00 00 00 00 00 00 00 00 0f 0b 0f 1f 44 00 00 f3 0f 1e fa

...so we were running grub2-mkconfig in a chroot to the installed system when it blew up. The:
list_del corruption. next->prev should be ffff9c39c4489298, but was 7b8d4840438d4840. (next=ffff9c3a0233d640)
message looks interesting.

hmm, that's actually different from the others. Not sure if it's UEFI vs. BIOS, this is my first time seeing the actual death on UEFI. I'll reproduce this test process on BIOS and see what we get there.

OK, here's what I got on a BIOS reproducer. Seems like some other stuff happened before the GPF...

Dec 03 01:16:49 localhost-live org.fedoraproject.Anaconda.Modules.Storage[3508]: INFO:program:Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
[  588.823333] ------------[ cut here ]------------
[  588.823338] WARNING: CPU: 0 PID: 4325 at arch/x86/kernel/ftrace.c:100 ftrace_verify_code+0x4c/0x90
[  588.823359] Modules linked in: libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec joydev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt intel_pmc_bxt snd_pcm iTCO_vendor_support pktcdvd i2c_i801 lpc_ich snd_timer snd soundcore pcspkr virtio_balloon i2c_smbus binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_blk virtio_console virtio_scsi virtio_gpu net_failover failover virtio_dma_buf
[  588.823418]  nvme_tcp nvme_fabrics nvme_keyring serio_raw nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev qemu_fw_cfg
[  588.823442] CPU: 0 UID: 0 PID: 4325 Comm: modprobe Not tainted 6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
[  588.823445] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
[  588.823450] RIP: 0010:ftrace_verify_code+0x4c/0x90
[  588.823453] Code: fe c6 44 24 07 00 48 89 c7 c7 44 24 03 00 00 00 00 e8 98 6e 30 00 48 85 c0 75 3e 8b 03 39 44 24 03 74 28 48 89 1d fc b3 7e 03 <0f> 0b b8 ea ff ff ff 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75
[  588.823454] RSP: 0018:ffffbb83833ef860 EFLAGS: 00010206
[  588.823459] RAX: 00000000b898a7e8 RBX: ffffffffbd7c2e08 RCX: 0000000000000010
[  588.823460] RDX: ffffa07bd290a880 RSI: 0000000000000005 RDI: ffffffffc1514064
[  588.823461] RBP: ffffffffc1514064 R08: 0000000000000000 R09: ffffa07c3bc3ee20
[  588.823462] R10: ffffbb83833ef8a0 R11: dead000000000040 R12: ffffffffbb60b5ea
[  588.823463] R13: 0000000002000000 R14: ffffa07bcbe50ce0 R15: 0000000000000f3e
[  588.823465] FS:  00007f65a27c8740(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
[  588.823466] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  588.823467] CR2: 00007f3ef9753f10 CR3: 00000000027a8000 CR4: 00000000000006f0
[  588.823471] Call Trace:
[  588.823478]  <TASK>
[  588.823479]  ? ftrace_verify_code+0x4c/0x90
[  588.823482]  ? __warn.cold+0x93/0xfa
[  588.823484]  ? ftrace_verify_code+0x4c/0x90
[  588.823493]  ? report_bug+0xff/0x140
[  588.823496]  ? handle_bug+0x58/0x90
[  588.823498]  ? exc_invalid_op+0x17/0x70
[  588.823500]  ? asm_exc_invalid_op+0x1a/0x20
[  588.823516]  ? ftrace_verify_code+0x4c/0x90
[  588.823518]  ftrace_modify_code_direct+0xf/0x70
[  588.823522]  ftrace_process_locs+0x313/0x5c0
[  588.823532]  ftrace_module_init+0x32/0x50
[  588.823537]  load_module+0x1b1a/0x23e0
[  588.823552]  ? __do_sys_init_module+0x17a/0x1b0
[  588.823554]  __do_sys_init_module+0x17a/0x1b0
[  588.823557]  do_syscall_64+0x82/0x160
[  588.823565]  ? __vm_munmap+0xb9/0x170
[  588.823580]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.823583]  ? do_syscall_64+0x8e/0x160
[  588.823585]  ? __alloc_pages_noprof+0x184/0x330
[  588.823591]  ? __mod_memcg_lruvec_state+0xdf/0x220
[  588.823599]  ? __count_memcg_events+0xc0/0x180
[  588.823601]  ? __lruvec_stat_mod_folio+0x83/0xd0
[  588.823604]  ? set_ptes.isra.0+0x41/0x90
[  588.823606]  ? do_anonymous_page+0xfc/0x8f0
[  588.823609]  ? __pte_offset_map+0x1b/0x180
[  588.823613]  ? __handle_mm_fault+0xb55/0xfd0
[  588.823616]  ? __count_memcg_events+0xc0/0x180
[  588.823619]  ? count_memcg_events.constprop.0+0x1a/0x30
[  588.823621]  ? handle_mm_fault+0x21b/0x330
[  588.823624]  ? do_user_addr_fault+0x55a/0x7b0
[  588.823631]  ? exc_page_fault+0x7e/0x180
[  588.823633]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  588.823637] RIP: 0033:0x7f65a210069e
[  588.823649] Code: 48 8b 0d 75 37 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 37 0f 00 f7 d8 64 89 01 48
[  588.823653] RSP: 002b:00007ffdb33ac7f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  588.823655] RAX: ffffffffffffffda RBX: 0000557dfc420ca0 RCX: 00007f65a210069e
[  588.823656] RDX: 0000557dbeee3715 RSI: 0000000000751c7e RDI: 00007f65a0b79010
[  588.823657] RBP: 00007ffdb33ac8b0 R08: 0000557dfc420010 R09: 0000000000000007
[  588.823658] R10: 0000000000000001 R11: 0000000000000246 R12: 0000557dbeee3715
[  588.823659] R13: 0000000000040000 R14: 0000557dfc420db0 R15: 0000000000000000
[  588.823661]  </TASK>
[  588.823662] ---[ end trace 0000000000000000 ]---
[  588.823665] ------------[ ftrace bug ]------------
[  588.823666] ftrace failed to modify 
[  588.823667] [<ffffffffc1514064>] 0xffffffffc1514064
[  588.823671]  actual:   e8:a7:d8:bc:f8
[  588.823673]  expected: e8:a7:98:b8:f8
[  588.823675] Initializing ftrace call sites
[  588.823675] ftrace record flags: 2000000
[  588.823676]  (0)    
                expected tramp: ffffffffba09d940
[  588.823704] ------------[ cut here ]------------
[  588.823705] WARNING: CPU: 0 PID: 4325 at kernel/trace/ftrace.c:2234 ftrace_bug+0x23c/0x264
[  588.823711] Modules linked in: libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec joydev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt intel_pmc_bxt snd_pcm iTCO_vendor_support pktcdvd i2c_i801 lpc_ich snd_timer snd soundcore pcspkr virtio_balloon i2c_smbus binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_blk virtio_console virtio_scsi virtio_gpu net_failover failover virtio_dma_buf
[  588.823752]  nvme_tcp nvme_fabrics nvme_keyring serio_raw nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev qemu_fw_cfg
[  588.823765] CPU: 0 UID: 0 PID: 4325 Comm: modprobe Tainted: G        W         -------  ---  6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
[  588.823768] Tainted: [W]=WARN
[  588.823769] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
[  588.823770] RIP: 0010:ftrace_bug+0x23c/0x264
[  588.823774] Code: ff 84 c0 74 d4 eb b8 48 c7 c7 99 27 ec bb e8 ab 71 ff ff 48 89 ef e8 03 e3 0e ff 48 c7 c7 aa 27 ec bb 48 89 c6 e8 94 71 ff ff <0f> 0b c7 05 d0 28 ad 01 01 00 00 00 5b 31 c0 5d 41 5c 89 05 d4 28
[  588.823776] RSP: 0018:ffffbb83833ef880 EFLAGS: 00010246
[  588.823777] RAX: 0000000000000022 RBX: ffffffffc1514064 RCX: 0000000000000027
[  588.823778] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffa07c3bc21900
[  588.823779] RBP: ffffa07bc832f3e0 R08: 0000000000000000 R09: 0000000000000000
[  588.823780] R10: 203a706d61727420 R11: 6465746365707865 R12: 00000000ffffffea
[  588.823781] R13: 0000000002000000 R14: ffffa07bcbe50ce0 R15: 0000000000000f3e
[  588.823782] FS:  00007f65a27c8740(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
[  588.823784] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  588.823785] CR2: 00007f3ef9753f10 CR3: 00000000027a8000 CR4: 00000000000006f0
[  588.823788] Call Trace:
[  588.823793]  <TASK>
[  588.823793]  ? ftrace_bug+0x23c/0x264
[  588.823796]  ? __warn.cold+0x93/0xfa
[  588.823798]  ? ftrace_bug+0x23c/0x264
[  588.823801]  ? report_bug+0xff/0x140
[  588.823804]  ? handle_bug+0x58/0x90
[  588.823805]  ? exc_invalid_op+0x17/0x70
[  588.823807]  ? asm_exc_invalid_op+0x1a/0x20
[  588.823810]  ? ftrace_bug+0x23c/0x264
[  588.823813]  ftrace_process_locs.cold+0x14/0xad
[  588.823817]  ftrace_module_init+0x32/0x50
[  588.823819]  load_module+0x1b1a/0x23e0
[  588.823823]  ? __do_sys_init_module+0x17a/0x1b0
[  588.823825]  __do_sys_init_module+0x17a/0x1b0
[  588.823828]  do_syscall_64+0x82/0x160
[  588.823831]  ? __vm_munmap+0xb9/0x170
[  588.823834]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.823837]  ? do_syscall_64+0x8e/0x160
[  588.823839]  ? __alloc_pages_noprof+0x184/0x330
[  588.823842]  ? __mod_memcg_lruvec_state+0xdf/0x220
[  588.823844]  ? __count_memcg_events+0xc0/0x180
[  588.823847]  ? __lruvec_stat_mod_folio+0x83/0xd0
[  588.823849]  ? set_ptes.isra.0+0x41/0x90
[  588.823851]  ? do_anonymous_page+0xfc/0x8f0
[  588.823854]  ? __pte_offset_map+0x1b/0x180
[  588.823856]  ? __handle_mm_fault+0xb55/0xfd0
[  588.823859]  ? __count_memcg_events+0xc0/0x180
[  588.823862]  ? count_memcg_events.constprop.0+0x1a/0x30
[  588.823864]  ? handle_mm_fault+0x21b/0x330
[  588.823866]  ? do_user_addr_fault+0x55a/0x7b0
[  588.823869]  ? exc_page_fault+0x7e/0x180
[  588.823872]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  588.823874] RIP: 0033:0x7f65a210069e
[  588.823877] Code: 48 8b 0d 75 37 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 37 0f 00 f7 d8 64 89 01 48
[  588.823880] RSP: 002b:00007ffdb33ac7f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  588.823882] RAX: ffffffffffffffda RBX: 0000557dfc420ca0 RCX: 00007f65a210069e
[  588.823885] RDX: 0000557dbeee3715 RSI: 0000000000751c7e RDI: 00007f65a0b79010
[  588.823886] RBP: 00007ffdb33ac8b0 R08: 0000557dfc420010 R09: 0000000000000007
[  588.823887] R10: 0000000000000001 R11: 0000000000000246 R12: 0000557dbeee3715
[  588.823888] R13: 0000000000040000 R14: 0000557dfc420db0 R15: 0000000000000000
[  588.823890]  </TASK>
[  588.823890] ---[ end trace 0000000000000000 ]---
[  588.872305] unexpected static_call insn opcode 0x85 at xchk_da_btree+0x402/0x4e0 [xfs]
[  588.872498] ------------[ cut here ]------------
[  588.872499] kernel BUG at arch/x86/kernel/static_call.c:139!
[  588.872514] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
[  588.872519] CPU: 1 UID: 0 PID: 4325 Comm: modprobe Tainted: G        W         -------  ---  6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
[  588.872523] Tainted: [W]=WARN
[  588.872524] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
[  588.872526] RIP: 0010:__static_call_validate.cold+0x12/0x14
[  588.872531] Code: 48 c7 c7 d0 94 dc bb e8 a6 ea 00 00 48 8b 43 08 4c 8b 33 e9 6b f7 ee fe 48 89 fa 0f b6 f0 48 c7 c7 00 95 dc bb e8 88 ea 00 00 <0f> 0b 65 8b 35 bb 01 ed 44 48 c7 c7 48 96 dc bb e8 73 ea 00 00 fa
[  588.872533] RSP: 0000:ffffbb83833ef7d8 EFLAGS: 00010246
[  588.872536] RAX: 000000000000004a RBX: ffffffffbb2353e0 RCX: 0000000000000000
[  588.872538] RDX: 0000000000000000 RSI: ffffa07c3bd21900 RDI: ffffa07c3bd21900
[  588.872539] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  588.872541] R10: 65657274625f6164 R11: 302f32303478302b R12: ffffffffc1517162
[  588.872542] R13: 0000000000000000 R14: ffffffffc1274368 R15: ffffffffbc878220
[  588.872543] FS:  00007f65a27c8740(0000) GS:ffffa07c3bd00000(0000) knlGS:0000000000000000
[  588.872545] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  588.872546] CR2: 00007f8418c0f000 CR3: 00000000027a8000 CR4: 00000000000006f0
[  588.872551] Call Trace:
[  588.872559]  <TASK>
[  588.872562]  ? __die_body.cold+0x19/0x27
[  588.872566]  ? die+0x2e/0x50
[  588.872578]  ? do_trap+0xca/0x110
[  588.872583]  ? do_error_trap+0x6a/0x90
[  588.872585]  ? __static_call_validate.cold+0x12/0x14
[  588.872587]  ? exc_invalid_op+0x50/0x70
[  588.872589]  ? __static_call_validate.cold+0x12/0x14
[  588.872591]  ? asm_exc_invalid_op+0x1a/0x20
[  588.872598]  ? xchk_da_btree+0x402/0x4e0 [xfs]
[  588.872735]  ? __pfx___cond_resched+0x10/0x10
[  588.872739]  ? __static_call_validate.cold+0x12/0x14
[  588.872741]  ? __static_call_validate.cold+0x12/0x14
[  588.872743]  arch_static_call_transform+0x6b/0xa0
[  588.872749]  ? xchk_da_btree+0x402/0x4e0 [xfs]
[  588.872862]  __static_call_init+0x276/0x300
[  588.872870]  ? __is_insn_slot_addr+0x45/0x70
[  588.872876]  ? __SCT__tp_func_ipi_exit+0x8/0x8
[  588.872880]  ? __SCT__tp_func_ipi_exit+0x8/0x8
[  588.872881]  static_call_module_notify+0x11f/0x150
[  588.872884]  notifier_call_chain+0x5d/0xd0
[  588.872897]  blocking_notifier_call_chain_robust+0x65/0xc0
[  588.872902]  load_module+0x1cce/0x23e0
[  588.872909]  ? __do_sys_init_module+0x17a/0x1b0
[  588.872911]  __do_sys_init_module+0x17a/0x1b0
[  588.872915]  do_syscall_64+0x82/0x160
[  588.872920]  ? __vm_munmap+0xb9/0x170
[  588.872924]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.872928]  ? do_syscall_64+0x8e/0x160
[  588.872931]  ? __alloc_pages_noprof+0x184/0x330
[  588.872934]  ? __mod_memcg_lruvec_state+0xdf/0x220
[  588.872938]  ? __count_memcg_events+0xc0/0x180
[  588.872941]  ? __lruvec_stat_mod_folio+0x83/0xd0
[  588.872944]  ? set_ptes.isra.0+0x41/0x90
[  588.872946]  ? do_anonymous_page+0xfc/0x8f0
[  588.872949]  ? __pte_offset_map+0x1b/0x180
[  588.872952]  ? __handle_mm_fault+0xb55/0xfd0
[  588.872956]  ? __count_memcg_events+0xc0/0x180
[  588.872959]  ? count_memcg_events.constprop.0+0x1a/0x30
[  588.872961]  ? handle_mm_fault+0x21b/0x330
[  588.872967]  ? do_user_addr_fault+0x55a/0x7b0
[  588.872971]  ? exc_page_fault+0x7e/0x180
[  588.872974]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  588.872978] RIP: 0033:0x7f65a210069e
[  588.872992] Code: 48 8b 0d 75 37 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 37 0f 00 f7 d8 64 89 01 48
[  588.872994] RSP: 002b:00007ffdb33ac7f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[  588.872997] RAX: ffffffffffffffda RBX: 0000557dfc420ca0 RCX: 00007f65a210069e
[  588.872998] RDX: 0000557dbeee3715 RSI: 0000000000751c7e RDI: 00007f65a0b79010
[  588.872999] RBP: 00007ffdb33ac8b0 R08: 0000557dfc420010 R09: 0000000000000007
[  588.873001] R10: 0000000000000001 R11: 0000000000000246 R12: 0000557dbeee3715
[  588.873002] R13: 0000000000040000 R14: 0000557dfc420db0 R15: 0000000000000000
[  588.873005]  </TASK>
[  588.873006] Modules linked in: xfs(+) libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec joydev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt intel_pmc_bxt snd_pcm iTCO_vendor_support pktcdvd i2c_i801 lpc_ich snd_timer snd soundcore pcspkr virtio_balloon i2c_smbus binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_blk virtio_console virtio_scsi virtio_gpu net_failover failover
[  588.873072]  virtio_dma_buf nvme_tcp nvme_fabrics nvme_keyring serio_raw nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev qemu_fw_cfg
[  588.873108] ---[ end trace 0000000000000000 ]---
[  588.873110] RIP: 0010:__static_call_validate.cold+0x12/0x14
[  588.873112] Code: 48 c7 c7 d0 94 dc bb e8 a6 ea 00 00 48 8b 43 08 4c 8b 33 e9 6b f7 ee fe 48 89 fa 0f b6 f0 48 c7 c7 00 95 dc bb e8 88 ea 00 00 <0f> 0b 65 8b 35 bb 01 ed 44 48 c7 c7 48 96 dc bb e8 73 ea 00 00 fa
[  588.873114] RSP: 0000:ffffbb83833ef7d8 EFLAGS: 00010246
[  588.873116] RAX: 000000000000004a RBX: ffffffffbb2353e0 RCX: 0000000000000000
[  588.873117] RDX: 0000000000000000 RSI: ffffa07c3bd21900 RDI: ffffa07c3bd21900
[  588.873119] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  588.873120] R10: 65657274625f6164 R11: 302f32303478302b R12: ffffffffc1517162
[  588.873121] R13: 0000000000000000 R14: ffffffffc1274368 R15: ffffffffbc878220
[  588.873122] FS:  00007f65a27c8740(0000) GS:ffffa07c3bd00000(0000) knlGS:0000000000000000
[  588.873124] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  588.873125] CR2: 00007f8418c0f000 CR3: 00000000027a8000 CR4: 00000000000006f0
[  588.891057] Oops: general protection fault, probably for non-canonical address 0x578a7de75fdfe55b: 0000 [#2] PREEMPT SMP PTI
[  588.891066] CPU: 0 UID: 1000 PID: 2146 Comm: kwin_wayland Tainted: G      D W         -------  ---  6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
[  588.891081] Tainted: [D]=DIE, [W]=WARN
[  588.891082] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
[  588.891084] RIP: 0010:kmem_cache_alloc_node_noprof+0x1cf/0x400
[  588.891094] Code: 84 12 ff ff ff 48 85 db 0f 84 09 ff ff ff 48 8b 03 48 c1 e8 36 41 39 c3 0f 85 f9 fe ff ff 41 8b 44 24 28 49 8b 34 24 48 01 f8 <48> 8b 18 48 89 c1 49 33 9c 24 b8 00 00 00 48 89 f8 48 0f c9 48 31
[  588.891096] RSP: 0018:ffffbb8381873690 EFLAGS: 00010202
[  588.891099] RAX: 578a7de75fdfe55b RBX: ffffee6f8418d740 RCX: 00000000ffffffff
[  588.891103] RDX: 00000000af4a6000 RSI: 0000000000040160 RDI: 578a7de75fdfe4eb
[  588.891106] RBP: ffffbb83818736e8 R08: 0000000000400cc0 R09: 0000000000000003
[  588.891108] R10: ffffffffbae8686a R11: 00000000ffffffff R12: ffffa07bc02af100
[  588.891109] R13: 0000000000400cc0 R14: 00000000ffffffff R15: 00000000000000e8
[  588.891111] FS:  00007f3f4367c280(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
[  588.891113] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  588.891114] CR2: 00007f3ef9b60324 CR3: 000000000128c000 CR4: 00000000000006f0
[  588.891119] Call Trace:
[  588.891121]  <TASK>
[  588.891125]  ? __die_body.cold+0x19/0x27
[  588.891130]  ? die_addr+0x3c/0x60
[  588.891133]  ? exc_general_protection+0x17d/0x400
[  588.891137]  ? asm_exc_general_protection+0x26/0x30
[  588.891141]  ? __alloc_skb+0x14a/0x1a0
[  588.891151]  ? kmem_cache_alloc_node_noprof+0x1cf/0x400
[  588.891155]  ? __alloc_skb+0x14a/0x1a0
[  588.891156]  __alloc_skb+0x14a/0x1a0
[  588.891159]  alloc_skb_with_frags+0x67/0x2d0
[  588.891163]  sock_alloc_send_pskb+0x1f7/0x240
[  588.891172]  ? avc_has_perm+0x5d/0xe0
[  588.891185]  unix_stream_sendmsg+0x17d/0x680
[  588.891190]  ____sys_sendmsg+0x3a0/0x3d0
[  588.891195]  ___sys_sendmsg+0x9a/0xe0
[  588.891199]  __sys_sendmsg+0x87/0xe0
[  588.891202]  do_syscall_64+0x82/0x160
[  588.891206]  ? _copy_to_user+0x36/0x50
[  588.891217]  ? drm_ioctl+0x2b7/0x530
[  588.891231]  ? __pfx_drm_mode_atomic_ioctl+0x10/0x10
[  588.891242]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.891246]  ? do_syscall_64+0x8e/0x160
[  588.891249]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.891252]  ? do_syscall_64+0x8e/0x160
[  588.891255]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.891257]  ? do_syscall_64+0x8e/0x160
[  588.891260]  ? drm_mode_addfb2+0xda/0xf0
[  588.891262]  ? __pfx_drm_mode_addfb2_ioctl+0x10/0x10
[  588.891264]  ? _copy_from_user+0x29/0x70
[  588.891265]  ? __check_object_size+0x58/0x230
[  588.891270]  ? sync_file_ioctl+0xac/0x5b0
[  588.891282]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.891285]  ? do_syscall_64+0x8e/0x160
[  588.891290]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.891293]  ? do_syscall_64+0x8e/0x160
[  588.891295]  ? do_syscall_64+0x8e/0x160
[  588.891298]  ? syscall_exit_to_user_mode+0x10/0x210
[  588.891300]  ? do_syscall_64+0x8e/0x160
[  588.891303]  ? exc_page_fault+0x7e/0x180
[  588.891306]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  588.891309] RIP: 0033:0x7f3f49ceadd2
[  588.891321] Code: 08 0f 85 51 3d ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 55 bf 01 00
[  588.891323] RSP: 002b:00007ffc3f949158 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[  588.891325] RAX: ffffffffffffffda RBX: 00007f3f4367c280 RCX: 00007f3f49ceadd2
[  588.891327] RDX: 0000000000004040 RSI: 00007ffc3f9491f0 RDI: 00000000000000d4
[  588.891328] RBP: 00007ffc3f949180 R08: 0000000000000000 R09: 0000000000000000
[  588.891329] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000003f4
[  588.891330] R13: 00007ffc3f9491f0 R14: 00005647557ee1c0 R15: ffffffffffffffff
[  588.891333]  </TASK>
[  588.891334] Modules linked in: xfs(+) libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec joydev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt intel_pmc_bxt snd_pcm iTCO_vendor_support pktcdvd i2c_i801 lpc_ich snd_timer snd soundcore pcspkr virtio_balloon i2c_smbus binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_blk virtio_console virtio_scsi virtio_gpu net_failover failover
[  588.891393]  virtio_dma_buf nvme_tcp nvme_fabrics nvme_keyring serio_raw nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev qemu_fw_cfg
[  588.891416] ---[ end trace 0000000000000000 ]---
[  588.891418] RIP: 0010:__static_call_validate.cold+0x12/0x14
[  588.891421] Code: 48 c7 c7 d0 94 dc bb e8 a6 ea 00 00 48 8b 43 08 4c 8b 33 e9 6b f7 ee fe 48 89 fa 0f b6 f0 48 c7 c7 00 95 dc bb e8 88 ea 00 00 <0f> 0b 65 8b 35 bb 01 ed 44 48 c7 c7 48 96 dc bb e8 73 ea 00 00 fa
[  588.891422] RSP: 0000:ffffbb83833ef7d8 EFLAGS: 00010246
[  588.891424] RAX: 000000000000004a RBX: ffffffffbb2353e0 RCX: 0000000000000000
[  588.891425] RDX: 0000000000000000 RSI: ffffa07c3bd21900 RDI: ffffa07c3bd21900
[  588.891427] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  588.891428] R10: 65657274625f6164 R11: 302f32303478302b R12: ffffffffc1517162
[  588.891429] R13: 0000000000000000 R14: ffffffffc1274368 R15: ffffffffbc878220
[  588.891430] FS:  00007f3f4367c280(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
[  588.891432] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  588.891433] CR2: 00007f3ef9b60324 CR3: 000000000128c000 CR4: 00000000000006f0
[  589.059598] Oops: general protection fault, probably for non-canonical address 0x578a7de75fdfe55b: 0000 [#3] PREEMPT SMP PTI
[  589.059608] CPU: 0 UID: 0 PID: 3472 Comm: gdbus Tainted: G      D W         -------  ---  6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
[  589.059613] Tainted: [D]=DIE, [W]=WARN
[  589.059615] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
[  589.059617] RIP: 0010:kmem_cache_alloc_node_noprof+0x1cf/0x400
[  589.059624] Code: 84 12 ff ff ff 48 85 db 0f 84 09 ff ff ff 48 8b 03 48 c1 e8 36 41 39 c3 0f 85 f9 fe ff ff 41 8b 44 24 28 49 8b 34 24 48 01 f8 <48> 8b 18 48 89 c1 49 33 9c 24 b8 00 00 00 48 89 f8 48 0f c9 48 31
[  589.059626] RSP: 0018:ffffbb838286f990 EFLAGS: 00010202
[  589.059630] RAX: 578a7de75fdfe55b RBX: ffffee6f8418d740 RCX: 00000000ffffffff
[  589.059631] RDX: 00000000af4a6000 RSI: 0000000000040160 RDI: 578a7de75fdfe4eb
[  589.059633] RBP: ffffbb838286f9e8 R08: 0000000000400cc0 R09: 0000000000000003
[  589.059634] R10: ffffffffbae8686a R11: 00000000ffffffff R12: ffffa07bc02af100
[  589.059636] R13: 0000000000400cc0 R14: 00000000ffffffff R15: 00000000000000e8
[  589.059638] FS:  00007f842ebce6c0(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
[  589.059640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  589.059642] CR2: 000055a734042708 CR3: 000000010c3c2000 CR4: 00000000000006f0
[  589.059647] Call Trace:
[  589.059662]  <TASK>
[  589.059665]  ? __die_body.cold+0x19/0x27
[  589.059671]  ? die_addr+0x3c/0x60
[  589.059675]  ? exc_general_protection+0x17d/0x400
[  589.059682]  ? asm_exc_general_protection+0x26/0x30
[  589.059686]  ? __alloc_skb+0x14a/0x1a0
[  589.059693]  ? kmem_cache_alloc_node_noprof+0x1cf/0x400
[  589.059696]  ? __alloc_skb+0x14a/0x1a0
[  589.059699]  __alloc_skb+0x14a/0x1a0
[  589.059702]  alloc_skb_with_frags+0x67/0x2d0
[  589.059706]  sock_alloc_send_pskb+0x1f7/0x240
[  589.059710]  ? avc_has_perm+0x5d/0xe0
[  589.059713]  unix_stream_sendmsg+0x17d/0x680
[  589.059718]  ____sys_sendmsg+0x3a0/0x3d0
[  589.059721]  ___sys_sendmsg+0x9a/0xe0
[  589.059755]  __sys_sendmsg+0x87/0xe0
[  589.059761]  do_syscall_64+0x82/0x160
[  589.059767]  ? syscall_exit_to_user_mode+0x10/0x210
[  589.059772]  ? do_syscall_64+0x8e/0x160
[  589.059776]  ? syscall_exit_to_user_mode+0x10/0x210
[  589.059779]  ? do_syscall_64+0x8e/0x160
[  589.059782]  ? do_syscall_64+0x8e/0x160
[  589.059785]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  589.059789] RIP: 0033:0x7f84400b0dd2
[  589.059802] Code: 08 0f 85 51 3d ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 55 bf 01 00
[  589.059804] RSP: 002b:00007f842ebcda18 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[  589.059811] RAX: ffffffffffffffda RBX: 00007f842ebce6c0 RCX: 00007f84400b0dd2
[  589.059813] RDX: 0000000000004000 RSI: 00007f842ebcdae0 RDI: 0000000000000011
[  589.059814] RBP: 00007f842ebcda40 R08: 0000000000000000 R09: 0000000000000000
[  589.059816] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f842ebcdae0
[  589.059817] R13: 0000000000004000 R14: 0000000000000000 R15: 00007f842ebcdb90
[  589.059820]  </TASK>
[  589.059822] Modules linked in: xfs(+) libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec joydev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt intel_pmc_bxt snd_pcm iTCO_vendor_support pktcdvd i2c_i801 lpc_ich snd_timer snd soundcore pcspkr virtio_balloon i2c_smbus binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_blk virtio_console virtio_scsi virtio_gpu net_failover failover
[  589.059892]  virtio_dma_buf nvme_tcp nvme_fabrics nvme_keyring serio_raw nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev qemu_fw_cfg
[  589.059943] ---[ end trace 0000000000000000 ]---
[  589.059945] RIP: 0010:__static_call_validate.cold+0x12/0x14
[  589.059948] Code: 48 c7 c7 d0 94 dc bb e8 a6 ea 00 00 48 8b 43 08 4c 8b 33 e9 6b f7 ee fe 48 89 fa 0f b6 f0 48 c7 c7 00 95 dc bb e8 88 ea 00 00 <0f> 0b 65 8b 35 bb 01 ed 44 48 c7 c7 48 96 dc bb e8 73 ea 00 00 fa
[  589.059950] RSP: 0000:ffffbb83833ef7d8 EFLAGS: 00010246
[  589.059952] RAX: 000000000000004a RBX: ffffffffbb2353e0 RCX: 0000000000000000
[  589.059954] RDX: 0000000000000000 RSI: ffffa07c3bd21900 RDI: ffffa07c3bd21900
[  589.059955] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  589.059956] R10: 65657274625f6164 R11: 302f32303478302b R12: ffffffffc1517162
[  589.059957] R13: 0000000000000000 R14: ffffffffc1274368 R15: ffffffffbc878220
[  589.059959] FS:  00007f842ebce6c0(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
[  589.059960] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  589.059962] CR2: 000055a734042708 CR3: 000000010c3c2000 CR4: 00000000000006f0
Dec 03 01:16:50 localhost-live kernel: ------------[ cut here ]------------
Dec 03 01:16:50 localhost-live kernel: WARNING: CPU: 0 PID: 4325 at arch/x86/kernel/ftrace.c:100 ftrace_verify_code+0x4c/0x90
Dec 03 01:16:50 localhost-live kernel: Modules linked in: libfc scsi_transport_fc iscsi_ibft dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables qrtr snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec joydev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt intel_pmc_bxt snd_pcm iTCO_vendor_support pktcdvd i2c_i801 lpc_ich snd_timer snd soundcore pcspkr virtio_balloon i2c_smbus binfmt_misc nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci overlay squashfs isofs crc32c_intel sha512_ssse3 sha256_ssse3 virtio_net sha1_ssse3 virtio_blk virtio_console virtio_scsi virtio_gpu net_failover failover virtio_dma_buf
Dec 03 01:16:50 localhost-live kernel:  nvme_tcp nvme_fabrics nvme_keyring serio_raw nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev qemu_fw_cfg
Dec 03 01:16:50 localhost-live kernel: CPU: 0 UID: 0 PID: 4325 Comm: modprobe Not tainted 6.13.0-0.rc0.20241126git7eef7e306d3c.10.fc42.x86_64 #1
Dec 03 01:16:50 localhost-live kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
Dec 03 01:16:50 localhost-live kernel: RIP: 0010:ftrace_verify_code+0x4c/0x90
Dec 03 01:16:50 localhost-live kernel: Code: fe c6 44 24 07 00 48 89 c7 c7 44 24 03 00 00 00 00 e8 98 6e 30 00 48 85 c0 75 3e 8b 03 39 44 24 03 74 28 48 89 1d fc b3 7e 03 <0f> 0b b8 ea ff ff ff 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75
Dec 03 01:16:50 localhost-live kernel: RSP: 0018:ffffbb83833ef860 EFLAGS: 00010206
Dec 03 01:16:50 localhost-live kernel: RAX: 00000000b898a7e8 RBX: ffffffffbd7c2e08 RCX: 0000000000000010
Dec 03 01:16:50 localhost-live kernel: RDX: ffffa07bd290a880 RSI: 0000000000000005 RDI: ffffffffc1514064
Dec 03 01:16:50 localhost-live kernel: RBP: ffffffffc1514064 R08: 0000000000000000 R09: ffffa07c3bc3ee20
Dec 03 01:16:50 localhost-live kernel: R10: ffffbb83833ef8a0 R11: dead000000000040 R12: ffffffffbb60b5ea
Dec 03 01:16:50 localhost-live kernel: R13: 0000000002000000 R14: ffffa07bcbe50ce0 R15: 0000000000000f3e
Dec 03 01:16:51 localhost-live kernel: FS:  00007f65a27c8740(0000) GS:ffffa07c3bc00000(0000) knlGS:0000000000000000
Dec 03 01:16:51 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 03 01:16:51 localhost-live kernel: CR2: 00007f3ef9753f10 CR3: 00000000027a8000 CR4: 00000000000006f0
Dec 03 01:16:51 localhost-live kernel: Call Trace:
Dec 03 01:16:51 localhost-live kernel:  <TASK>
Dec 03 01:16:51 localhost-live kernel:  ? ftrace_verify_code+0x4c/0x90
Dec 03 01:16:51 localhost-live kernel:  ? __warn.cold+0x93/0xfa
Dec 03 01:16:51 localhost-live kernel:  ? ftrace_verify_code+0x4c/0x90
Dec 03 01:16:51 localhost-live kernel:  ? report_bug+0xff/0x140
Dec 03 01:16:51 localhost-live kernel:  ? handle_bug+0x58/0x90
Dec 03 01:16:51 localhost-live kernel:  ? exc_invalid_op+0x17/0x70
Dec 03 01:16:51 localhost-live kernel:  ? asm_exc_invalid_op+0x1a/0x20
Dec 03 01:16:51 localhost-live kernel:  ? ftrace_verify_code+0x4c/0x90
Dec 03 01:16:51 localhost-live kernel:  ftrace_modify_code_direct+0xf/0x70
Dec 03 01:16:51 localhost-live kernel:  ftrace_process_locs+0x313/0x5c0
Dec 03 01:16:51 localhost-live kernel:  ftrace_module_init+0x32/0x50
Dec 03 01:16:51 localhost-live kernel:  load_module+0x1b1a/0x23e0
Dec 03 01:16:51 localhost-live kernel:  ? __do_sys_init_module+0x17a/0x1b0
Dec 03 01:16:51 localhost-live kernel:  __do_sys_init_module+0x17a/0x1b0
Dec 03 01:16:51 localhost-live kernel:  do_syscall_64+0x82/0x160
Dec 03 01:16:51 localhost-live kernel:  ? __vm_munmap+0xb9/0x170
Dec 03 01:16:51 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Dec 03 01:16:51 localhost-live kernel:  ? do_syscall_64+0x8e/0x160
Dec 03 01:16:51 localhost-live kernel:  ? __alloc_pages_noprof+0x184/0x330
Dec 03 01:16:51 localhost-live kernel:  ? __mod_memcg_lruvec_state+0xdf/0x220
Dec 03 01:16:51 localhost-live kernel:  ? __count_memcg_events+0xc0/0x180
Dec 03 01:16:51 localhost-live kernel:  ? __lruvec_stat_mod_folio+0x83/0xd0
Dec 03 01:16:51 localhost-live kernel:  ? set_ptes.isra.0+0x41/0x90
Dec 03 01:16:51 localhost-live kernel:  ? do_anonymous_page+0xfc/0x8f0
Dec 03 01:16:51 localhost-live kernel:  ? __pte_offset_map+0x1b/0x180
Dec 03 01:16:51 localhost-live kernel:  ? __handle_mm_fault+0xb55/0xfd0
Dec 03 01:16:51 localhost-live kernel:  ? __count_memcg_events+0xc0/0x180
Dec 03 01:16:51 localhost-live kernel:  ? count_memcg_events.constprop.0+0x1a/0x30
Dec 03 01:16:51 localhost-live kernel:  ? handle_mm_fault+0x21b/0x330
Dec 03 01:16:51 localhost-live kernel:  ? do_user_addr_fault+0x55a/0x7b0
Dec 03 01:16:51 localhost-live kernel:  ? exc_page_fault+0x7e/0x180
Dec 03 01:16:51 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 03 01:16:51 localhost-live kernel: RIP: 0033:0x7f65a210069e
Dec 03 01:16:51 localhost-live kernel: Code: 48 8b 0d 75 37 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 37 0f 00 f7 d8 64 89 01 48
Dec 03 01:16:51 localhost-live kernel: RSP: 002b:00007ffdb33ac7f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
Dec 03 01:16:51 localhost-live kernel: RAX: ffffffffffffffda RBX: 0000557dfc420ca0 RCX: 00007f65a210069e
Dec 03 01:16:51 localhost-live kernel: RDX: 0000557dbeee3715 RSI: 0000000000751c7e RDI: 00007f65a0b79010
Dec 03 01:16:51 localhost-live kernel: RBP: 00007ffdb33ac8b0 R08: 0000557dfc420010 R09: 0000000000000007
Dec 03 01:16:51 localhost-live kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000557dbeee3715
Dec 03 01:16:51 localhost-live kernel: R13: 0000000000040000 R14: 0000557dfc420db0 R15: 0000000000000000
Dec 03 01:16:51 localhost-live kernel:  </TASK>
Dec 03 01:16:51 localhost-live kernel: ---[ end trace 0000000000000000 ]---

Welp, as this has got no traction anywhere, I guess I'll have to try and bisect it.

I think I have a workable procedure for this, but it'll take about 2.5 hours per step, and it'll be a 12 step bisect, so it's probably going to take me till next week.

Current bisect progress:

[adamw@xps13a linux ((a8cd9d4ce35e...)|BISECTING)]$ git bisect visualize --oneline
82475d111de7 selftests/damon/_debugfs_common: hide expected error message from test_write_result()
e06a6b55ed3d selftests/damon/huge_count_read_write: remove unnecessary debugging message
45488345d4b6 selftests/damon/huge_count_read_write: provide sufficiently large buffer for DEPRECATED file read
2b1d55498b67 memcg: factor out mem_cgroup_stat_aggregate()
e8c1a296b806 mm/show_mem: use str_yes_no() helper in show_free_areas()
1bc542c6a0d1 mm/vmscan: wake up flushers conditionally to avoid cgroup OOM
33d7f15f916e mm: use page->private instead of page->index in percpu
544ec0ed3764 mm: remove references to page->index in huge_memory.c
0386aaa6e9c8 bootmem: stop using page->index
68158bfa3dbd mm: mass constification of folio/page pointers
713da0b33b3e mm: renovate page_address_in_vma()
7d3e93eca3ca mm: use page_pgoff() in more places
f7470591f8db mm: convert page_to_pgoff() to page_pgoff()
e664c2cd98cb mm/zsmalloc: use memcpy_from/to_page whereever possible
91d0ec834786 zsmalloc: replace kmap_atomic with kmap_local_page
b7fc16a16b08 mm/codetag: uninline and move pgalloc_tag_copy and pgalloc_tag_split
4835f747d3ed alloc_tag: support for page allocation tag compression
42895a861244 alloc_tag: introduce pgtag_ref_handle to abstract page tag references
0f9b685626da alloc_tag: populate memory for module tags as needed
0db6f8d7820a alloc_tag: load module tags into separate contiguous memory
3e09c500bb5b alloc_tag: introduce shutdown_mem_profiling helper function
7c8c76e446ca maple_tree: add mas_for_each_rev() helper
5185e7f9f3bd x86/module: enable ROX caches for module text on 64 bit
2e45474ab14f execmem: add support for cache of large ROX pages
9bfc4824fd48 x86/module: prepare module loading for ROX allocations of text
0c6378a71574 arch: introduce set_direct_map_valid_noflush()
0c133b1e78cd module: prepare to handle ROX allocations for text
0c3beacf681e asm-generic: introduce text-patching.h
c82be0be9576 mm: vmalloc: don't account for number of nodes for HUGE_VMAP allocations
beeb9220c730 mm: vmalloc: group declarations depending on CONFIG_MMU together
906c38ff52e9 memcg: workingset: remove folio_memcg_rcu usage
642c66d84cd4 mm/vma: the pgoff is correct if can_merge_right
5ac87a885aec mm: defer second attempt at merge on mmap()
5a689bac0bbc mm: remove unnecessary reset state logic on merge new VMA
0d11630cc50a mm: refactor __mmap_region()
52956b0d7fb9 mm: isolate mmap internal logic to mm/vma.c
c14f8046cd7c tools: testing: add additional vma_internal.h stubs
a29c0e4b2e86 memcg-v1: remove memcg move locking code
cf4a65539c13 memcg-v1: no need for memcg locking for MGLRU
568bcf414849 memcg-v1: no need for memcg locking for writeback tracking
a8cd9d4ce35e (HEAD) memcg-v1: no need for memcg locking for dirty tracking
6b611388b626 memcg-v1: remove charge move code
aa6b4fdf5940 memcg-v1: fully deprecate move_charge_at_immigrate
729881ffd390 mm: shmem: fallback to page size splice if large folio has poisoned pages
477327e10639 mm/damon/vaddr: add 'nr_piece == 1' check in damon_va_evenly_split_region()
f3c7a1ede435 mm/damon/vaddr: fix issue in damon_va_evenly_split_region()
ab505e8be024 mm/page_alloc: use str_off_on() helper in build_all_zonelists()
8717734fdcc8 mm/memcontrol: fix seq_buf size to save memory when PAGE_SIZE is large
628e1b8c4777 mm: add missing mmu_notifier_clear_young for !MMU_NOTIFIER
3f1f947a322d tools/mm: free the allocated memory
39ac99852fca mm/page-writeback: raise wb_thresh to prevent write blocking with strictlimit
722376934b6c mm/memory.c: simplify pfnmap_lockdep_assert
ed265529d39a mm/codetag: fix arg in pgalloc_tag_copy alloc_tag_sub
78c018e3942c maple_tree: fix outdated flag name in comment
a284cb8472ec mm: shmem: improve the tmpfs large folio read performance
f3650ef89b87 mm: shmem: update iocb->ki_pos directly to simplify tmpfs read logic
b7f058f82739 mm: remove unused has_isolate_pageblock
5bb6345cd2ed mm: remove redundant condition for THP folio
4b6b0a5188c2 mm/mremap: remove goto from mremap_to()
58f1069311db mm/mremap: cleanup vma_to_resize()
38dc8f495246 maple_tree: remove sanity check from mas_wr_slot_store()
61e9df7085cc maple_tree: calculate new_end when needed
0938b1614648 mm: don't set readahead flag on a folio when lookahead_size > nr_to_read
4a9a27fdf7bf mm: shmem: remove __shmem_huge_global_enabled()
9884efd795cc mm: huge_memory: move file_thp_enabled() into huge_memory.c
5a90c155defa tmpfs: don't enable large folios if not supported
7146de5ff504 tools: testing: fix phys_addr_t size on 64-bit systems
f1001f3d3b68 mm/mglru: reset page lru tier bits when activating
d3ea85c6c5f7 mm: swap: use str_true_false() helper function
4a7bba1df001 percpu: add a test case for the specific 64-bit value addition
6c2625e9c2ef x86/percpu: fix clang warning when dealing with unsigned types
e4137f08816b mm, kasan, kmsan: instrument copy_from/to_kernel_nofault
908378a30b09 maple_tree: simplify mas_push_node()
4223dd93bfc9 maple_tree: total is not changed for nomem_one case
e852cb1d00ce maple_tree: clear request_count for new allocated one
0cc8d68abe2f maple_tree: root node could be handled by !p_slot too
0f85eb3395c7 maple_tree: add some alloc node test case
5b2100f723bd maple_tree: fix alloc node fail issue
f69c2e4dc684 mm/vmstat: defer the refresh_zone_stat_thresholds after all CPUs bringup

OK, this bisects to:

[adamw@xps13a linux ((5185e7f9f3bd...)|BISECTING)]$ git bisect bad
5185e7f9f3bd754ab60680814afd714e2673ef88 is the first bad commit
commit 5185e7f9f3bd754ab60680814afd714e2673ef88 (HEAD)
Author: Mike Rapoport (Microsoft) <rppt>
Date:   Wed Oct 23 19:27:11 2024 +0300

    x86/module: enable ROX caches for module text on 64 bit
    
    Enable execmem's cache of PMD_SIZE'ed pages mapped as ROX for module text
    allocations on 64 bit.
    
    Link: https://lkml.kernel.org/r/20241023162711.2579610-9-rppt@kernel.org
    Signed-off-by: Mike Rapoport (Microsoft) <rppt>
    Reviewed-by: Luis Chamberlain <mcgrof>
    Tested-by: kdevops <kdevops.dev>
    Cc: Andreas Larsson <andreas>
    Cc: Andy Lutomirski <luto>
    Cc: Ard Biesheuvel <ardb>
    Cc: Arnd Bergmann <arnd>
    Cc: Borislav Petkov (AMD) <bp>
    Cc: Brian Cain <bcain>
    Cc: Catalin Marinas <catalin.marinas>
    Cc: Christophe Leroy <christophe.leroy>
    Cc: Christoph Hellwig <hch>
    Cc: Dave Hansen <dave.hansen.com>
    Cc: Dinh Nguyen <dinguyen>
    Cc: Geert Uytterhoeven <geert>
    Cc: Guo Ren <guoren>
    Cc: Helge Deller <deller>
    Cc: Huacai Chen <chenhuacai>
    Cc: Ingo Molnar <mingo>
    Cc: Johannes Berg <johannes>
    Cc: John Paul Adrian Glaubitz <glaubitz.de>
    Cc: Kent Overstreet <kent.overstreet>
    Cc: Liam R. Howlett <Liam.Howlett>
    Cc: Mark Rutland <mark.rutland>
    Cc: Masami Hiramatsu (Google) <mhiramat>
    Cc: Matt Turner <mattst88>
    Cc: Max Filippov <jcmvbkbc>
    Cc: Michael Ellerman <mpe.au>
    Cc: Michal Simek <monstr>
    Cc: Oleg Nesterov <oleg>
    Cc: Palmer Dabbelt <palmer>
    Cc: Peter Zijlstra <peterz>
    Cc: Richard Weinberger <richard>
    Cc: Russell King <linux.uk>
    Cc: Song Liu <song>
    Cc: Stafford Horne <shorne>
    Cc: Steven Rostedt (Google) <rostedt>
    Cc: Suren Baghdasaryan <surenb>
    Cc: Thomas Bogendoerfer <tsbogend.de>
    Cc: Thomas Gleixner <tglx>
    Cc: Uladzislau Rezki (Sony) <urezki>
    Cc: Vineet Gupta <vgupta>
    Cc: Will Deacon <will>
    Signed-off-by: Andrew Morton <akpm>

 arch/x86/Kconfig   |  1 +
 arch/x86/mm/init.c | 37 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 37 insertions(+), 1 deletion(-)

(In reply to Adam Williamson from comment #5)

Hi,

I also responded on the kernel bug here: https://bugzilla.kernel.org/show_bug.cgi?id=219554#c7

> ah, yeah, I think that *is* relevant. I hadn't been able to reproduce this
> locally before, but I just set my local VM's CPU to Nehalem, and reproduced
> the UEFI version of this first try. I noticed the VM did indeed seem to be
> stuck at the bootloader install stage - it was not responding to any kind of
> input, couldn't switch to a VT - then after it sat like that for a few
> seconds or minutes (don't know how long it was sitting like that before I
> saw it), it spontaneously rebooted and then failed to boot properly from the
> disk, just like the openQA test in the video.

[...]


On "Nehalem" (copy/pasting from my kernel bug above):

The virtual CPU model "Nehalem" indeed looks not like a problem here.  I
think a reason "Nehalem" is used as the *default* because if your
underlying host OS is CentOS9 or RHEL9.

As you might know, RHEL and CentOS 9 use "x86-64-v2" 
micro-architecture[1].  Thus, the virtual CPU model, "Nehalem", is the
only common CPU model that works across both Intel and AMD hosts on
"x86-64-v2".  AMD had the necessary CPU features to run "Nehalem".  So, 
it is very handy in a mixed hardware (CI) setup that is running CentOS 9
or RHEL 9.  Upstream OpenStack Infrastructure team also uses "Nehalem" 
as default across its CI cluster for this reason[2].


In your openQA cluster with mixed CPUs, you may have to find a "lowest 
common denominator" CPU model that you can give to your guests.  More on 
it below; it's a bit involved.  An example here:

    https://kashyapc.fedorapeople.org/Calculate-CPU-hypervisor-baseline.html

[1] https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level

[2] "Use Nehalem CPU model by default" — 
    https://review.opendev.org/c/openstack/devstack/+/815020

A slightly related note on "Haswell" model for ELN in openQA:

I see in your commit[1] here to use "Haswell" QEMU model for ELN 
buildroot.  I guess you're using it because CentOS Stream 10 is based on
the "x86_64-v3" (the CPU baseline arch) — it requires something higher
than "Nehalem".

As you might know, the moment you add an AMD node to your cluster,
"Haswell" will not work.

The CPU arch for  x86_64-v2 got "lucky" in that QEMU's "Nehalem" model
could work on both Intel *and* AMD.  For x86_64-v3 (CentOS Stream 10 
and above), there's no such option.

[1] https://pagure.io/fedora-qa/fedora_openqa/c/4a888ce — Force QEMUCPU
    to "Haswell" for ELN jobs

No, Nehalem is not used as a default, I explicitly configure the openQA VMs to use Nehalem because we want to ensure Fedora doesn't inadvertently stop supporting it (this is explained above). There's been more than one case in the past where some package or other inadvertently started using features only available on newer CPUs, even though Nehalem is meant to be our baseline, so I do this as a check on that.

However, I have found that since setting openQA to use `-cpu host`, the bug is still happening sometimes, so that was a bit of a blind alley :/ I still couldn't reproduce it *locally* with `-cpu host`, though, I had to use `-cpu Nehalem` to reproduce it locally (on a Core i7-1250U). Haven't actually tried on my new laptop yet (which has a Ryzen AI 9 365).

"As you might know, the moment you add an AMD node to your cluster, "Haswell" will not work."

Yeah, we don't have any of those. It's all Intel.

(In reply to Adam Williamson from comment #14)
> No, Nehalem is not used as a default, I explicitly configure the openQA VMs
> to use Nehalem because we want to ensure Fedora doesn't inadvertently stop
> supporting it (this is explained above). 

Sorry, my phrasing was poor; I did read your comments 4 & 5 above.  I 
just meant it in this sense of default in *your* cluster, configured by  
you.  That's what I was implying with my link to the OpenStack Infra
commit, where we also had to explicitly configure "Nehalem" because
there was a mixed AMD and Intel hosts — you confirm, it's non-problem 
for Fedora's openQA.

> There's been more than one case in
> the past where some package or other inadvertently started using features
> only available on newer CPUs, even though Nehalem is meant to be our
> baseline, so I do this as a check on that.

Yes, I can completely understand that.

> However, I have found that since setting openQA to use `-cpu host`, the bug
> is still happening sometimes, so that was a bit of a blind alley :/ I still
> couldn't reproduce it *locally* with `-cpu host`, though, I had to use `-cpu
> Nehalem` to reproduce it locally (on a Core i7-1250U). Haven't actually
> tried on my new laptop yet (which has a Ryzen AI 9 365).

Hmm, so many variables in the air for this big, let's see if I got it 
right so far:

  - you've bisected the issue to the commit that you link to in 
    comment#11 above;

  - the patch that adds a test for PSE flag didn't help (as predicted on 
    the kernel bug):
    https://lore.kernel.org/lkml/20250103065631.26459-1-jgross@suse.com/T/#u

  - you ruled out on the kernel bug that it's not a problem specific to 
    any specific named model:
    https://bugzilla.kernel.org/show_bug.cgi?id=219554#c5

  - in your *cluster*, it is happening "sometimes" —  I saw your comment
    on the upstream bug, where you list the hardware you got
    (https://bugzilla.kernel.org/show_bug.cgi?id=219554#c9)

  - you can still reproduce the issue *locally*, with `-cpu host`

  - live *and* unattended installs seem to be affected

  - BIOS *and* UEFI installs are affected, but in a slightly different 
    way (BIOS: hangs; UEFI: reboots the system)

    * * *

I'm now curious if the 'mm' patch from Mike Rappaport that you shared on
the upstream bug still works for you: 

    https://bugzilla.kernel.org/show_bug.cgi?id=219554#c9

> "As you might know, the moment you add an AMD node to your cluster,
> "Haswell" will not work."
> 
> Yeah, we don't have any of those. It's all Intel.

Great; that makes matter more bearable.

>   - you've bisected the issue to the commit that you link to in 
>     comment#11 above;

Correct.

>  - the patch that adds a test for PSE flag didn't help (as predicted on 
>    the kernel bug):
>    https://lore.kernel.org/lkml/20250103065631.26459-1-jgross@suse.com/T/#u

Correct.

>  - you ruled out on the kernel bug that it's not a problem specific to 
>    any specific named model:
>    https://bugzilla.kernel.org/show_bug.cgi?id=219554#c5

Well, I ruled out the named models I tested. I didn't go higher, because I was testing on the cluster and reaching the top end of what the cluster actually *are*. I suspect if I tested on a newer CPU and went higher, I'd probably find a cut-off at some point.

>  - in your *cluster*, it is happening "sometimes" —  I saw your comment
>    on the upstream bug, where you list the hardware you got
>    (https://bugzilla.kernel.org/show_bug.cgi?id=219554#c9)

It is happening "sometimes", yes, and interestingly *seems* to happen only on Xeon Gold 5218 and Xeon Gold 6130 hosts, from a sample I did yesterday I couldn't find a case where the bug happened on the Xeon E5-2683v4 or Xeon E5-2680 hosts. On those systems, it happens whether we use `-cpu host` or `-cpu Nehalem`.

>  - you can still reproduce the issue *locally*, with `-cpu host`

No. Locally - on my i7-1250U - I can only reproduce with `-cpu Nehalem`, I cannot reproduce with `-cpu host`.

>  - live *and* unattended installs seem to be affected

Live and traditional installer, yeah, but it does seem to happen *more* on lives, at least that's my impression.

>  - BIOS *and* UEFI installs are affected, but in a slightly different 
>    way (BIOS: hangs; UEFI: reboots the system)

Yeah, although even in the UEFI case the system hangs for a long time, then *eventually* reboots. I'm guessing that's maybe some kinda cutout in the edk2 firmware, or something, I hadn't looked into it. There's a kernel trace in both cases.

> I'm now curious if the 'mm' patch from Mike Rappaport that you shared on
> the upstream bug still works for you: 

So far, yes, it still seems to be working. I've run 50 installs with it in openQA, and none of them hung.

It looks like the whole execmem ROX thing is being disabled for final - assuming https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=a9bbe341333109465605e8733bab0b573cddcc8c is pulled - so this should stop being an issue whenever that lands, until it's re-enabled, I guess?

(In reply to Adam Williamson from comment #16)

[...]

> >  - you ruled out on the kernel bug that it's not a problem specific to 
> >    any specific named model:
> >    https://bugzilla.kernel.org/show_bug.cgi?id=219554#c5
> 
> Well, I ruled out the named models I tested. I didn't go higher, because I
> was testing on the cluster and reaching the top end of what the cluster
> actually *are*. I suspect if I tested on a newer CPU and went higher, I'd
> probably find a cut-off at some point.

Yeah, it gets tedious to test these combinations.

> >  - in your *cluster*, it is happening "sometimes" —  I saw your comment
> >    on the upstream bug, where you list the hardware you got
> >    (https://bugzilla.kernel.org/show_bug.cgi?id=219554#c9)
> 
> It is happening "sometimes", yes, and interestingly *seems* to happen only
> on Xeon Gold 5218 and Xeon Gold 6130 hosts, from a sample I did yesterday I
> couldn't find a case where the bug happened on the Xeon E5-2683v4 or Xeon
> E5-2680 hosts. On those systems, it happens whether we use `-cpu host` or
> `-cpu Nehalem`.

I see, I don't know what to say about these Xeon Gold 5218, 6130

> >  - you can still reproduce the issue *locally*, with `-cpu host`
> 
> No. Locally - on my i7-1250U - I can only reproduce with `-cpu Nehalem`, I
> cannot reproduce with `-cpu host`.

Ah, thanks for correcting.

> >  - live *and* unattended installs seem to be affected
> 
> Live and traditional installer, yeah, but it does seem to happen *more* on
> lives, at least that's my impression.
> 
> >  - BIOS *and* UEFI installs are affected, but in a slightly different 
> >    way (BIOS: hangs; UEFI: reboots the system)
> 
> Yeah, although even in the UEFI case the system hangs for a long time, then
> *eventually* reboots. I'm guessing that's maybe some kinda cutout in the
> edk2 firmware, or something, I hadn't looked into it. There's a kernel trace
> in both cases.
> 
> > I'm now curious if the 'mm' patch from Mike Rappaport that you shared on
> > the upstream bug still works for you: 
> 
> So far, yes, it still seems to be working. I've run 50 installs with it in
> openQA, and none of them hung.

Nice; I hope the positive trend continues.

Overall, nice sleuthing.

Since this got reverted upstream, I think let's close this for now. If the rox stuff comes back without a fix for this, I can re-open.