Created attachment 1847176 [details]
kernel_config

System: Fedora35 (VM, qemu 6.0.0). 
Kernel: 5.15.6 (5.15.6-200.fc35.x86_64)
GCC: 11.2.1 (gcc-11.2.1-7.fc35.x86_64)

When attempting to use infiniband's rxe device, the process gets stuck and eventually kernel's watchdog reports a soft lockup of the target cpu. Trace looks like the following:

[Tue Dec 21 09:20:28 2021] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [perf:12617]
[Tue Dec 21 09:20:28 2021] Modules linked in: nvme_rdma nvme_fabrics rdma_rxe ip6_udp_tunnel udp_tunnel ib_umad uio_pci_generic nbd rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_core rfkill usdm_drv(OE) intel_qat(OE) vfat fat intel_rapl_msr intel_rapl_common cirrus pcspkr drm_kms_helper joydev virtio_balloon cec i2c_piix4 drm zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel nvme ghash_clmulni_intel virtio_net ixgbevf nvme_core net_failover serio_raw failover virtio_blk ata_generic floppy pata_acpi qemu_fw_cfg be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [last unloaded: ip6_udp_tunnel]
[Tue Dec 21 09:20:28 2021] CPU: 0 PID: 12617 Comm: perf Tainted: G           OE     5.15.6-200.fc35.x86_64 #1
[Tue Dec 21 09:20:28 2021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[Tue Dec 21 09:20:28 2021] RIP: 0010:rxe_icrc_hdr.isra.0+0xc/0x160 [rdma_rxe]
[Tue Dec 21 09:20:28 2021] Code: ea 4c 89 e6 89 ef e8 d3 66 b8 ef eb b1 e8 0c 5a 18 f0 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 53 0f b7 f6 48 89 cb <48> 01 f2 48 83 ec 48 65 48 8b 04 25 28 00 00 00 48 89 44 24 40 31
[Tue Dec 21 09:20:28 2021] RSP: 0018:ffffb157c0c57b40 EFLAGS: 00000246
[Tue Dec 21 09:20:28 2021] RAX: 0000000000000050 RBX: ffff8f30401f9f28 RCX: ffff8f30401f9f28
[Tue Dec 21 09:20:28 2021] RDX: ffff8f3049696400 RSI: 000000000000001e RDI: 0000000000000008
[Tue Dec 21 09:20:28 2021] RBP: ffff8f30401f9f28 R08: 0000000000000000 R09: 0000000000000006
[Tue Dec 21 09:20:28 2021] R10: ffff8f3041dee000 R11: 0000000000000040 R12: ffff8f30401f9f00
[Tue Dec 21 09:20:28 2021] R13: ffff8f31efd49000 R14: ffff8f30401f9f00 R15: 0000000000000012
[Tue Dec 21 09:20:28 2021] FS:  00007f3f8f1eb9c0(0000) GS:ffff8f3273800000(0000) knlGS:0000000000000000
[Tue Dec 21 09:20:28 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Dec 21 09:20:28 2021] CR2: 00007f3f90d85e20 CR3: 000000016c752005 CR4: 0000000000170ef0
[Tue Dec 21 09:20:28 2021] Call Trace:
[Tue Dec 21 09:20:28 2021]  <TASK>
[Tue Dec 21 09:20:28 2021]  rxe_icrc_check+0x38/0x100 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  rxe_rcv+0x1e8/0x860 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  ? rxe_icrc_hdr.isra.0+0xfa/0x160 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  rxe_xmit_packet+0xc4/0x270 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  rxe_requester+0x5fa/0xf20 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  rxe_do_task+0x83/0xe0 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  rxe_post_send+0x356/0x5b0 [rdma_rxe]
[Tue Dec 21 09:20:28 2021]  ? rdma_lookup_get_uobject+0x3b/0x190 [ib_uverbs]
[Tue Dec 21 09:20:28 2021]  ib_uverbs_post_send+0x5c3/0x640 [ib_uverbs]
[Tue Dec 21 09:20:28 2021]  ? __mod_lruvec_page_state+0x5d/0x90
[Tue Dec 21 09:20:28 2021]  ib_uverbs_write+0x3a2/0x4d0 [ib_uverbs]
[Tue Dec 21 09:20:28 2021]  vfs_write+0xbc/0x260
[Tue Dec 21 09:20:28 2021]  ksys_write+0x87/0xc0
[Tue Dec 21 09:20:28 2021]  do_syscall_64+0x3b/0x90
[Tue Dec 21 09:20:28 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[Tue Dec 21 09:20:28 2021] RIP: 0033:0x7f3f8f71994f
[Tue Dec 21 09:20:28 2021] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 c9 87 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c 88 f8 ff 48
[Tue Dec 21 09:20:28 2021] RSP: 002b:00007ffcd30e4cf0 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[Tue Dec 21 09:20:28 2021] RAX: ffffffffffffffda RBX: 0000000000a44f80 RCX: 00007f3f8f71994f
[Tue Dec 21 09:20:28 2021] RDX: 0000000000000020 RSI: 00007ffcd30e4d30 RDI: 000000000000000f
[Tue Dec 21 09:20:28 2021] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000a46270
[Tue Dec 21 09:20:28 2021] R10: 00007f3f8f7ce3e0 R11: 0000000000000293 R12: 00007f3f8dada180
[Tue Dec 21 09:20:28 2021] R13: 0000000000000040 R14: 0000000000000000 R15: 0000000000426190
[Tue Dec 21 09:20:28 2021]  </TASK>

The process is stuck on the kernel side indefinitely and bashing it even with the SIGKILL doesn't help (watchdog keeps dumping the traces, continuously reporting soft lockups).

The "perf" process is an application meant to run io workloads on the nvmf-rdma target (https://github.com/spdk/spdk/blob/master/test/nvmf/target/nvmf_example.sh#L59, https://github.com/spdk/spdk/tree/master/examples/nvme/perf for details).

The reason why I submit this as a bug for fedora35 is that it seems it's triggered only under this particular environment. I built several versions of the upstream kernel (including the latest, master branch) and tested it under fedora33 and fedora35 - .config I used on fedora35 attached, it differs from fedora33's only in some gcc-specific options which are available in fedora35's gcc:

--- ./.config_fedora35  2021-12-13 08:03:59.317649730 +0000
+++ ./.config   2021-12-10 18:16:20.970672108 +0000
@@ -2,18 +2,17 @@
 # Automatically generated file; DO NOT EDIT.
 # Linux/x86 5.16.0-rc4 Kernel Configuration
 #
-CONFIG_CC_VERSION_TEXT="gcc (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7)"
+CONFIG_CC_VERSION_TEXT="gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)"
 CONFIG_CC_IS_GCC=y
-CONFIG_GCC_VERSION=110201
+CONFIG_GCC_VERSION=100301
 CONFIG_CLANG_VERSION=0
 CONFIG_AS_IS_GNU=y
-CONFIG_AS_VERSION=23700
+CONFIG_AS_VERSION=23500
 CONFIG_LD_IS_BFD=y
-CONFIG_LD_VERSION=23700
+CONFIG_LD_VERSION=23500
 CONFIG_LLD_VERSION=0
 CONFIG_CC_CAN_LINK=y
 CONFIG_CC_HAS_ASM_GOTO=y
-CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
 CONFIG_CC_HAS_ASM_INLINE=y
 CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
 CONFIG_IRQ_WORK=y
@@ -4147,8 +4146,6 @@
 CONFIG_INIT_STACK_NONE=y
 # CONFIG_INIT_ON_ALLOC_DEFAULT_ON is not set
 # CONFIG_INIT_ON_FREE_DEFAULT_ON is not set
-CONFIG_CC_HAS_ZERO_CALL_USED_REGS=y
-# CONFIG_ZERO_CALL_USED_REGS is not set
 # end of Memory initialization
 # end of Kernel hardening options
 # end of Security options
@@ -4527,8 +4524,6 @@
 CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
 # CONFIG_UBSAN is not set
 CONFIG_HAVE_ARCH_KCSAN=y
-CONFIG_HAVE_KCSAN_COMPILER=y
-# CONFIG_KCSAN is not set
 # end of Generic Kernel Debugging Instruments

With these kernels, same issue occurred under fedora35, but there was none under fedora33. Moreover, I took fedora33's kernel rpms (5.12.10-200.fc33.x86_64) and installed it under fedora35 - lockup still occurs, but on fedora33 it works like a charm. 

This actually makes me wonder if this is indeed a kernel issue or something with the newer tooling (like gcc) used for building the kernel image and the affected application and which is shipped with fedora35. 

Any hints, thoughts on the above would be greatly appreciated. :) We also track this issue under https://github.com/spdk/spdk/issues/2277.