1. Please describe the problem: Connecting an NVMe-oF (either TCP or RDMA) volume while specifying --nr-poll-queues causes a NULL pointer dereference in the kernel. $ nvme connect -t tcp -s 4420 -n nqn.2018-09.io.blah:cnode1 -a 10.0.0.1 --nr-poll-queues=1 [ 298.069655] nvme nvme0: creating 113 I/O queues. [ 298.130679] nvme nvme0: mapped 112/0/1 default/read/poll queues. [ 298.206167] BUG: kernel NULL pointer dereference, address: 0000000000000034 2. What is the Version-Release number of the kernel: 5.16.18-200.fc35.x86_64 nvme-cli 1.11.1-4.fc35 On the target side: 5.16.18-200.fc35.x86_64 nvme-cli 1.11.1-4.fc35 nvmetcli 0.7-4.fc35 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : It worked on 5.16.18-200.fc35.x86_64, no idea when it started to appear first. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Create following NVMe-oF Target configuration using nvmet-cli and following configuration file. Either TCP or RDMA transport layer is fine for reproduction. { "ports": [ { "addr": { "adrfam": "ipv4", "traddr": "10.0.0.1", "trsvcid": "4420", "trtype": "tcp" }, "portid": 1, "referrals": [], "subsystems": [ "nqn.2018-09.io.blah:cnode1" ] } ], "hosts": [], "subsystems": [ { "allowed_hosts": [], "attr": { "allow_any_host": "1", "serial": "BLAH001", "version": "1.3" }, "namespaces": [ { "device": { "path": "/dev/nullb0", "uuid": "1393827f-9c71-49fa-b320-54d4395bc310" }, "enable": 1, "nsid": 1 } ], "nqn": "nqn.2018-09.io.blah:cnode1" } ] } Discover target subsystems from remote host: $ nvme discover -s 4420 -t tcp -a 10.0.0.1 Discovery Log Number of Records 2, Generation counter 64 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: unrecognized treq: not specified, sq flow control disable supported portid: 1 trsvcid: 4420 subnqn: nqn.2014-08.org.nvmexpress.discovery traddr: 10.0.0.1 sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified, sq flow control disable supported portid: 1 trsvcid: 4420 subnqn: nqn.2018-09.io.blah:cnode1 traddr: 10.0.0.1 sectype: none Try to connect: $ nvme connect -t tcp -s 4420 -n nqn.2018-09.io.blah:cnode1 -a 10.0.0.1 echo $? 0 $ nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 BLAH001 Linux 1 268.44 GB / 268.44 GB 512 B + 0 B 5.16.18- Disconnect and connect again with poll queues > 0: $ nvme disconnect -d /dev/nvme0 $ nvme connect -t tcp -s 4420 -n nqn.2018-09.io.blah:cnode1 -a 10.0.0.1 -P 1 Killed [ NULL ptr dereference ] It also happens with lower number of IO queues: $ nvme connect -t tcp -s 4420 -n nqn.2018-09.io.blah:cnode1 -a 10.0.0.1 -P 1 -i 1 Killed [ NULL ptr dereference ] 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Haven't checked that yet - will provide an update once it's verified. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. [ 298.069655] nvme nvme0: creating 113 I/O queues. [ 298.130679] nvme nvme0: mapped 112/0/1 default/read/poll queues. [ 298.206167] BUG: kernel NULL pointer dereference, address: 0000000000000034 [ 298.206241] #PF: supervisor read access in kernel mode [ 298.206281] #PF: error_code(0x0000) - not-present page [ 298.206319] PGD 1bdf19067 P4D 0 [ 298.206355] Oops: 0000 [#1] PREEMPT SMP NOPTI [ 298.206399] CPU: 7 PID: 2816 Comm: nvme Not tainted 5.16.18-200.fc35.x86_64 #1 [ 298.206469] Hardware name: Intel Corporation M50CYP2SBSTD/M50CYP2SBSTD, BIOS SE5C620.86B.01.01.0005.2202160810 02/16/2022 [ 298.206565] RIP: 0010:bio_poll+0x12/0x140 [ 298.206616] Code: 48 83 c4 20 4c 89 f7 5b 5d 41 5c 41 5d 41 5e 41 5f e9 82 ad 00 00 66 90 0f 1f 44 00 00 41 57 41 56 45 31 f6 41 55 41 54 55 53 <8b> 6f 34 83 fd ff 74 79 48 8b 47 08 48 8b 98 60 03 00 00 48 8b 43 [ 298.206779] RSP: 0018:ff345b92204cbc20 EFLAGS: 00010246 [ 298.206831] RAX: 0000000000000000 RBX: ff2b5046fd380000 RCX: 0000000000000000 [ 298.206897] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 298.206961] RBP: ff2b5046fd380000 R08: 0000000000000000 R09: ff2b50653f431700 [ 298.207025] R10: 000000000000b025 R11: 0000000000000000 R12: 0000000000000000 [ 298.207090] R13: ff2b50466ccb9fc0 R14: 0000000000000000 R15: 0000000000000000 [ 298.207155] FS: 00007fd5a4737800(0000) GS:ff2b50653f5c0000(0000) knlGS:0000000000000000 [ 298.207229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 298.207282] CR2: 0000000000000034 CR3: 000000011028e002 CR4: 0000000000771ee0 [ 298.207348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 298.207412] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 298.207476] PKRU: 55555554 [ 298.207503] Call Trace: [ 298.207531] <TASK> [ 298.207559] blk_execute_rq+0x69/0xa0 [ 298.207606] __nvme_submit_sync_cmd+0x169/0x210 [nvme_core] [ 298.207684] nvmf_connect_io_queue+0x11e/0x170 [nvme_fabrics] [ 298.207748] ? nvme_tcp_start_queue+0x1e/0x80 [nvme_tcp] [ 298.207803] nvme_tcp_start_queue+0x1e/0x80 [nvme_tcp] [ 298.207856] ? blk_mq_init_queue+0x35/0x60 [ 298.207903] nvme_tcp_setup_ctrl.cold+0x1c9/0x329 [nvme_tcp] [ 298.207961] ? _raw_spin_unlock_irqrestore+0x25/0x40 [ 298.208015] nvme_tcp_create_ctrl+0x313/0x370 [nvme_tcp] [ 298.208071] nvmf_dev_write+0x9ba/0xc31 [nvme_fabrics] [ 298.208127] ? inode_security+0x22/0x60 [ 298.208171] vfs_write+0xb9/0x2a0 [ 298.208210] ksys_write+0x4f/0xc0 [ 298.208247] do_syscall_64+0x38/0x90 [ 298.208289] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 298.208342] RIP: 0033:0x7fd5a4b02bc7 [ 298.208382] Code: 0b 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 298.210392] RSP: 002b:00007fff7d083d28 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 298.212335] RAX: ffffffffffffffda RBX: 00000000000000a4 RCX: 00007fd5a4b02bc7 [ 298.214278] RDX: 00000000000000a4 RSI: 00007fff7d085250 RDI: 0000000000000003 [ 298.216224] RBP: 00007fff7d085250 R08: 0000000000000001 R09: 000055dd2ad36fa1 [ 298.218160] R10: 0000000000000000 R11: 0000000000000246 R12: 000055dd2ad364b9 [ 298.220069] R13: 000055dd2ad59be0 R14: 000000000000000b R15: 00007fff7d087833 [ 298.222911] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr vfat fat rpcrdma intel_rapl_msr sunrpc intel_rapl_common rdma_ucm i10nm_edac nfit libnvdimm ib_srpt x86_pkg_temp_thermal intel_powerclamp ib_isert coretemp iscsi_target_mod target_core_mod ib_iser libiscsi ib_umad iTCO _wdt kvm_intel intel_pmc_bxt ib_ipoib scsi_transport_iscsi iTCO_vendor_support ipmi_ssif kvm irqbypass rapl irdma intel_cstate ice intel_uncore mlx5_ib mei_me i2c_i801 isst_if_mbox_pci isst_if_mmio pcspkr ib_uverbs ioatdma joydev iss t_if_common mei i2c_smbus acpi_ipmi intel_pmt intel_pch_thermal dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad nvmet_tcp nvmet_rdma nvmet nvme_tcp nvme_rdma nvme_fabrics nvme_core rdma_cm iw_cm ib_cm ib_core zram ip_tables xfs mlx5_core ast i2c_algo_bit drm_vram_helper drm_kms_helper [ 298.223029] cec drm_ttm_helper ttm mlxfw crct10dif_pclmul crc32_pclmul drm i40e tls crc32c_intel ghash_clmulni_intel psample pci_hyperv_intf wmi fuse [ 298.235590] CR2: 0000000000000034 [ 298.236610] ---[ end trace 1d42f71cc8fa8f2a ]--- [ 298.244550] RIP: 0010:bio_poll+0x12/0x140 [ 298.245240] Code: 48 83 c4 20 4c 89 f7 5b 5d 41 5c 41 5d 41 5e 41 5f e9 82 ad 00 00 66 90 0f 1f 44 00 00 41 57 41 56 45 31 f6 41 55 41 54 55 53 <8b> 6f 34 83 fd ff 74 79 48 8b 47 08 48 8b 98 60 03 00 00 48 8b 43 [ 298.246653] RSP: 0018:ff345b92204cbc20 EFLAGS: 00010246 [ 298.247356] RAX: 0000000000000000 RBX: ff2b5046fd380000 RCX: 0000000000000000 [ 298.248063] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 298.248770] RBP: ff2b5046fd380000 R08: 0000000000000000 R09: ff2b50653f431700 [ 298.249480] R10: 000000000000b025 R11: 0000000000000000 R12: 0000000000000000 [ 298.250187] R13: ff2b50466ccb9fc0 R14: 0000000000000000 R15: 0000000000000000 [ 298.250892] FS: 00007fd5a4737800(0000) GS:ff2b50653f5c0000(0000) knlGS:0000000000000000 [ 298.251615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 298.252340] CR2: 0000000000000034 CR3: 000000011028e002 CR4: 0000000000771ee0 [ 298.253070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 298.253796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 298.254516] PKRU: 55555554 [ 335.201789] xfs filesystem being remounted at /run/systemd/unit-root/var/tmp supports timestamps until 2038 (0x7fffffff) [ 335.204219] xfs filesystem being remounted at /run/systemd/unit-root/etc supports timestamps until 2038 (0x7fffffff) [ 335.210900] xfs filesystem being remounted at /run/systemd/unit-root/etc supports timestamps until 2038 (0x7fffffff) [ 335.237344] xfs filesystem being remounted at /run/systemd/unit-root/var/tmp supports timestamps until 2038 (0x7fffffff)
> 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Just checked. I installed 5.19.0-0.rc0.20220531git8ab2afa23bd1.8.fc37.x86_64 on the host system (the one doing nvme connect) and tested again - same result.
This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13. Fedora Linux 35 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.