1. Please describe the problem: found error info in the test filesystem fio test[1] [ 1154.946592] ================================================================== [ 1154.953815] BUG: KFENCE: use-after-free read in bfq_exit_icq_bfqq+0x132/0x270 [ 1154.962431] Use-after-free read at 0x0000000029335f71 (in kfence-#248): [ 1154.969043] bfq_exit_icq_bfqq+0x132/0x270 [ 1154.973142] bfq_exit_icq+0x5b/0x80 [ 1154.976635] exit_io_context+0x81/0xb0 [ 1154.980389] do_exit+0x74b/0xaf0 [ 1154.983619] kthread_exit+0x25/0x30 [ 1154.987115] kthread+0xc8/0x110 [ 1154.990259] ret_from_fork+0x1f/0x30 [ 1154.995337] kfence-#248: 0x00000000b346995a-0x000000007c67ce2d, size=568, cache=bfq_queue [ 1155.004984] allocated by task 19138 on cpu 13 at 499.287974s: [ 1155.010735] bfq_get_queue+0xe0/0x530 [ 1155.014406] bfq_get_bfqq_handle_split+0x75/0x120 [ 1155.019112] bfq_insert_requests+0x1d15/0x2710 [ 1155.023558] blk_mq_sched_insert_requests+0x5c/0x170 [ 1155.028523] blk_mq_flush_plug_list+0x115/0x2e0 [ 1155.033054] __blk_flush_plug+0xf2/0x130 [ 1155.036981] blk_finish_plug+0x25/0x40 [ 1155.040733] __iomap_dio_rw+0x520/0x7b0 [ 1155.044574] btrfs_dio_write+0x42/0x50 [ 1155.048327] btrfs_do_write_iter+0x2f4/0x5d0 [ 1155.052599] 0xffffffffc0998ab6 [ 1155.055761] 0xffffffffc0998d14 [ 1155.058907] process_one_work+0x1c4/0x380 [ 1155.062920] worker_thread+0x4d/0x380 [ 1155.066587] kthread+0xe6/0x110 [ 1155.069731] ret_from_fork+0x1f/0x30 [ 1155.074812] freed by task 19138 on cpu 13 at 1154.946573s: [ 1155.080308] bfq_put_queue+0x183/0x2c0 [ 1155.084058] bfq_exit_icq_bfqq+0x129/0x270 [ 1155.088158] bfq_exit_icq+0x5b/0x80 [ 1155.091651] exit_io_context+0x81/0xb0 [ 1155.095403] do_exit+0x74b/0xaf0 [ 1155.098637] kthread_exit+0x25/0x30 [ 1155.102129] kthread+0xc8/0x110 [ 1155.105274] ret_from_fork+0x1f/0x30 [ 1155.110355] CPU: 13 PID: 19138 Comm: kworker/dying Tainted: G B 6.1.0 #1 [ 1155.118438] Hardware name: Dell Inc. PowerEdge R640/0X45NX, BIOS 2.15.1 06/15/2022 [ 1155.126007] ================================================================== 2. What is the Version-Release number of the kernel: upstream branch 'block-6.2' into for-next: https://datawarehouse.cki-project.org/kcidb/checkouts/63789 https://datawarehouse.cki-project.org/kcidb/checkouts/63407 3. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: [1] https://gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip#storage/block/fs_fio 4. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. https://datawarehouse.cki-project.org/kcidb/tests/6445169 https://datawarehouse.cki-project.org/kcidb/tests/6406930
Closing this as the issue is on upstream kernel and not on Fedora Rawhide kernel. The issue has been reported upstream: https://lore.kernel.org/linux-block/CAHj4cs-MzFV6WTfveRXTARsik9wTGgado2U4vnT8oH6vmfFjzQ@mail.gmail.com/
Reopening this with Version 37 as I just saw this in dmesg in F37 running kernel-6.0.16-300.fc37.x86_64 which was just submitted for stable. [ 918.698510] ================================================================== [ 918.698517] BUG: KFENCE: use-after-free read in bfq_exit_icq_bfqq+0x132/0x270 [ 918.698526] Use-after-free read at 0x00000000515173b1 (in kfence-#99): [ 918.698530] bfq_exit_icq_bfqq+0x132/0x270 [ 918.698533] bfq_exit_icq+0x5b/0x80 [ 918.698537] exit_io_context+0x84/0xb0 [ 918.698543] do_exit+0x750/0xaf0 [ 918.698547] kthread_exit+0x25/0x30 [ 918.698552] kthread+0xc8/0x110 [ 918.698555] ret_from_fork+0x22/0x30 [ 918.698561] kfence-#99: 0x0000000035cdb515-0x000000000d97f733, size=576, cache=bfq_queue [ 918.698566] allocated by task 661 on cpu 0 at 133.830242s: [ 918.698573] bfq_get_queue+0xe0/0x530 [ 918.698577] bfq_get_bfqq_handle_split+0x75/0x120 [ 918.698580] bfq_insert_requests+0x7b9/0x2700 [ 918.698584] blk_mq_sched_insert_request+0xb9/0x130 [ 918.698588] blk_mq_submit_bio+0x49c/0x5b0 [ 918.698592] __submit_bio+0xf5/0x180 [ 918.698596] submit_bio_noacct_nocheck+0x1e8/0x2a0 [ 918.698599] blkg_async_bio_workfn+0xb4/0xd0 [ 918.698604] process_one_work+0x1c7/0x380 [ 918.698608] worker_thread+0x4d/0x380 [ 918.698612] kthread+0xe9/0x110 [ 918.698616] ret_from_fork+0x22/0x30 [ 918.698621] freed by task 661 on cpu 1 at 918.698497s: [ 918.698632] bfq_put_queue+0x191/0x2d0 [ 918.698635] bfq_exit_icq_bfqq+0x129/0x270 [ 918.698638] bfq_exit_icq+0x5b/0x80 [ 918.698641] exit_io_context+0x84/0xb0 [ 918.698645] do_exit+0x750/0xaf0 [ 918.698649] kthread_exit+0x25/0x30 [ 918.698652] kthread+0xc8/0x110 [ 918.698656] ret_from_fork+0x22/0x30 [ 918.698661] CPU: 1 PID: 661 Comm: kworker/dying Tainted: G OE 6.0.16-300.fc37.x86_64 #1 [ 918.698666] Hardware name: LENOVO 7630AF3/LENOVO, BIOS 5CKT77AUS 05/07/2012 [ 918.698669] ==================================================================
Created attachment 1935781 [details] journalctl --no-hostname -k
Confirmed here as well: [Thu Jan 5 06:29:34 2023] NFSD: Using nfsdcld client tracking operations. [Thu Jan 5 06:29:34 2023] NFSD: starting 90-second grace period (net f0000000) [Thu Jan 5 06:29:38 2023] NFSD: all clients done reclaiming, ending NFSv4 grace period (net f0000000) [Thu Jan 5 07:06:04 2023] ================================================================== [Thu Jan 5 07:06:04 2023] BUG: KFENCE: use-after-free read in bfq_exit_icq_bfqq+0x132/0x270 [Thu Jan 5 07:06:04 2023] Use-after-free read at 0x00000000dde068ff (in kfence-#232): [Thu Jan 5 07:06:04 2023] bfq_exit_icq_bfqq+0x132/0x270 [Thu Jan 5 07:06:04 2023] bfq_exit_icq+0x5b/0x80 [Thu Jan 5 07:06:04 2023] exit_io_context+0x81/0xb0 [Thu Jan 5 07:06:04 2023] do_exit+0x750/0xaf0 [Thu Jan 5 07:06:04 2023] kthread_exit+0x25/0x30 [Thu Jan 5 07:06:04 2023] kthread+0xc8/0x110 [Thu Jan 5 07:06:04 2023] ret_from_fork+0x1f/0x30 [Thu Jan 5 07:06:04 2023] kfence-#232: 0x00000000fd866421-0x000000001b90ec37, size=576, cache=bfq_queue [Thu Jan 5 07:06:04 2023] allocated by task 11 on cpu 7 at 42.003811s: [Thu Jan 5 07:06:04 2023] bfq_get_queue+0xe0/0x530 [Thu Jan 5 07:06:04 2023] bfq_get_bfqq_handle_split+0x75/0x120 [Thu Jan 5 07:06:04 2023] bfq_insert_requests+0x7b9/0x2700 [Thu Jan 5 07:06:04 2023] blk_mq_sched_insert_request+0xb6/0x130 [Thu Jan 5 07:06:04 2023] blk_mq_submit_bio+0x49c/0x5b0 [Thu Jan 5 07:06:04 2023] __submit_bio+0xf5/0x180 [Thu Jan 5 07:06:04 2023] submit_bio_noacct_nocheck+0x1e8/0x2a0 [Thu Jan 5 07:06:04 2023] xlog_state_release_iclog+0xd6/0x1a0 [xfs] [Thu Jan 5 07:06:04 2023] xlog_write_get_more_iclog_space+0x72/0xe0 [xfs] [Thu Jan 5 07:06:04 2023] xlog_write+0x2fc/0x420 [xfs] [Thu Jan 5 07:06:04 2023] xlog_cil_push_work+0x6ea/0x8b0 [xfs] [Thu Jan 5 07:06:04 2023] process_one_work+0x1c4/0x380 [Thu Jan 5 07:06:04 2023] worker_thread+0x4d/0x380 [Thu Jan 5 07:06:04 2023] kthread+0xe6/0x110 [Thu Jan 5 07:06:04 2023] ret_from_fork+0x1f/0x30 [Thu Jan 5 07:06:04 2023] freed by task 11 on cpu 2 at 2220.852912s: [Thu Jan 5 07:06:04 2023] bfq_put_queue+0x191/0x2d0 [Thu Jan 5 07:06:04 2023] bfq_exit_icq_bfqq+0x129/0x270 [Thu Jan 5 07:06:04 2023] bfq_exit_icq+0x5b/0x80 [Thu Jan 5 07:06:04 2023] exit_io_context+0x81/0xb0 [Thu Jan 5 07:06:04 2023] do_exit+0x750/0xaf0 [Thu Jan 5 07:06:04 2023] kthread_exit+0x25/0x30 [Thu Jan 5 07:06:04 2023] kthread+0xc8/0x110 [Thu Jan 5 07:06:04 2023] ret_from_fork+0x1f/0x30 [Thu Jan 5 07:06:04 2023] CPU: 2 PID: 11 Comm: kworker/dying Tainted: G O 6.0.16-300.fc37.x86_64 #1 [Thu Jan 5 07:06:04 2023] Hardware name: Dell Inc. PowerEdge R230/0FRVY0, BIOS 2.13.0 01/18/2022 [Thu Jan 5 07:06:04 2023] ==================================================================
This looks to be resolved in kernel-6.0.17-300.fc37.x86_64
I haven't seen it either in kernel-6.0.17-300.fc37.x86_64, though I've been running it for less than a day and I only saw it twice in 6.0.16 after running that for over 3 days. There have been no further comments at https://lore.kernel.org/linux-block/CAHj4cs-MzFV6WTfveRXTARsik9wTGgado2U4vnT8oH6vmfFjzQ@mail.gmail.com/ and it would be nice if someone could explicitly indicate it's fixed.
If I understand correctly, the following links indicate that the issue was indeed fixed between 6.0.16 and 6.0.17. https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=246cf66e300b76099b5dbd3fdd39e9a5dbc53f02 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/block/bfq-iosched.c?id=v6.0.17&id2=v6.0.16
I have not seen this since (up to 6.1.7). Closing.