Bug 2326393 - EMERG: SCSI driver issue with Fedora 40 - disk left bus - bus reset
Summary: EMERG: SCSI driver issue with Fedora 40 - disk left bus - bus reset
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 41
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL: https://forum.proxmox.com/threads/fed...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-11-15 03:44 UTC by customercare
Modified: 2025-12-16 17:34 UTC (History)
15 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2025-12-16 17:34:05 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description customercare 2024-11-15 03:44:13 UTC
1. Please describe the problem:

Short:

there seems to be an issue with scsi drivers under high load in Fedora 40, but it's not the kernel alone.
Long:

https://forum.proxmox.com/threads/fedora-40-vm-losses-disks-scsi-target2-0-0-target-reset-operation-started.157452/

Today, or better 6 hours ago, two Proxmox VMs ( kvm, qemu ) with SCSI simulated drives, got upgraded from F39 to F40.

Both VMs experienced a lot of these messages, anytime something io intensive was done, it took 4-10 minutes to recover:

Nov14 23:49] sd 2:0:0:0: [sda] tag#76 ABORT operation started
[  +4,898867] sd 2:0:0:0: ABORT operation timed-out.
[  +0,000008] sd 2:0:1:0: [sdb] tag#383 ABORT operation started
[  +4,914194] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000014] sd 2:0:1:0: [sdb] tag#322 ABORT operation started
[  +4,916147] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000007] sd 2:0:1:0: [sdb] tag#321 ABORT operation started
[  +4,914256] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000009] sd 2:0:1:0: [sdb] tag#320 ABORT operation started
[  +4,915142] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000006] sd 2:0:1:0: [sdb] tag#323 ABORT operation started
[  +4,915177] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000007] sd 2:0:1:0: [sdb] tag#324 ABORT operation started
[  +4,915221] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000007] sd 2:0:1:0: [sdb] tag#325 ABORT operation started
[Nov14 23:50] sd 2:0:1:0: ABORT operation timed-out.
[  +0,009594] scsi target2:0:0: TARGET RESET operation started
[  +4,905550] scsi target2:0:0: TARGET RESET operation timed-out.
[  +0,000005] scsi target2:0:1: TARGET RESET operation started
[ +34,407328] scsi target2:0:1: TARGET RESET operation timed-out.
[  +9,829432] sd 2:0:0:0: [sda] tag#76 ABORT operation started
[  +0,002336] sym0: SCSI BUS reset detected.
[  +0,002062] sd 2:0:0:0: ABORT operation complete.
[  +0,005062] sym0: SCSI BUS has been reset.
[  +3,193215] sd 2:0:1:0: Power-on or device reset occurred
[  +0,000057] sd 2:0:0:0: [sda] tag#76 BUS RESET operation started
[  +0,002319] sd 2:0:0:0: BUS RESET operation complete.
[  +0,000008] sym0: SCSI BUS reset detected.
[  +0,006945] sym0: SCSI BUS has been reset. 

(see DEV ML or URL for more log messages)


These kind of scsi issues had reported and fixed in 2015. If you google the SCSI error, you will see what i mean.

As it's the SCSI subsystem I rebooted  with an old F39 kernel, which was running stable at the same VM for days.
The VM was running on Proxmox since January this year, without having such issues. No issues at all.

The old Kernel had the same issues as the new F40 kernel, so, whatever changed with F40, it is triggered by a change outside of the kernel.
It does not mean, it's not an kernel issues, but the trigger must be somewhere else. That's also the reason i write it to the list, instead of opening a kernel issue alone.

The Proxmox system was not changed for the last months, so we can rule it out as a cause.

An easy way to reproduce it, was to TAR the diskcontent and syphone it i.e. via SSH out, or, doing a mysql_upgrade on a 7 GB database, where it stopped working and causing trouble in step 3, but step 4+ was ok, so that we only have READ IO. It looks like, it's overwelming the drivers capabilities processing such amount of data. it also disturbes writing to disks as well.

While the bus was reseting, cpu load went up and over 500+ (6 cores) , the real cpu usage was ~ 1% and io wait was 99%-100% perma on all cores.

The storage is a 7 GB/s NVME on Proxmox side, so disk io is not an issue. The proxmox host did not record any issues with the drives, while this was happening inside the vms. It's happend soley inside the Fedora 40 VM .


Attempted workaround:

The VMs disks got cloned with a SATA based driver and boot order was changed.
The SCSI drives are still connected, but not used.

Since this attempt, the system did not had an issue again nor logged something remotely similar.  In this timeframe, the SCSI based drivers would have had at least 3 major breakdowns.

It's now 4:43 in the morning here, so pls, do not expect a reaction soon, if you have questions.











2. What is the Version-Release number of the kernel:

at least 6.8.x, maybe older -> 6.11.x 

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

YES.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

YES. Two VMs acted more or less instantly the same way after the Upgrade to F40


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

We will never find out.


6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.


7. 
 +10,058980] sd 2:0:0:0: Power-on or device reset occurred
[  +0,016222] sd 2:0:1:0: [sdb] tag#393 timing out command, waited 180s
[  +0,000005] sd 2:0:1:0: [sdb] tag#393 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=279s
[  +0,000002] sd 2:0:1:0: [sdb] tag#393 CDB: Write(10) 2a 00 00 00 a6 40 00 02 00 00
[  +0,000001] I/O error, dev sdb, sector 42560 op 0x1:(WRITE) flags 0x8800 phys_seg 64 prio class 2
[  +0,000003] I/O error, dev sdb, sector 42560 op 0x1:(WRITE) flags 0x8800 phys_seg 64 prio class 2
[  +0,000002] sd 2:0:1:0: [sdb] tag#394 timing out command, waited 180s
[  +0,000002] sd 2:0:1:0: [sdb] tag#394 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=279s
[  +0,000001] sd 2:0:1:0: [sdb] tag#394 CDB: Write(10) 2a 00 00 00 a3 40 00 03 00 00
[  +0,000000] I/O error, dev sdb, sector 41792 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000001] I/O error, dev sdb, sector 41792 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000002] sd 2:0:1:0: [sdb] tag#395 timing out command, waited 180s
[  +0,000001] sd 2:0:1:0: [sdb] tag#395 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=279s
[  +0,000001] sd 2:0:1:0: [sdb] tag#395 CDB: Write(10) 2a 00 00 00 a0 40 00 03 00 00
[  +0,000000] I/O error, dev sdb, sector 41024 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000001] I/O error, dev sdb, sector 41024 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000013] sd 2:0:1:0: [sdb] tag#396 timing out command, waited 180s
[  +0,000002] sd 2:0:1:0: [sdb] tag#396 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=279s
[  +0,000000] sd 2:0:1:0: [sdb] tag#396 CDB: Write(10) 2a 00 00 00 9e 40 00 02 00 00
[  +0,000001] I/O error, dev sdb, sector 40512 op 0x1:(WRITE) flags 0x8800 phys_seg 64 prio class 2
[  +0,000000] I/O error, dev sdb, sector 40512 op 0x1:(WRITE) flags 0x8800 phys_seg 64 prio class 2
[  +0,000001] sd 2:0:1:0: [sdb] tag#397 timing out command, waited 180s
[  +0,000002] sd 2:0:1:0: [sdb] tag#397 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=279s
[  +0,000000] sd 2:0:1:0: [sdb] tag#397 CDB: Write(10) 2a 00 00 00 9b 40 00 03 00 00
[  +0,000001] I/O error, dev sdb, sector 39744 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000001] I/O error, dev sdb, sector 39744 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000001] sd 2:0:1:0: [sdb] tag#398 timing out command, waited 180s
[  +0,000001] sd 2:0:1:0: [sdb] tag#398 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=279s
[  +0,000000] sd 2:0:1:0: [sdb] tag#398 CDB: Write(10) 2a 00 00 00 98 40 00 03 00 00
[  +0,005144] sd 2:0:1:0: Power-on or device reset occurred

and it ended in a KERNEL CRASH finally: (thats the old fc39 kernel crashing, the newest f40 one did it presumably also)

[  +0,000001]  static_key_enable_cpuslocked+0x63/0x90
[  +0,000004]  static_key_enable+0x1a/0x20
[  +0,000001]  process_one_work+0x173/0x340
[  +0,000003]  worker_thread+0x27b/0x3a0
[  +0,000002]  ? __pfx_worker_thread+0x10/0x10
[  +0,000001]  kthread+0xe5/0x120
[  +0,000002]  ? __pfx_kthread+0x10/0x10
[  +0,000001]  ret_from_fork+0x31/0x50
[  +0,000002]  ? __pfx_kthread+0x10/0x10
[  +0,000001]  ret_from_fork_asm+0x1b/0x30
[  +0,000003]  </TASK>
[ +26,879827] watchdog: BUG: soft lockup - CPU#1 stuck for 118s! [kworker/1:3:120797]
[  +0,000015] Modules linked in: xt_owner ipt_REJECT nf_reject_ipv4 xt_connlimit nf_conncount nf_conntrack nf_defrag_ipv6 nf_defrag_ip
v4 ip6table_filter ip6_tables xt_multiport iptable_filter ip_tables cfg80211 rfkill sunrpc intel_rapl_msr intel_rapl_common intel_unco
re_frequency_common nfit libnvdimm kvm_intel kvm irqbypass i2c_piix4 rapl virtio_balloon joydev fuse loop nfnetlink crct10dif_pclmul c
rc32_pclmul crc32c_intel polyval_clmulni bochs polyval_generic ghash_clmulni_intel sha512_ssse3 drm_vram_helper sha256_ssse3 drm_ttm_h
elper virtio_net ttm sym53c8xx sha1_ssse3 net_failover scsi_transport_spi failover serio_raw ata_generic pata_acpi qemu_fw_cfg
[  +0,000024] CPU: 1 PID: 120797 Comm: kworker/1:3 Tainted: G             L     6.8.11-200.fc39.x86_64 #1
[  +0,000002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[  +0,000001] Workqueue: events netstamp_clear
[  +0,000006] RIP: 0010:smp_call_function_many_cond+0x12e/0x560
[  +0,000004] Code: 73 35 48 63 e8 49 8b 1f 48 81 fd 00 20 00 00 0f 83 f6 03 00 00 48 03 1c ed a0 4c c0 bb 8b 53 08 48 89 de 83 e2 01
74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb b1 48 83 c4 40 5b 5d 41 5c 41
[  +0,000001] RSP: 0018:ff36e8b984593d60 EFLAGS: 00000202
[  +0,000001] RAX: 0000000000000000 RBX: ff26fb3f2483ce00 RCX: 0000000000000001
[  +0,000001] RDX: 0000000000000001 RSI: ff26fb3f2483ce00 RDI: ff26fb3940156b98
[  +0,000000] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000006
[  +0,000001] R10: 0000000000000002 R11: 0000000000000000 R12: ff26fb3f248b5ac0
[  +0,000000] R13: 0000000000000001 R14: 0000000000000001 R15: ff26fb3f248b5ac0
[  +0,000001] FS:  0000000000000000(0000) GS:ff26fb3f24880000(0000) knlGS:0000000000000000
[  +0,000001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0,000001] CR2: 000014bce6b69d60 CR3: 000000010d428003 CR4: 0000000000771ef0
[  +0,000002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  +0,000001] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  +0,000000] PKRU: 55555554
[  +0,000001] Call Trace:
[  +0,000002]  <IRQ>
[  +0,000002]  ? watchdog_timer_fn+0x1e6/0x270
[  +0,000003]  ? __pfx_watchdog_timer_fn+0x10/0x10
[  +0,000002]  ? __hrtimer_run_queues+0x113/0x280
[  +0,000001]  ? ktime_get_update_offsets_now+0x49/0x110
[  +0,000004]  ? hrtimer_interrupt+0xf8/0x230
[  +0,000001]  ? __sysvec_apic_timer_interrupt+0x4d/0x140
[  +0,000003]  ? sysvec_apic_timer_interrupt+0x6d/0x90
[  +0,000002]  </IRQ>
[  +0,000001]  <TASK>
[  +0,000000]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  +0,000004]  ? smp_call_function_many_cond+0x12e/0x560
[  +0,000001]  ? __pfx_do_sync_core+0x10/0x10
[  +0,000003]  on_each_cpu_cond_mask+0x24/0x40
[  +0,000001]  text_poke_bp_batch+0xbe/0x300
[  +0,000001]  ? arch_jump_label_transform_queue+0x57/0x80
[  +0,000003]  text_poke_finish+0x1f/0x30
[  +0,000002]  arch_jump_label_transform_apply+0x1a/0x30
[  +0,000002]  static_key_enable_cpuslocked+0x63/0x90
[  +0,000003]  static_key_enable+0x1a/0x20
[  +0,000001]  process_one_work+0x173/0x340
[  +0,000004]  worker_thread+0x27b/0x3a0
[  +0,000002]  ? __pfx_worker_thread+0x10/0x10
[  +0,000001]  kthread+0xe5/0x120
[  +0,000002]  ? __pfx_kthread+0x10/0x10
[  +0,000001]  ret_from_fork+0x31/0x50
[  +0,000002]  ? __pfx_kthread+0x10/0x10 

Reproducible: Always

Comment 1 customercare 2024-11-18 08:55:27 UTC
@ Proxmox Users:

O== HOW TO FIX

You have to shut your VM down.
Create new harddisks with SATA drivers
Boot with a LiveImage for Fedora 38-40 ( does not matter )
become root
use "dd if=/dev/sd{old} of=/dev/sd{new} bs=128M status=progress" but MAKE SURE (with gnome-disks), you use the old and new drive ids correctly!!!
change the boot priority from SCSI to SATA drives 
Boot up
Delete the old SCSI drives

Comment 2 Aoife Moloney 2025-04-28 15:23:01 UTC
This message is a reminder that Fedora Linux 40 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 40 on 2025-05-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '40'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 40 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 3 customercare 2025-04-29 07:33:36 UTC
we never found out the change that caused this.

Comment 4 Adam Williamson (Red Hat non-Fedora) 2025-12-02 01:36:19 UTC
This message is a reminder that Fedora Linux 41 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 41 on 2025-12-15.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '41'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 41 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 5 Samyak Jain (RedHat) 2025-12-16 17:34:05 UTC
Fedora Linux 41 entered end-of-life (EOL) status on 2025-12-15.

Fedora Linux 41 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.