Bug 2155748

Summary: qemu crash on void blk_drain(BlockBackend *): Assertion qemu_in_main_thread() failed
Product: Red Hat Enterprise Linux 9 Reporter: qing.wang <qinwang>
Component: qemu-kvmAssignee: Stefan Hajnoczi <stefanha>
qemu-kvm sub component: virtio-blk,scsi QA Contact: qing.wang <qinwang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aliang, chayang, coli, hreitz, jinzhao, juzhang, kwolf, lijin, mrezanin, pvlasin, qinwang, qizhu, stefanha, vgoyal, virt-maint, xuwei, ymankad, zhenyzha
Version: 9.2Keywords: CustomerScenariosInitiative, Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-7.2.0-12.el9_2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2176926 (view as bug list) Environment:
Last Closed: 2023-05-09 07:20:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2090990, 2176926    

Description qing.wang 2022-12-22 08:05:10 UTC
Description of problem:
The qemu hit crash issue when repeatly hotplug/unplug mutli disks.
qemu-kvm: ../block/block-backend.c:1780: void blk_drain(BlockBackend *): Assertion `qemu_in_main_thread()' failed.
  

BT:
              #0  0x00007f4b6a0a154c __pthread_kill_implementation (libc.so.6>
                #1  0x00007f4b6a054d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007f4b6a0287f3 abort (libc.so.6 + 0x287f3)
                #3  0x00007f4b6a02871b __assert_fail_base.cold (libc.so.6 + 0x2>
                #4  0x00007f4b6a04dce6 __assert_fail (libc.so.6 + 0x4dce6)
                #5  0x000055c811253ac0 blk_drain (qemu-kvm + 0x808ac0)
                #6  0x000055c810ecb951 scsi_device_purge_requests (qemu-kvm + 0>
                #7  0x000055c810ed6ab8 scsi_disk_reset (qemu-kvm + 0x48bab8)
                #8  0x000055c8111dc908 resettable_phase_hold (qemu-kvm + 0x7919>
                #9  0x000055c8111db7f0 device_cold_reset (qemu-kvm + 0x7907f0)
                #10 0x000055c8110bb1a6 virtio_scsi_handle_ctrl (qemu-kvm + 0x67>
                #11 0x000055c8110d458f virtio_queue_host_notifier_read (qemu-kv>
                #12 0x000055c811404a73 aio_dispatch_handler (qemu-kvm + 0x9b9a7>
                #13 0x000055c811405743 aio_poll (qemu-kvm + 0x9ba743)
                #14 0x000055c81121d532 iothread_run (qemu-kvm + 0x7d2532)
                #15 0x000055c81140926a qemu_thread_start (qemu-kvm + 0x9be26a)
                #16 0x00007f4b6a09f802 start_thread (libc.so.6 + 0x9f802)
                #17 0x00007f4b6a03f450 __clone3 (libc.so.6 + 0x3f450)


Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 9.2 Beta (Plow)
5.14.0-212.el9.x86_64
qemu-kvm-7.2.0-1.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20220826gitba0e0e4c6a-2.el9.noarch
libvirt-8.10.0-2.el9.x86_64
virtio-win-prewhql-0.1-231.iso


How reproducible:
30%

Steps to Reproduce:
1. Boot vm with multi disks.
/usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -machine q35,memory-backend=mem-machine_mem \
     -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
     -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
     -nodefaults \
     -device VGA,bus=pcie.0,addr=0x2 \
     -m 12288 \
     -object '{"qom-type": "memory-backend-ram", "size": 12884901888, "id": "mem-machine_mem"}'  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
     \
     -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
     -object '{"qom-type": "iothread", "id": "iothread0"}' \
     -object '{"qom-type": "iothread", "id": "iothread1"}' \
     -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0", "iothread": "iothread0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel910-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
     -device '{"id": "virtio_scsi_pci1", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-3", "addr": "0x0", "iothread": "iothread1"}' \
     -blockdev '{"node-name": "file_stg0", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg0.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg0", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg0"}' \
     -device '{"driver": "scsi-hd", "id": "stg0", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg0", "write-cache": "on", "lun": 0}' \
     -blockdev '{"node-name": "file_stg1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg1.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg1"}' \
     -device '{"driver": "scsi-hd", "id": "stg1", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg1", "write-cache": "on", "lun": 1}' \
     -blockdev '{"node-name": "file_stg2", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg2.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg2", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg2"}' \
     -device '{"driver": "scsi-hd", "id": "stg2", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg2", "write-cache": "on", "lun": 2}' \
     -blockdev '{"node-name": "file_stg3", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg3.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg3", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg3"}' \
     -device '{"driver": "scsi-hd", "id": "stg3", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg3", "write-cache": "on", "lun": 3}' \
     -blockdev '{"node-name": "file_stg4", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg4.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg4", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg4"}' \
     -device '{"driver": "scsi-hd", "id": "stg4", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg4", "write-cache": "on", "lun": 4}' \
     -blockdev '{"node-name": "file_stg5", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg5.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg5", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg5"}' \
     -device '{"driver": "scsi-hd", "id": "stg5", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg5", "write-cache": "on", "lun": 5}' \
     -blockdev '{"node-name": "file_stg6", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg6.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg6", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg6"}' \
     -device '{"driver": "scsi-hd", "id": "stg6", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg6", "write-cache": "on", "lun": 6}' \
     -blockdev '{"node-name": "file_stg7", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg7.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg7", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg7"}' \
     -device '{"driver": "scsi-hd", "id": "stg7", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg7", "write-cache": "on", "lun": 7}' \
     -blockdev '{"node-name": "file_stg8", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg8.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg8", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg8"}' \
     -device '{"driver": "scsi-hd", "id": "stg8", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg8", "write-cache": "on", "lun": 8}' \
     -blockdev '{"node-name": "file_stg9", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg9.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg9", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg9"}' \
     -device '{"driver": "scsi-hd", "id": "stg9", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg9", "write-cache": "on", "lun": 9}' \
     -blockdev '{"node-name": "file_stg10", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg10.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg10", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg10"}' \
     -device '{"driver": "scsi-hd", "id": "stg10", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg10", "write-cache": "on", "lun": 10}' \
     -blockdev '{"node-name": "file_stg11", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg11.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg11", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg11"}' \
     -device '{"driver": "scsi-hd", "id": "stg11", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg11", "write-cache": "on", "lun": 11}' \
     -blockdev '{"node-name": "file_stg12", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg12.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg12", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg12"}' \
     -device '{"driver": "scsi-hd", "id": "stg12", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg12", "write-cache": "on", "lun": 12}' \
     -blockdev '{"node-name": "file_stg13", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg13.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg13", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg13"}' \
     -device '{"driver": "scsi-hd", "id": "stg13", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg13", "write-cache": "on", "lun": 13}' \
     -blockdev '{"node-name": "file_stg14", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg14.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg14", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg14"}' \
     -device '{"driver": "scsi-hd", "id": "stg14", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg14", "write-cache": "on", "lun": 14}' \
     -blockdev '{"node-name": "file_stg15", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg15.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg15", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg15"}' \
     -device '{"driver": "scsi-hd", "id": "stg15", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg15", "write-cache": "on", "lun": 15}' \
     -blockdev '{"node-name": "file_stg16", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg16.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg16", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg16"}' \
     -device '{"driver": "scsi-hd", "id": "stg16", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg16", "write-cache": "on", "lun": 16}' \
     -blockdev '{"node-name": "file_stg17", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg17.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg17", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg17"}' \
     -device '{"driver": "scsi-hd", "id": "stg17", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg17", "write-cache": "on", "lun": 17}' \
     -blockdev '{"node-name": "file_stg18", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg18.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg18", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg18"}' \
     -device '{"driver": "scsi-hd", "id": "stg18", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg18", "write-cache": "on", "lun": 18}' \
     -blockdev '{"node-name": "file_stg19", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg19.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg19", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg19"}' \
     -device '{"driver": "scsi-hd", "id": "stg19", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg19", "write-cache": "on", "lun": 19}' \
     -blockdev '{"node-name": "file_stg20", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg20.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg20", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg20"}' \
     -device '{"driver": "scsi-hd", "id": "stg20", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg20", "write-cache": "on", "lun": 20}' \
     -blockdev '{"node-name": "file_stg21", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg21.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg21", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg21"}' \
     -device '{"driver": "scsi-hd", "id": "stg21", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg21", "write-cache": "on", "lun": 21}' \
     -blockdev '{"node-name": "file_stg22", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg22.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg22", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg22"}' \
     -device '{"driver": "scsi-hd", "id": "stg22", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg22", "write-cache": "on", "lun": 22}' \
     -blockdev '{"node-name": "file_stg23", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg23.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg23", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg23"}' \
     -device '{"driver": "scsi-hd", "id": "stg23", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg23", "write-cache": "on", "lun": 23}' \
     -blockdev '{"node-name": "file_stg24", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg24.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg24", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg24"}' \
     -device '{"driver": "scsi-hd", "id": "stg24", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg24", "write-cache": "on", "lun": 24}' \
     -blockdev '{"node-name": "file_stg25", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg25.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg25", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg25"}' \
     -device '{"driver": "scsi-hd", "id": "stg25", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg25", "write-cache": "on", "lun": 25}' \
     -blockdev '{"node-name": "file_stg26", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg26.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg26", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg26"}' \
     -device '{"driver": "scsi-hd", "id": "stg26", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg26", "write-cache": "on", "lun": 26}' \
     -blockdev '{"node-name": "file_stg27", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg27.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg27", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg27"}' \
     -device '{"driver": "scsi-hd", "id": "stg27", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg27", "write-cache": "on", "lun": 27}' \
     -blockdev '{"node-name": "file_stg28", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg28.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg28", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg28"}' \
     -device '{"driver": "scsi-hd", "id": "stg28", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg28", "write-cache": "on", "lun": 28}' \
     -blockdev '{"node-name": "file_stg29", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg29.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg29", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg29"}' \
     -device '{"driver": "scsi-hd", "id": "stg29", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg29", "write-cache": "on", "lun": 29}' \
     -blockdev '{"node-name": "file_stg30", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg30.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg30", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg30"}' \
     -device '{"driver": "scsi-hd", "id": "stg30", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg30", "write-cache": "on", "lun": 30}' \
     -blockdev '{"node-name": "file_stg31", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg31.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg31", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg31"}' \
     -device '{"driver": "scsi-hd", "id": "stg31", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg31", "write-cache": "on", "lun": 31}' \
     -blockdev '{"node-name": "file_stg32", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg32.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg32", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg32"}' \
     -device '{"driver": "scsi-hd", "id": "stg32", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg32", "write-cache": "on", "lun": 32}' \
     -blockdev '{"node-name": "file_stg33", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg33.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg33", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg33"}' \
     -device '{"driver": "scsi-hd", "id": "stg33", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg33", "write-cache": "on", "lun": 33}' \
     -blockdev '{"node-name": "file_stg34", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg34.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg34", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg34"}' \
     -device '{"driver": "scsi-hd", "id": "stg34", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg34", "write-cache": "on", "lun": 34}' \
     -blockdev '{"node-name": "file_stg35", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg35.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg35", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg35"}' \
     -device '{"driver": "scsi-hd", "id": "stg35", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg35", "write-cache": "on", "lun": 35}' \
     -blockdev '{"node-name": "file_stg36", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg36.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg36", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg36"}' \
     -device '{"driver": "scsi-hd", "id": "stg36", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg36", "write-cache": "on", "lun": 36}' \
     -blockdev '{"node-name": "file_stg37", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg37.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg37", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg37"}' \
     -device '{"driver": "scsi-hd", "id": "stg37", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg37", "write-cache": "on", "lun": 37}' \
     -blockdev '{"node-name": "file_stg38", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg38.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg38", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg38"}' \
     -device '{"driver": "scsi-hd", "id": "stg38", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg38", "write-cache": "on", "lun": 38}' \
     -blockdev '{"node-name": "file_stg39", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg39.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg39", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg39"}' \
     -device '{"driver": "scsi-hd", "id": "stg39", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg39", "write-cache": "on", "lun": 39}' \
     -blockdev '{"node-name": "file_stg40", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/mstg40.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_stg40", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg40"}' \
     -device '{"driver": "scsi-hd", "id": "stg40", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg40", "write-cache": "on", "lun": 40}' \
     -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
     -device virtio-net-pci,mac=9a:c1:77:39:18:55,id=idNY9b9u,netdev=idpVDVCD,bus=pcie-root-port-4,addr=0x0  \
     -netdev tap,id=idpVDVCD,vhost=on  \
     -vnc :5  \
     -monitor stdio \
     -qmp tcp:0:5955,server,nowait \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=6



2. run guest_sg_luns.sh in guest
cat ./qemu/deps/guest_sg_luns.sh
trap 'kill $(jobs -p)' EXIT SIGINT

for i in `seq 0 32` ; do
  while true ; do
    sg_luns /dev/sdb > /dev/null 2>&1
  done &
done
echo "wait"
wait


3.delete all data disks

{"execute": "device_del", "arguments": {"id": "stg1"}}
....
{"execute": "device_del", "arguments": {"id": "stg40"}}

4.sleep 3 seconds

5.replug all data disks
{"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg1", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg1", "write-cache": "on", "lun": 1}, "id": "lDc9ERuj"}
....
{"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg40", "bus": "virtio_scsi_pci1.0", "drive": "drive_stg1", "write-cache": "on", "lun": 40}, "id": "lDc9ERuj"}

6.sleep 3 seconds


7.repeat step 3-6 500 times

Actual results:
qemu crash 

Expected results:
qemu should keep running


Additional info:

Comment 3 Vivek Goyal 2023-01-03 21:39:32 UTC
Is this something new? Is this a regression?

Can you try with older qemu versions and figure out when was this issue introduced?

Comment 4 Stefan Hajnoczi 2023-01-04 16:48:16 UTC
I think the assertion is too strict. I'll send a QEMU patch for Kevin and Emanuele to review.

Comment 8 qing.wang 2023-01-06 10:32:52 UTC
(In reply to Vivek Goyal from comment #3)
> Is this something new? Is this a regression?
> 
> Can you try with older qemu versions and figure out when was this issue
> introduced?

maybe it is not a regression issue, if we use an older version (7.1-0) it will hit 
Bug 2090139 - qemu crash with error scsi_req_unref(SCSIRequest *): Assertion `req->refcount > 0' failed or scsi_dma_complete(void *, int): Assertion `r->req.aiocb != NULL' failed

Comment 10 Stefan Hajnoczi 2023-01-19 21:48:09 UTC
New upstream patch:
https://lore.kernel.org/qemu-devel/20230119214326.1339323-1-stefanha@redhat.com/T/#u

Comment 26 Stefan Hajnoczi 2023-02-27 18:41:22 UTC
Backport posted: https://gitlab.com/redhat/rhel/src/qemu-kvm/qemu-kvm/-/merge_requests/261

Exception requested because this bug crashes QEMU, which may cause customers to lose data.

I have set needinfo from Qing Wang just to make sure everyone is aware that this BZ is being merged for RHEL 9.2 qemu-kvm and can be verified soon.

Comment 29 Kevin Wolf 2023-02-28 11:01:01 UTC
*** Bug 2090139 has been marked as a duplicate of this bug. ***

Comment 40 Yanan Fu 2023-03-14 02:46:29 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 41 qing.wang 2023-03-17 09:00:02 UTC
Passed test on 
Red Hat Enterprise Linux release 9.2 Beta (Plow)
5.14.0-283.el9.x86_64
qemu-kvm-7.2.0-12.el9_2.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
virtio-win-prewhql-0.1-234.iso


python ConfigTest.py --testcase=multi_disk_wild_hotplug --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.9.1.0 --driveformat=virtio_scsi --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="vm_mem_limit = 12G\nimage_aio=threads" --firmware=default_bios --netdst=virbr0 --nrepeat=10

Comment 45 qing.wang 2023-03-21 08:01:58 UTC
passed the test in comment #41

Comment 47 errata-xmlrpc 2023-05-09 07:20:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2162