Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1812399 - Qemu crash when detach disk with cache="none" discard="ignore" io="native"
Summary: Qemu crash when detach disk with cache="none" discard="ignore" io="native"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.2
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 8.0
Assignee: Maxim Levitsky
QA Contact: qing.wang
URL:
Whiteboard:
: 1844343 (view as bug list)
Depends On:
Blocks: 1844343 1844983 1869994 1888131 1897525
TreeView+ depends on / blocked
 
Reported: 2020-03-11 08:57 UTC by Han Han
Modified: 2021-04-13 02:17 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-5.1.0-15.module+el8.3.1+8772+a3fdeccd
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1888131 (view as bug list)
Environment:
Last Closed: 2021-02-22 15:39:38 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
vm xml, disk xml, all threads backtrace (8.04 KB, application/gzip)
2020-03-11 08:57 UTC, Han Han
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 5341841 0 None None None 2020-11-13 12:11:19 UTC

Description Han Han 2020-03-11 08:57:02 UTC
Created attachment 1669195 [details]
vm xml, disk xml, all threads backtrace

Description of problem:
As subject

Version-Release number of selected component (if applicable):
libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64
qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start an VM
# virsh create pc.xml

2. Prepare a disk with cache="none" discard="ignore" io="native"
<disk type="file" device="disk">
      <driver name="qemu" type="raw" cache="none" discard="ignore" io="native"/>
      <source file="/tmp/disk-raw">
      </source>
      <backingStore/>
      <target dev="sdb" bus="scsi"/>
    </disk>


3. Live attach/detach the disk in loop:
# while true;do virsh attach-device pc disk.xml; virsh detach-device pc disk.xml;done

Wait until vm crashes at detach.

Backtrace:
(gdb) bt
#0  0x0000557abd252603 in qemu_strnlen (max_len=8, s=0x0) at util/cutils.c:109
#1  0x0000557abd252603 in strpadcpy (buf=buf@entry=0x557abf0de008 "", buf_size=buf_size@entry=8, str=0x0, pad=pad@entry=32 ' ') at util/cutils.c:39
#2  0x0000557abd0df735 in scsi_disk_emulate_inquiry (outbuf=<optimized out>, req=0x557abf11fe00) at hw/scsi/scsi-disk.c:784
#3  0x0000557abd0df735 in scsi_disk_emulate_command (req=0x557abf11fe00, buf=<optimized out>) at hw/scsi/scsi-disk.c:1926
#4  0x0000557abd0e32e6 in scsi_req_enqueue (req=req@entry=0x557abf11fe00) at hw/scsi/scsi-bus.c:1303
#5  0x0000557abcf8668a in virtio_scsi_handle_cmd_req_submit (s=0x557abf98e7f0, req=<optimized out>) at /usr/src/debug/qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64/hw/scsi/virtio-scsi.c:634
#6  0x0000557abcf8668a in virtio_scsi_handle_cmd_vq (s=s@entry=0x557abf98e7f0, vq=vq@entry=0x7fafc865a140) at /usr/src/debug/qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64/hw/scsi/virtio-scsi.c:634
#7  0x0000557abcf873de in virtio_scsi_data_plane_handle_cmd (vdev=<optimized out>, vq=0x7fafc865a140) at /usr/src/debug/qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64/hw/scsi/virtio-scsi-dataplane.c:60
#8  0x0000557abcf94f0e in virtio_queue_notify_aio_vq (vq=<optimized out>) at /usr/src/debug/qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64/hw/virtio/virtio.c:2243
#9  0x0000557abd256b92 in aio_dispatch_handlers (ctx=ctx@entry=0x557abeeb8810) at util/aio-posix.c:429
#10 0x0000557abd257797 in aio_poll (ctx=ctx@entry=0x557abeeb8810, blocking=blocking@entry=true) at util/aio-posix.c:731
#11 0x0000557abd1c223d in blk_prw
    (blk=blk@entry=0x557abfc0daa0, offset=offset@entry=0, buf=buf@entry=0x7fff3ddcdde0 "", bytes=bytes@entry=512, co_entry=co_entry@entry=0x557abd1c3760 <blk_read_entry>, flags=flags@entry=0)
    at block/block-backend.c:1271
#12 0x0000557abd1c32ba in blk_pread (blk=blk@entry=0x557abfc0daa0, offset=offset@entry=0, buf=buf@entry=0x7fff3ddcdde0, count=count@entry=512) at block/block-backend.c:1430
#13 0x0000557abd06582f in guess_disk_lchs (blk=blk@entry=0x557abfc0daa0, pcylinders=pcylinders@entry=0x7fff3ddce024, pheads=pheads@entry=0x7fff3ddce028, psectors=psectors@entry=0x7fff3ddce02c)
    at hw/block/hd-geometry.c:66
#14 0x0000557abd06598f in hd_geometry_guess (blk=0x557abfc0daa0, pcyls=pcyls@entry=0x557abf3970bc, pheads=pheads@entry=0x557abf3970c0, psecs=psecs@entry=0x557abf3970c4, ptrans=ptrans@entry=0x0)
    at hw/block/hd-geometry.c:131
#15 0x0000557abd06559f in blkconf_geometry
    (conf=conf@entry=0x557abf3970a0, ptrans=ptrans@entry=0x0, cyls_max=cyls_max@entry=65535, heads_max=heads_max@entry=255, secs_max=secs_max@entry=255, errp=errp@entry=0x7fff3ddce110) at hw/block/block.c:145
#16 0x0000557abd0dcb7d in scsi_realize (dev=dev@entry=0x557abf397010, errp=errp@entry=0x7fff3ddce110) at hw/scsi/scsi-disk.c:2367
#17 0x0000557abd0dceb2 in scsi_hd_realize (dev=0x557abf397010, errp=0x7fff3ddce110) at hw/scsi/scsi-disk.c:2447
#18 0x0000557abd0e4757 in scsi_device_realize (errp=0x7fff3ddce110, s=0x557abf397010) at hw/scsi/scsi-bus.c:58
#19 0x0000557abd0e4757 in scsi_qdev_realize (qdev=<optimized out>, errp=0x7fff3ddce170) at hw/scsi/scsi-bus.c:216
#20 0x0000557abd06ed94 in device_set_realized (obj=<optimized out>, value=<optimized out>, errp=0x7fff3ddce298) at hw/core/qdev.c:876
#21 0x0000557abd18154b in property_set_bool (obj=0x557abf397010, v=<optimized out>, name=<optimized out>, opaque=0x557abfbe7d10, errp=0x7fff3ddce298) at qom/object.c:2078
#22 0x0000557abd185a13 in object_property_set_qobject (obj=0x557abf397010, value=<optimized out>, name=0x557abd33b3dd "realized", errp=0x7fff3ddce298) at qom/qom-qobject.c:26
#23 0x0000557abd183279 in object_property_set_bool (obj=0x557abf397010, value=<optimized out>, name=0x557abd33b3dd "realized", errp=0x7fff3ddce298) at qom/object.c:1336
#24 0x0000557abd0354f1 in qdev_device_add (opts=opts@entry=0x557abef77680, errp=errp@entry=0x7fff3ddce370) at qdev-monitor.c:673
#25 0x0000557abd03589b in qmp_device_add (qdict=<optimized out>, ret_data=<optimized out>, errp=0x7fff3ddce3b8) at qdev-monitor.c:798
#26 0x0000557abd20c74c in do_qmp_dispatch (errp=0x7fff3ddce3b0, allow_oob=<optimized out>, request=<optimized out>, cmds=0x557abdaf7c60 <qmp_commands>) at qapi/qmp-dispatch.c:132
#27 0x0000557abd20c74c in qmp_dispatch (cmds=0x557abdaf7c60 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>) at qapi/qmp-dispatch.c:175
#28 0x0000557abd12a8e1 in monitor_qmp_dispatch (mon=0x557abef08300, req=<optimized out>) at monitor/qmp.c:145
#29 0x0000557abd12af7a in monitor_qmp_bh_dispatcher (data=<optimized out>) at monitor/qmp.c:234
#30 0x0000557abd254146 in aio_bh_call (bh=0x557abee2ca20) at util/async.c:117
#31 0x0000557abd254146 in aio_bh_poll (ctx=ctx@entry=0x557abee2b5d0) at util/async.c:117
#32 0x0000557abd257534 in aio_dispatch (ctx=0x557abee2b5d0) at util/aio-posix.c:459
#33 0x0000557abd254022 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:260
#34 0x00007fafe631a67d in g_main_dispatch (context=0x557abeeb9c60) at gmain.c:3176
#35 0x00007fafe631a67d in g_main_context_dispatch (context=context@entry=0x557abeeb9c60) at gmain.c:3829
#36 0x0000557abd2565e8 in glib_pollfds_poll () at util/main-loop.c:219
#37 0x0000557abd2565e8 in os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:242
#38 0x0000557abd2565e8 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:518
#39 0x0000557abd038bb1 in main_loop () at vl.c:1828
#40 0x0000557abcee4d22 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4504


Actual results:
As subject

Expected results:
NO segment fault

Additional info:
See vm xml, disk xml, all threads backtrace in attachment.

QMP of attach-disk:
2020-03-11 08:53:07.308+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"blockdev-add","arguments":{"driver":"file","filename":"/tmp/disk-raw","aio":"native","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"},"id":"libvirt-385"}                                                                           
2020-03-11 08:53:07.311+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"blockdev-add","arguments":{"node-name":"libvirt-4-format","read-only":false,"discard":"ignore","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-4-storage"},"id":"libvirt-386"}                                                                                               
2020-03-11 08:53:07.313+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"device_add","arguments":{"driver":"scsi-hd","bus":"scsi0.0","channel":"0","scsi-id":"0","lun":"1","device_id":"drive-scsi0-0-0-1","drive":"libvirt-4-format","id":"scsi0-0-0-1","write-cache":"on"},"id":"libvirt-387"}                                                                              
2020-03-11 08:53:07.325+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-388"} 

QMP of detach-disk:
2020-03-11 08:53:42.693+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"device_del","arguments":{"id":"scsi0-0-0-1"},"id":"libvirt-389"}                  
2020-03-11 08:53:42.701+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"blockdev-del","arguments":{"node-name":"libvirt-4-format"},"id":"libvirt-390"}    
2020-03-11 08:53:42.705+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"blockdev-del","arguments":{"node-name":"libvirt-4-storage"},"id":"libvirt-391"}   
2020-03-11 08:53:42.715+0000: 185345: info : qemuMonitorIOWrite:453 : QEMU_MONITOR_IO_WRITE: mon=0x7f14f0363ad0 buf={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-392"}

Comment 7 qing.wang 2020-03-23 09:15:41 UTC
I tried change spice port to other port in xml, like as 5908. 
but it still using 5900 after virsh create pc.xml.

And i change the /tmp/disk-raw to other location like as /home/images/test.raw , it is not reproduced. 

what about your test result.

Comment 8 Han Han 2020-03-23 09:37:08 UTC
(In reply to qing.wang from comment #7)
> I tried change spice port to other port in xml, like as 5908. 
> but it still using 5900 after virsh create pc.xml.
For this bug, it doesn't matter whatever is spice port is.
If you want to set it manually, remember to set autoport='no' in graphic element, too.
> 
> And i change the /tmp/disk-raw to other location like as
> /home/images/test.raw , it is not reproduced. 
In libvirt, we usually use /var/lib/libvirt/images/ as image dir. Image in /home may trigger selinux permission deny issue.

I cannot reproduce it for image in /var/lib/libvirt/images/ at libvirt-6.0.0-14.module+el8.2.0+6069+78a1cb09.x86_64 qemu-kvm-4.2.0-15.module+el8.2.0+6029+618ef2ec.x86_64
> 
> what about your test result.

Comment 9 Maxim Levitsky 2020-03-31 14:43:53 UTC
I mostly got to the bottom of this, but I suspect that issue might be deeper so I will send a mail upstream to hear opinion of other developers.

What happens is this:

* On scsi_realize of scsi-hd we get call to blk_pread almost immediately :

  scsi_hd_realize -> scsi_realize -> blkconf_geometry -> guess_disk_lchs -> blk_pread


* blk_pread causes aio polling to kick in and call scsi_disk_emulate_inquiry because the virtio-scsi queues
  already contain an INQUIRY command (probably from the previos iteration of the hotplug loop
  (I didn't see anywhere any flushing of virtio-scsi queues when a scsi disk is unplugged, which can be considered to some extent a bug as well)

  blk_pread -> blk_prw -> aio_poll -> try_poll_mode -> run_poll_handlers_once -> 
  virtio_queue_host_notifier_aio_poll -> ... scsi_disk_emulate_command -> scsi_disk_emulate_inquiry

* scsi_disk_emulate_inquiry crashes since it tries to access the s->vendor which is not yet set by still running scsi_realize

This reproduces on qemu-kvm-4.2.0-16.module+el8.2.0+6092+4f2391c1 when running in a nested VM.
I also managed to reproduce this upstream when I slightly change the code that does polling (upstream code got smarter in regard
to when to poll so in this case the polling doens't happen anymore and thus the bug doesn't reproduce).

The change I did was:

diff --git a/util/aio-posix.c b/util/aio-posix.c
index cd6cf0a4a9..2fa11f4834 100644
--- a/util/aio-posix.c
+++ b/util/aio-posix.c
@@ -530,6 +530,8 @@ static bool try_poll_mode(AioContext *ctx, int64_t *timeout)
 {
     int64_t max_ns;
 
+    return run_poll_handlers_once(ctx, qemu_clock_get_ns(QEMU_CLOCK_REALTIME), timeout);
+
     if (QLIST_EMPTY_RCU(&ctx->poll_aio_handlers)) {
         return false;
     }

With that I can make the upstream (as of yestarday) qemu crash when it runs nested in RHEL8 VM. Native seems not
to reproduce but that is most likely difference in timing.


I can trivially fix this by delaying the call to blkconf_geometry to end of scsi_realize, however I think that the issue is deeper here.

I examined the code of the qdev_device_add (upstream) and I see the following pattern:


qdev_set_parent_bus(dev, bus);
...
object_property_set_bool(OBJECT(dev), true, "realized", &err);


The call to qdev_set_parent_bus, plugs the not yet realized device to the parent scsi bus.
and in that exact moment the device becames visible to the user since scsi_device_find which is
what scsi HBA drivers use to dispatch the requests to the scsi devices just scans the child devices
on the bus

So from this point and till the device is realized there is a window where non realized or partially
realized scsi device can be called to do IO.

I think that the right solution here is to make scsi bus implement the proper hotplug handlers instead
of scanning child devices like that.

Comment 10 John Ferlan 2020-06-30 16:00:47 UTC
NB: Bug 1844343 may also be related. It seems Sergio has found that the series posted upstream for this bug:

https://patchew.org/QEMU/20200511160951.8733-1-mlevitsk@redhat.com/

has resolved the issue from the bug.

Comment 11 Maxim Levitsky 2020-07-15 15:08:18 UTC
Also bug 1844343 is the pretty much sure has the same root cause.

I just posted an updated series upstream, and this time I reproduced a proper
race vs iothread while doing IO and unplugging and it crashes upstream pretty much right away,
and with my series there are no more crashes.

For the reference, this is what I used:

-blockdev node-name=test_disk,driver=file,filename=./test/image.raw
-object iothread,id=iothread1
-device virtio-scsi,id=scsi-test,iothread=iothread1
-device scsi-hd,drive=test_disk,bus=scsi-test.0,bootindex=-1,id=scsi_disk

Then I would plug/unplug them using

while true ; do
   vmadm hmp "device_del scsi_disk"
   vmadm hmp "device_add scsi-hd,drive=test_disk,bus=scsi-test.0,bootindex=-1,id=scsi_disk"
   sleep 2
done 

(vmadm is my small hmp wrapper script)

In the guest I was running this good old fio job from the days of my nvme work in a loop:

while true ; do
sudo fio --numjobs=32 --name=job --runtime=60 --time_based \
         --filename='/dev/disk/by-path/pci-0000\:00\:01.0-scsi-0\:0\:0\:0' \
         --ioengine=libaio --direct=1 --rw=randread --bs=4K --cpus_allowed_policy=split \
         --thread  --clocksource=cpu --group_reporting \
         --iodepth=128 \
         --iodepth_batch_submit=8 --iodepth_batch_complete_min=1 --iodepth_batch_complete_max=16

done

Comment 12 John Ferlan 2020-08-11 11:03:25 UTC
v3 was posted/reviewed: https://patchew.org/QEMU/20200715150159.95050-1-mlevitsk@redhat.com/ - waiting for 5.2 release to open up for pull/merge upstream.

Comment 16 John Ferlan 2020-09-30 17:51:25 UTC
FYI: Paolo rework: https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg09572.html

Comment 23 John Ferlan 2020-11-13 12:11:20 UTC
*** Bug 1844343 has been marked as a duplicate of this bug. ***

Comment 27 qing.wang 2020-11-20 06:10:32 UTC
Verified on Red Hat Enterprise Linux release 8.3 (Ootpa)
4.18.0-240.4.1.el8_3.x86_64
qemu-kvm-common-5.1.0-15.module+el8.3.1+8772+a3fdeccd.x86_64

Scenario 1: refer bug 1812399 comment 0
1.boot vm
virsh define pc.xml;virsh start pc

2.hotplug-unplug disk repeatly
while true;do virsh attach-device pc disk.xml; virsh detach-device pc disk.xml;done

Running over 1 hour , no crash issue found.

Scenario 2: 

1. create 40 image files 
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg0.qcow2 1G
...
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg40.qcow2 1G

2.boot vm
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine pc \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pci.0,addr=0x2,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x3 \
    -m 2048  \
    -smp 12,maxcpus=12,cores=6,threads=1,sockets=2  \
    -device pcie-root-port,id=pcie-root-port-1,bus=pci.0,chassis=2 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,bus=pci.0,chassis=3 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread1 \
    -device virtio-scsi,id=scsi0 \
    -device virtio-scsi,id=scsi1,iothread=iothread1 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel831-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,bus=scsi0.0 \
    \
    -blockdev node-name=test_disk0,driver=file,filename=/home/kvm_autotest_root/images/stg0.qcow2 \
    -device scsi-hd,drive=test_disk0,bus=scsi1.0,bootindex=-1,id=scsi_disk0,channel=0,scsi-id=0,channel=0,scsi-id=0,lun=0,share-rw \
    -blockdev node-name=test_disk1,driver=file,filename=/home/kvm_autotest_root/images/stg1.qcow2 \
    -blockdev node-name=test_disk2,driver=file,filename=/home/kvm_autotest_root/images/stg2.qcow2 \
    -blockdev node-name=test_disk3,driver=file,filename=/home/kvm_autotest_root/images/stg3.qcow2 \
    -blockdev node-name=test_disk4,driver=file,filename=/home/kvm_autotest_root/images/stg4.qcow2 \
    -blockdev node-name=test_disk5,driver=file,filename=/home/kvm_autotest_root/images/stg5.qcow2 \
    -blockdev node-name=test_disk6,driver=file,filename=/home/kvm_autotest_root/images/stg6.qcow2 \
    -blockdev node-name=test_disk7,driver=file,filename=/home/kvm_autotest_root/images/stg7.qcow2 \
    -blockdev node-name=test_disk8,driver=file,filename=/home/kvm_autotest_root/images/stg8.qcow2 \
    -blockdev node-name=test_disk9,driver=file,filename=/home/kvm_autotest_root/images/stg9.qcow2 \
    -blockdev node-name=test_disk10,driver=file,filename=/home/kvm_autotest_root/images/stg10.qcow2 \
    -blockdev node-name=test_disk11,driver=file,filename=/home/kvm_autotest_root/images/stg11.qcow2 \
    -blockdev node-name=test_disk12,driver=file,filename=/home/kvm_autotest_root/images/stg12.qcow2 \
    -blockdev node-name=test_disk13,driver=file,filename=/home/kvm_autotest_root/images/stg13.qcow2 \
    -blockdev node-name=test_disk14,driver=file,filename=/home/kvm_autotest_root/images/stg14.qcow2 \
    -blockdev node-name=test_disk15,driver=file,filename=/home/kvm_autotest_root/images/stg15.qcow2 \
    -blockdev node-name=test_disk16,driver=file,filename=/home/kvm_autotest_root/images/stg16.qcow2 \
    -blockdev node-name=test_disk17,driver=file,filename=/home/kvm_autotest_root/images/stg17.qcow2 \
    -blockdev node-name=test_disk18,driver=file,filename=/home/kvm_autotest_root/images/stg18.qcow2 \
    -blockdev node-name=test_disk19,driver=file,filename=/home/kvm_autotest_root/images/stg19.qcow2 \
    -blockdev node-name=test_disk20,driver=file,filename=/home/kvm_autotest_root/images/stg20.qcow2 \
    -blockdev node-name=test_disk21,driver=file,filename=/home/kvm_autotest_root/images/stg21.qcow2 \
    -blockdev node-name=test_disk22,driver=file,filename=/home/kvm_autotest_root/images/stg22.qcow2 \
    -blockdev node-name=test_disk23,driver=file,filename=/home/kvm_autotest_root/images/stg23.qcow2 \
    -blockdev node-name=test_disk24,driver=file,filename=/home/kvm_autotest_root/images/stg24.qcow2 \
    -blockdev node-name=test_disk25,driver=file,filename=/home/kvm_autotest_root/images/stg25.qcow2 \
    -blockdev node-name=test_disk26,driver=file,filename=/home/kvm_autotest_root/images/stg26.qcow2 \
    -blockdev node-name=test_disk27,driver=file,filename=/home/kvm_autotest_root/images/stg27.qcow2 \
    -blockdev node-name=test_disk28,driver=file,filename=/home/kvm_autotest_root/images/stg28.qcow2 \
    -blockdev node-name=test_disk29,driver=file,filename=/home/kvm_autotest_root/images/stg29.qcow2 \
    -blockdev node-name=test_disk30,driver=file,filename=/home/kvm_autotest_root/images/stg30.qcow2 \
    -blockdev node-name=test_disk31,driver=file,filename=/home/kvm_autotest_root/images/stg31.qcow2 \
    -blockdev node-name=test_disk32,driver=file,filename=/home/kvm_autotest_root/images/stg32.qcow2 \
    -blockdev node-name=test_disk33,driver=file,filename=/home/kvm_autotest_root/images/stg33.qcow2 \
    -blockdev node-name=test_disk34,driver=file,filename=/home/kvm_autotest_root/images/stg34.qcow2 \
    -blockdev node-name=test_disk35,driver=file,filename=/home/kvm_autotest_root/images/stg35.qcow2 \
    -blockdev node-name=test_disk36,driver=file,filename=/home/kvm_autotest_root/images/stg36.qcow2 \
    -blockdev node-name=test_disk37,driver=file,filename=/home/kvm_autotest_root/images/stg37.qcow2 \
    -blockdev node-name=test_disk38,driver=file,filename=/home/kvm_autotest_root/images/stg38.qcow2 \
    -blockdev node-name=test_disk39,driver=file,filename=/home/kvm_autotest_root/images/stg39.qcow2 \
    -blockdev node-name=test_disk40,driver=file,filename=/home/kvm_autotest_root/images/stg40.qcow2 \
    \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,bus=pci.0,chassis=4 \
    -device virtio-net-pci,mac=9a:21:f7:4a:1e:bd,id=idRuZxfv,netdev=idOpPVAe,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idOpPVAe,vhost=on  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -vnc :5  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,bus=pci.0 \
    -monitor stdio \
    -chardev file,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpdbg.log,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -qmp tcp:0:5955,server,nowait  \
    -chardev file,path=/var/tmp/monitor-serialdbg.log,id=serial_id_serial0 \
    -device isa-serial,chardev=serial_id_serial0  \

3.login guest and execute sg_luns with multi instances
trap 'kill $(jobs -p)' EXIT SIGINT

for i in `seq 0 32` ; do
	while true ; do
#		sg_luns /dev/sdb > /dev/null 2>&1
    sg_luns /dev/sdb
	done &
done
echo "wait"
wait

4.hotplug-unlug multi disks repeatly on each 3 seconds
NUM_LUNS=40
add_devices() {
  exec 3<>/dev/tcp/localhost/5955
  echo "$@"
  echo -e "{'execute':'qmp_capabilities'}" >&3
  read response <&3
  echo $response
  for i in $(seq 1 $NUM_LUNS) ; do
  cmd="{'execute':'device_add', 'arguments': {'driver':'scsi-hd','drive':'test_disk$i','id':'scsi_disk$i','bus':'scsi1.0','lun':$i}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  done
}

remove_devices() {
  exec 3<>/dev/tcp/localhost/5955
  echo "$@"
  echo -e "{'execute':'qmp_capabilities'}" >&3
  read response <&3
  echo $response
  for i in $(seq 1 $NUM_LUNS) ; do
  cmd="{'execute':'device_del', 'arguments': {'id':'scsi_disk$i'}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  done
}


while true ; do
    echo "adding devices"
    add_devices
    sleep 3
    echo "removing devices"
    remove_devices
    sleep 3
done

running over 1 hour, no crash issue found.

Comment 29 errata-xmlrpc 2021-02-22 15:39:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0639


Note You need to log in before you can comment on or make changes to this bug.