Bug 1812399

I tried change spice port to other port in xml, like as 5908. 
but it still using 5900 after virsh create pc.xml.

And i change the /tmp/disk-raw to other location like as /home/images/test.raw , it is not reproduced. 

what about your test result.

(In reply to qing.wang from comment #7)
> I tried change spice port to other port in xml, like as 5908. 
> but it still using 5900 after virsh create pc.xml.
For this bug, it doesn't matter whatever is spice port is.
If you want to set it manually, remember to set autoport='no' in graphic element, too.
> 
> And i change the /tmp/disk-raw to other location like as
> /home/images/test.raw , it is not reproduced. 
In libvirt, we usually use /var/lib/libvirt/images/ as image dir. Image in /home may trigger selinux permission deny issue.

I cannot reproduce it for image in /var/lib/libvirt/images/ at libvirt-6.0.0-14.module+el8.2.0+6069+78a1cb09.x86_64 qemu-kvm-4.2.0-15.module+el8.2.0+6029+618ef2ec.x86_64
> 
> what about your test result.

I mostly got to the bottom of this, but I suspect that issue might be deeper so I will send a mail upstream to hear opinion of other developers.

What happens is this:

* On scsi_realize of scsi-hd we get call to blk_pread almost immediately :

  scsi_hd_realize -> scsi_realize -> blkconf_geometry -> guess_disk_lchs -> blk_pread


* blk_pread causes aio polling to kick in and call scsi_disk_emulate_inquiry because the virtio-scsi queues
  already contain an INQUIRY command (probably from the previos iteration of the hotplug loop
  (I didn't see anywhere any flushing of virtio-scsi queues when a scsi disk is unplugged, which can be considered to some extent a bug as well)

  blk_pread -> blk_prw -> aio_poll -> try_poll_mode -> run_poll_handlers_once -> 
  virtio_queue_host_notifier_aio_poll -> ... scsi_disk_emulate_command -> scsi_disk_emulate_inquiry

* scsi_disk_emulate_inquiry crashes since it tries to access the s->vendor which is not yet set by still running scsi_realize

This reproduces on qemu-kvm-4.2.0-16.module+el8.2.0+6092+4f2391c1 when running in a nested VM.
I also managed to reproduce this upstream when I slightly change the code that does polling (upstream code got smarter in regard
to when to poll so in this case the polling doens't happen anymore and thus the bug doesn't reproduce).

The change I did was:

diff --git a/util/aio-posix.c b/util/aio-posix.c
index cd6cf0a4a9..2fa11f4834 100644
--- a/util/aio-posix.c
+++ b/util/aio-posix.c
@@ -530,6 +530,8 @@ static bool try_poll_mode(AioContext *ctx, int64_t *timeout)
 {
     int64_t max_ns;
 
+    return run_poll_handlers_once(ctx, qemu_clock_get_ns(QEMU_CLOCK_REALTIME), timeout);
+
     if (QLIST_EMPTY_RCU(&ctx->poll_aio_handlers)) {
         return false;
     }

With that I can make the upstream (as of yestarday) qemu crash when it runs nested in RHEL8 VM. Native seems not
to reproduce but that is most likely difference in timing.


I can trivially fix this by delaying the call to blkconf_geometry to end of scsi_realize, however I think that the issue is deeper here.

I examined the code of the qdev_device_add (upstream) and I see the following pattern:


qdev_set_parent_bus(dev, bus);
...
object_property_set_bool(OBJECT(dev), true, "realized", &err);


The call to qdev_set_parent_bus, plugs the not yet realized device to the parent scsi bus.
and in that exact moment the device becames visible to the user since scsi_device_find which is
what scsi HBA drivers use to dispatch the requests to the scsi devices just scans the child devices
on the bus

So from this point and till the device is realized there is a window where non realized or partially
realized scsi device can be called to do IO.

I think that the right solution here is to make scsi bus implement the proper hotplug handlers instead
of scanning child devices like that.

NB: Bug 1844343 may also be related. It seems Sergio has found that the series posted upstream for this bug:

https://patchew.org/QEMU/20200511160951.8733-1-mlevitsk@redhat.com/

has resolved the issue from the bug.

Also bug 1844343 is the pretty much sure has the same root cause.

I just posted an updated series upstream, and this time I reproduced a proper
race vs iothread while doing IO and unplugging and it crashes upstream pretty much right away,
and with my series there are no more crashes.

For the reference, this is what I used:

-blockdev node-name=test_disk,driver=file,filename=./test/image.raw
-object iothread,id=iothread1
-device virtio-scsi,id=scsi-test,iothread=iothread1
-device scsi-hd,drive=test_disk,bus=scsi-test.0,bootindex=-1,id=scsi_disk

Then I would plug/unplug them using

while true ; do
   vmadm hmp "device_del scsi_disk"
   vmadm hmp "device_add scsi-hd,drive=test_disk,bus=scsi-test.0,bootindex=-1,id=scsi_disk"
   sleep 2
done 

(vmadm is my small hmp wrapper script)

In the guest I was running this good old fio job from the days of my nvme work in a loop:

while true ; do
sudo fio --numjobs=32 --name=job --runtime=60 --time_based \
         --filename='/dev/disk/by-path/pci-0000\:00\:01.0-scsi-0\:0\:0\:0' \
         --ioengine=libaio --direct=1 --rw=randread --bs=4K --cpus_allowed_policy=split \
         --thread  --clocksource=cpu --group_reporting \
         --iodepth=128 \
         --iodepth_batch_submit=8 --iodepth_batch_complete_min=1 --iodepth_batch_complete_max=16

done

v3 was posted/reviewed: https://patchew.org/QEMU/20200715150159.95050-1-mlevitsk@redhat.com/ - waiting for 5.2 release to open up for pull/merge upstream.

FYI, v4: https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg08143.html

FYI, v5: https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg04586.html

FYI: Paolo rework: https://lists.nongnu.org/archive/html/qemu-devel/2020-09/msg09572.html

*** Bug 1844343 has been marked as a duplicate of this bug. ***

Verified on Red Hat Enterprise Linux release 8.3 (Ootpa)
4.18.0-240.4.1.el8_3.x86_64
qemu-kvm-common-5.1.0-15.module+el8.3.1+8772+a3fdeccd.x86_64

Scenario 1: refer bug 1812399 comment 0
1.boot vm
virsh define pc.xml;virsh start pc

2.hotplug-unplug disk repeatly
while true;do virsh attach-device pc disk.xml; virsh detach-device pc disk.xml;done

Running over 1 hour , no crash issue found.

Scenario 2: 

1. create 40 image files 
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg0.qcow2 1G
...
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg40.qcow2 1G

2.boot vm
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine pc \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pci.0,addr=0x2,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x3 \
    -m 2048  \
    -smp 12,maxcpus=12,cores=6,threads=1,sockets=2  \
    -device pcie-root-port,id=pcie-root-port-1,bus=pci.0,chassis=2 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,bus=pci.0,chassis=3 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread1 \
    -device virtio-scsi,id=scsi0 \
    -device virtio-scsi,id=scsi1,iothread=iothread1 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel831-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,bus=scsi0.0 \
    \
    -blockdev node-name=test_disk0,driver=file,filename=/home/kvm_autotest_root/images/stg0.qcow2 \
    -device scsi-hd,drive=test_disk0,bus=scsi1.0,bootindex=-1,id=scsi_disk0,channel=0,scsi-id=0,channel=0,scsi-id=0,lun=0,share-rw \
    -blockdev node-name=test_disk1,driver=file,filename=/home/kvm_autotest_root/images/stg1.qcow2 \
    -blockdev node-name=test_disk2,driver=file,filename=/home/kvm_autotest_root/images/stg2.qcow2 \
    -blockdev node-name=test_disk3,driver=file,filename=/home/kvm_autotest_root/images/stg3.qcow2 \
    -blockdev node-name=test_disk4,driver=file,filename=/home/kvm_autotest_root/images/stg4.qcow2 \
    -blockdev node-name=test_disk5,driver=file,filename=/home/kvm_autotest_root/images/stg5.qcow2 \
    -blockdev node-name=test_disk6,driver=file,filename=/home/kvm_autotest_root/images/stg6.qcow2 \
    -blockdev node-name=test_disk7,driver=file,filename=/home/kvm_autotest_root/images/stg7.qcow2 \
    -blockdev node-name=test_disk8,driver=file,filename=/home/kvm_autotest_root/images/stg8.qcow2 \
    -blockdev node-name=test_disk9,driver=file,filename=/home/kvm_autotest_root/images/stg9.qcow2 \
    -blockdev node-name=test_disk10,driver=file,filename=/home/kvm_autotest_root/images/stg10.qcow2 \
    -blockdev node-name=test_disk11,driver=file,filename=/home/kvm_autotest_root/images/stg11.qcow2 \
    -blockdev node-name=test_disk12,driver=file,filename=/home/kvm_autotest_root/images/stg12.qcow2 \
    -blockdev node-name=test_disk13,driver=file,filename=/home/kvm_autotest_root/images/stg13.qcow2 \
    -blockdev node-name=test_disk14,driver=file,filename=/home/kvm_autotest_root/images/stg14.qcow2 \
    -blockdev node-name=test_disk15,driver=file,filename=/home/kvm_autotest_root/images/stg15.qcow2 \
    -blockdev node-name=test_disk16,driver=file,filename=/home/kvm_autotest_root/images/stg16.qcow2 \
    -blockdev node-name=test_disk17,driver=file,filename=/home/kvm_autotest_root/images/stg17.qcow2 \
    -blockdev node-name=test_disk18,driver=file,filename=/home/kvm_autotest_root/images/stg18.qcow2 \
    -blockdev node-name=test_disk19,driver=file,filename=/home/kvm_autotest_root/images/stg19.qcow2 \
    -blockdev node-name=test_disk20,driver=file,filename=/home/kvm_autotest_root/images/stg20.qcow2 \
    -blockdev node-name=test_disk21,driver=file,filename=/home/kvm_autotest_root/images/stg21.qcow2 \
    -blockdev node-name=test_disk22,driver=file,filename=/home/kvm_autotest_root/images/stg22.qcow2 \
    -blockdev node-name=test_disk23,driver=file,filename=/home/kvm_autotest_root/images/stg23.qcow2 \
    -blockdev node-name=test_disk24,driver=file,filename=/home/kvm_autotest_root/images/stg24.qcow2 \
    -blockdev node-name=test_disk25,driver=file,filename=/home/kvm_autotest_root/images/stg25.qcow2 \
    -blockdev node-name=test_disk26,driver=file,filename=/home/kvm_autotest_root/images/stg26.qcow2 \
    -blockdev node-name=test_disk27,driver=file,filename=/home/kvm_autotest_root/images/stg27.qcow2 \
    -blockdev node-name=test_disk28,driver=file,filename=/home/kvm_autotest_root/images/stg28.qcow2 \
    -blockdev node-name=test_disk29,driver=file,filename=/home/kvm_autotest_root/images/stg29.qcow2 \
    -blockdev node-name=test_disk30,driver=file,filename=/home/kvm_autotest_root/images/stg30.qcow2 \
    -blockdev node-name=test_disk31,driver=file,filename=/home/kvm_autotest_root/images/stg31.qcow2 \
    -blockdev node-name=test_disk32,driver=file,filename=/home/kvm_autotest_root/images/stg32.qcow2 \
    -blockdev node-name=test_disk33,driver=file,filename=/home/kvm_autotest_root/images/stg33.qcow2 \
    -blockdev node-name=test_disk34,driver=file,filename=/home/kvm_autotest_root/images/stg34.qcow2 \
    -blockdev node-name=test_disk35,driver=file,filename=/home/kvm_autotest_root/images/stg35.qcow2 \
    -blockdev node-name=test_disk36,driver=file,filename=/home/kvm_autotest_root/images/stg36.qcow2 \
    -blockdev node-name=test_disk37,driver=file,filename=/home/kvm_autotest_root/images/stg37.qcow2 \
    -blockdev node-name=test_disk38,driver=file,filename=/home/kvm_autotest_root/images/stg38.qcow2 \
    -blockdev node-name=test_disk39,driver=file,filename=/home/kvm_autotest_root/images/stg39.qcow2 \
    -blockdev node-name=test_disk40,driver=file,filename=/home/kvm_autotest_root/images/stg40.qcow2 \
    \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,bus=pci.0,chassis=4 \
    -device virtio-net-pci,mac=9a:21:f7:4a:1e:bd,id=idRuZxfv,netdev=idOpPVAe,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idOpPVAe,vhost=on  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -vnc :5  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,bus=pci.0 \
    -monitor stdio \
    -chardev file,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpdbg.log,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -qmp tcp:0:5955,server,nowait  \
    -chardev file,path=/var/tmp/monitor-serialdbg.log,id=serial_id_serial0 \
    -device isa-serial,chardev=serial_id_serial0  \

3.login guest and execute sg_luns with multi instances
trap 'kill $(jobs -p)' EXIT SIGINT

for i in `seq 0 32` ; do
	while true ; do
#		sg_luns /dev/sdb > /dev/null 2>&1
    sg_luns /dev/sdb
	done &
done
echo "wait"
wait

4.hotplug-unlug multi disks repeatly on each 3 seconds
NUM_LUNS=40
add_devices() {
  exec 3<>/dev/tcp/localhost/5955
  echo "$@"
  echo -e "{'execute':'qmp_capabilities'}" >&3
  read response <&3
  echo $response
  for i in $(seq 1 $NUM_LUNS) ; do
  cmd="{'execute':'device_add', 'arguments': {'driver':'scsi-hd','drive':'test_disk$i','id':'scsi_disk$i','bus':'scsi1.0','lun':$i}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  done
}

remove_devices() {
  exec 3<>/dev/tcp/localhost/5955
  echo "$@"
  echo -e "{'execute':'qmp_capabilities'}" >&3
  read response <&3
  echo $response
  for i in $(seq 1 $NUM_LUNS) ; do
  cmd="{'execute':'device_del', 'arguments': {'id':'scsi_disk$i'}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  done
}


while true ; do
    echo "adding devices"
    add_devices
    sleep 3
    echo "removing devices"
    remove_devices
    sleep 3
done

running over 1 hour, no crash issue found.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0639