Bug 1528260

Created attachment 1370832 [details]
back trace01

Created attachment 1370833 [details]
back trace02

This is a problem virtio-scsi and I/O threads.

We have two threads at play here: The main loop thread is trying to shut down all I/O threads in iothread_stop_all(). Apparently this happens while the I/O thread is still active and processing events.

Now iothread_stop_all() forces the associated BlockDriverState into the main loop AioContext without asking the virtio-scsi device model:

        aio_context_acquire(ctx);
        bdrv_set_aio_context(bs, qemu_get_aio_context());
        aio_context_release(ctx);

And when virtio-scsi asserts in the I/O thread that the BDS is in the expected AioContext while processing requests from the queue, it obviously fails.

We probably need to shut down dataplane first so that the I/O thread is unused when it gets destroyed. Just forcing BDSes into a different AioContext without even telling the device about it is just asking for trouble.

Could reproduce the issue with qemu-kvm-rhev-2.9.0-14.el7 after tried the steps several times .

Additional info:
(qemu) qemu-kvm: terminating on signal 2
qemu-kvm: /builddir/build/BUILD/qemu-2.9.0/hw/scsi/virtio-scsi.c:245: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
install.sh: line 41: 22646 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -vga std -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/pingl/monitor-qmpmonitor1-20171221-212809-KpKW84ek,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/pingl/monitor-catch_monitor-20171221-212809-KpKW84ek,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idUuj9Ci -chardev socket,id=serial_id_serial0,path=/var/tmp/pingl/serial-serial0-20171221-212809-KpKW84ek,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20171221-212809-KpKW84ek,path=/var/tmp/pingl/seabios-20171221-212809-KpKW84ek,server,nowait -device isa-debugcon,chardev=seabioslog_id_20171221-212809-KpKW84ek,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -object iothread,id=iothread0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/tests/diskfile/win2016.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:d7:d8:d9:da:db,id=idNNd5BR,vectors=4,netdev=idoSJmIM,bus=pci.0,addr=0x4 -netdev tap,id=idoSJmIM -m 4096 -smp 4,cores=2,threads=1,sockets=2 -cpu 'SandyBridge',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn,once=c -enable-kvm -monitor stdio -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Win2016/en_windows_server_2016_x64_dvd_9718492.iso -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_winutils,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso -device ide-cd,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 -drive id=drive_virtio,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-145.iso -device ide-cd,id=virtio,drive=drive_virtio,bus=ide.1,unit=0

Created attachment 1371156 [details]
backtrace with qemu-kvm-rhev-2.10.0-1.el7

Created attachment 1371157 [details]
backtrace with qemu-kvm-rhev-2.9.0-14.el7

Error log with qemu-kvm-rhev-2.10.0-1.el7:

(qemu) qemu-kvm: terminating on signal 2
qemu-kvm: /builddir/build/BUILD/qemu-2.10.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
install.sh: line 41:  6738 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -vga std -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/pingl/monitor-qmpmonitor1-20171221-212851-hWW4ALr6,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/pingl/monitor-catch_monitor-20171221-212851-hWW4ALr6,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=id3388fN -chardev socket,id=serial_id_serial0,path=/var/tmp/pingl/serial-serial0-20171221-212851-hWW4ALr6,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20171221-212851-hWW4ALr6,path=/var/tmp/pingl/seabios-20171221-212851-hWW4ALr6,server,nowait -device isa-debugcon,chardev=seabioslog_id_20171221-212851-hWW4ALr6,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -object iothread,id=iothread0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/tests/diskfile/win2016.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:9b:9c:9d:9e:9f,id=idGlLJEQ,vectors=4,netdev=id4rfs5a,bus=pci.0,addr=0x4 -netdev tap,id=id4rfs5a -m 4096 -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 -cpu 'SandyBridge',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn,once=c -enable-kvm -monitor stdio -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Win2016/en_windows_server_2016_x64_dvd_9718492.iso -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_winutils,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso -device ide-cd,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 -drive id=drive_virtio,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-145.iso -device ide-cd,id=virtio,drive=drive_virtio,bus=ide.1,unit=0

Patch posted upstream:
https://patchwork.ozlabs.org/patch/867549/

Don't hit this issue in latest qemu version, set bug's status to "Verified".
 kernel version:3.10.0-931.el7.x86_64
 qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-12.el7.x86_64


Test steps:
 1.Start guest with virtio_scsi+data_plane enable:
     /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20180822-083928-y8PRqUqd,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20180822-083928-y8PRqUqd,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idVmd181  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20180822-083928-y8PRqUqd,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20180822-083928-y8PRqUqd,path=/var/tmp/seabios-20180822-083928-y8PRqUqd,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20180822-083928-y8PRqUqd,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,iothread=iothread0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=directsync,format=qcow2,file=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2 \
    -device scsi-hd,drive=drive_image1,bus=scsi0.0,id=image1 \
    -device virtio-net-pci,mac=9a:1e:1f:20:21:22,id=idVobBWE,vectors=4,netdev=idH1zOQ2,bus=pci.0,addr=0x6  \
    -netdev tap,id=idH1zOQ2,vhost=on \
    -m 4096  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=d \
    -enable-kvm \
    -monitor stdio \
    -spice disable-ticketing,port=5000 \

2. During its boot stage, input "Ctrl+C"

3. Repeat step 1~2 for times.


Test result:
  qemu can quit with correct info "(qemu) qemu-kvm: terminating on signal 2".

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443