Bug 1528260 - [data-plane] qemu core dumps after "Ctrl+C"
Summary: [data-plane] qemu core dumps after "Ctrl+C"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: ---
Assignee: Stefan Hajnoczi
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-21 11:15 UTC by Ping Li
Modified: 2018-11-01 11:05 UTC (History)
15 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-01 11:04:00 UTC
Target Upstream Version:


Attachments (Terms of Use)
back trace01 (97.73 KB, text/plain)
2017-12-21 11:16 UTC, Ping Li
no flags Details
back trace02 (99.35 KB, text/plain)
2017-12-21 11:16 UTC, Ping Li
no flags Details
backtrace with qemu-kvm-rhev-2.10.0-1.el7 (109.27 KB, text/plain)
2017-12-22 03:35 UTC, Ping Li
no flags Details
backtrace with qemu-kvm-rhev-2.9.0-14.el7 (98.70 KB, text/plain)
2017-12-22 03:37 UTC, Ping Li
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3443 None None None 2018-11-01 11:05:51 UTC

Description Ping Li 2017-12-21 11:15:02 UTC
Description of problem:
After boot a vm with data plane, kill the command line by "Ctrl+C". Then qemu core dump occurs. If remove data plane from the qemu cml, cannot reproduce the issue with 15 times trial.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.10.0-13.el7
kernel-3.10.0-823.el7.x86_64

How reproducible:
5/10

Steps to Reproduce:
1. Install win2016 on the qcow2 image
2. Boot the vm with data plane
    -object iothread,id=iothread0 
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/tests/diskfile/win2016.qcow2,rerror=stop,werror=stop 
    -device scsi-hd,id=image1,drive=drive_image1
3. Kill the guest by "Ctrl+C"

Actual results:
qemu core dump

Expected results:
qemu was killed without error

Additional info:

Comment 2 Ping Li 2017-12-21 11:16:03 UTC
Created attachment 1370832 [details]
back trace01

Comment 3 Ping Li 2017-12-21 11:16:38 UTC
Created attachment 1370833 [details]
back trace02

Comment 4 Kevin Wolf 2017-12-21 11:37:40 UTC
This is a problem virtio-scsi and I/O threads.

We have two threads at play here: The main loop thread is trying to shut down all I/O threads in iothread_stop_all(). Apparently this happens while the I/O thread is still active and processing events.

Now iothread_stop_all() forces the associated BlockDriverState into the main loop AioContext without asking the virtio-scsi device model:

        aio_context_acquire(ctx);
        bdrv_set_aio_context(bs, qemu_get_aio_context());
        aio_context_release(ctx);

And when virtio-scsi asserts in the I/O thread that the BDS is in the expected AioContext while processing requests from the queue, it obviously fails.

We probably need to shut down dataplane first so that the I/O thread is unused when it gets destroyed. Just forcing BDSes into a different AioContext without even telling the device about it is just asking for trouble.

Comment 5 Ping Li 2017-12-22 03:34:27 UTC
Could reproduce the issue with qemu-kvm-rhev-2.9.0-14.el7 after tried the steps several times .

Additional info:
(qemu) qemu-kvm: terminating on signal 2
qemu-kvm: /builddir/build/BUILD/qemu-2.9.0/hw/scsi/virtio-scsi.c:245: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
install.sh: line 41: 22646 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -vga std -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/pingl/monitor-qmpmonitor1-20171221-212809-KpKW84ek,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/pingl/monitor-catch_monitor-20171221-212809-KpKW84ek,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idUuj9Ci -chardev socket,id=serial_id_serial0,path=/var/tmp/pingl/serial-serial0-20171221-212809-KpKW84ek,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20171221-212809-KpKW84ek,path=/var/tmp/pingl/seabios-20171221-212809-KpKW84ek,server,nowait -device isa-debugcon,chardev=seabioslog_id_20171221-212809-KpKW84ek,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -object iothread,id=iothread0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/tests/diskfile/win2016.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:d7:d8:d9:da:db,id=idNNd5BR,vectors=4,netdev=idoSJmIM,bus=pci.0,addr=0x4 -netdev tap,id=idoSJmIM -m 4096 -smp 4,cores=2,threads=1,sockets=2 -cpu 'SandyBridge',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn,once=c -enable-kvm -monitor stdio -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Win2016/en_windows_server_2016_x64_dvd_9718492.iso -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_winutils,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso -device ide-cd,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 -drive id=drive_virtio,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-145.iso -device ide-cd,id=virtio,drive=drive_virtio,bus=ide.1,unit=0

Comment 6 Ping Li 2017-12-22 03:35:55 UTC
Created attachment 1371156 [details]
backtrace with qemu-kvm-rhev-2.10.0-1.el7

Comment 7 Ping Li 2017-12-22 03:37:06 UTC
Created attachment 1371157 [details]
backtrace with qemu-kvm-rhev-2.9.0-14.el7

Comment 8 Ping Li 2017-12-22 03:41:01 UTC
Error log with qemu-kvm-rhev-2.10.0-1.el7:

(qemu) qemu-kvm: terminating on signal 2
qemu-kvm: /builddir/build/BUILD/qemu-2.10.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
install.sh: line 41:  6738 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -vga std -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/pingl/monitor-qmpmonitor1-20171221-212851-hWW4ALr6,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/pingl/monitor-catch_monitor-20171221-212851-hWW4ALr6,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=id3388fN -chardev socket,id=serial_id_serial0,path=/var/tmp/pingl/serial-serial0-20171221-212851-hWW4ALr6,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20171221-212851-hWW4ALr6,path=/var/tmp/pingl/seabios-20171221-212851-hWW4ALr6,server,nowait -device isa-debugcon,chardev=seabioslog_id_20171221-212851-hWW4ALr6,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -object iothread,id=iothread0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/tests/diskfile/win2016.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:9b:9c:9d:9e:9f,id=idGlLJEQ,vectors=4,netdev=id4rfs5a,bus=pci.0,addr=0x4 -netdev tap,id=id4rfs5a -m 4096 -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 -cpu 'SandyBridge',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn,once=c -enable-kvm -monitor stdio -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Win2016/en_windows_server_2016_x64_dvd_9718492.iso -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_winutils,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso -device ide-cd,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 -drive id=drive_virtio,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-145.iso -device ide-cd,id=virtio,drive=drive_virtio,bus=ide.1,unit=0

Comment 10 Stefan Hajnoczi 2018-01-30 15:50:38 UTC
Patch posted upstream:
https://patchwork.ozlabs.org/patch/867549/

Comment 16 aihua liang 2018-08-30 09:33:26 UTC
Don't hit this issue in latest qemu version, set bug's status to "Verified".
 kernel version:3.10.0-931.el7.x86_64
 qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-12.el7.x86_64


Test steps:
 1.Start guest with virtio_scsi+data_plane enable:
     /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20180822-083928-y8PRqUqd,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20180822-083928-y8PRqUqd,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idVmd181  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20180822-083928-y8PRqUqd,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20180822-083928-y8PRqUqd,path=/var/tmp/seabios-20180822-083928-y8PRqUqd,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20180822-083928-y8PRqUqd,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,iothread=iothread0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=directsync,format=qcow2,file=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2 \
    -device scsi-hd,drive=drive_image1,bus=scsi0.0,id=image1 \
    -device virtio-net-pci,mac=9a:1e:1f:20:21:22,id=idVobBWE,vectors=4,netdev=idH1zOQ2,bus=pci.0,addr=0x6  \
    -netdev tap,id=idH1zOQ2,vhost=on \
    -m 4096  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=d \
    -enable-kvm \
    -monitor stdio \
    -spice disable-ticketing,port=5000 \

2. During its boot stage, input "Ctrl+C"

3. Repeat step 1~2 for times.


Test result:
  qemu can quit with correct info "(qemu) qemu-kvm: terminating on signal 2".

Comment 18 errata-xmlrpc 2018-11-01 11:04:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443


Note You need to log in before you can comment on or make changes to this bug.