Bug 1480202
| Summary: | Occurred core dump with multi-object when quitted qemu during doing IO | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yongxue Hong <yhong> | |
| Component: | qemu-kvm-rhev | Assignee: | Stefan Hajnoczi <stefanha> | |
| Status: | CLOSED ERRATA | QA Contact: | aihua liang <aliang> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 7.4-Alt | CC: | aliang, chayang, coli, famz, jen, juzhang, knoel, michen, mrezanin, mtessun, ngu, qzhang, virt-maint, xianwang | |
| Target Milestone: | rc | Keywords: | ZStream | |
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | qemu-kvm-rhev-2.12.0-1.el7 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1566586 (view as bug list) | Environment: | ||
| Last Closed: | 2018-11-01 11:01:10 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1558351, 1566586 | |||
|
Description
Yongxue Hong
2017-08-10 11:57:45 UTC
It also could be reproduced on x86 and P8.
Version of x86:
Host : 4.11.0-22.el7a.x86_64
Guest : 4.11.0-22.el7a.x86_64
Qemu-kvm : qemu-kvm-2.9.0-19.el7a
Version of P8:
Host : 3.10.0-693.el7.ppc64le
Guest : 3.10.0-693.el7.ppc64le
Qemu-kvm-rhev : qemu-kvm-rhev-2.9.0-14.el7
Reporduce it on x86, the same problem exist.
Test version:
kernel version:3.10.0-693.el7.x86_64
qemu-kvm-rhev：qemu-kvm-rhev-2.9.0-17.el7a.x86_64
Test steps:
1. start guest with cmd:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-machine pc \
-vga std \
-object iothread,id=iothread0 \
-object iothread,id=iothread1 \
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=03,iothread=iothread0 \
-drive id=drive_image1,if=none,snapshot=on,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1,bus=scsi0.0,lun=0 \
-device virtio-net-pci,mac=9a:b2:b3:b4:b5:b6,id=iduCv1Ln,vectors=4,netdev=idKgexFk,bus=pci.0,addr=05 \
-netdev tap,id=idKgexFk,vhost=on \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu host \
-vnc :1 \
-enable-kvm \
-monitor stdio \
-device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=04,iothread=iothread1 \
-drive id=data_image1,if=none,werror=stop,rerror=stop,cache=none,format=qcow2,file=/home/test.qcow2 \
-device scsi-hd,id=data1,drive=data_image1,bus=scsi1.0,lun=0 \
2. run dd test on data disk in guest.
dd if=/dev/zero of=/dev/sdb bs=4K count=1000000000 oflag=direct status=progress
3. during its io test，quit qemu
(qemu)quit
Test Result：
qemu core dump with msg：qemu-kvm: /builddir/build/BUILD/qemu-2.9.0/hw/scsi/virtio-scsi.c:245: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
gdb info:
[root@intel-e31225-16-3 home]# gdb -c core.16257
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
....
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name avocado-vt-vm1 -machine pc -vga std -object iothrea'.
Program terminated with signal 6, Aborted.
#0 0x00007f6abf37b1f7 in raise () from /lib64/libc.so.6
.....
(gdb) bt
#0 0x00007f6abf37b1f7 in raise () at /lib64/libc.so.6
#1 0x00007f6abf37c8e8 in abort () at /lib64/libc.so.6
#2 0x00007f6abf374266 in __assert_fail_base () at /lib64/libc.so.6
#3 0x00007f6abf374312 in () at /lib64/libc.so.6
#4 0x000055def692ac50 in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x55defafa7c00)
at /usr/src/debug/qemu-2.9.0/hw/scsi/virtio-scsi.c:245
#5 0x000055def69b4e16 in virtio_scsi_handle_cmd_vq (s=<optimized out>, s=<optimized out>, d=0x55defafa7c00)
at /usr/src/debug/qemu-2.9.0/hw/scsi/virtio-scsi.c:245
#6 0x000055def69b4e16 in virtio_scsi_handle_cmd_vq (req=0x55def8d98b40, s=0x55defb000510) at /usr/src/debug/qemu-2.9.0/hw/scsi/virtio-scsi.c:558
#7 0x000055def69b4e16 in virtio_scsi_handle_cmd_vq (s=s@entry=0x55defb000510, vq=vq@entry=0x55defb0ba100)
at /usr/src/debug/qemu-2.9.0/hw/scsi/virtio-scsi.c:598
#8 0x000055def69b59fa in virtio_scsi_data_plane_handle_cmd (vdev=<optimized out>, vq=0x55defb0ba100)
at /usr/src/debug/qemu-2.9.0/hw/scsi/virtio-scsi-dataplane.c:60
#9 0x000055def6c404c8 in aio_dispatch_handlers (ctx=ctx@entry=0x55def8069980) at util/aio-posix.c:399
#10 0x000055def6c40f0a in aio_poll (ctx=0x55def8069980, blocking=blocking@entry=true) at util/aio-posix.c:685
#11 0x000055def6a4933e in iothread_run (opaque=0x55def80c8bb0) at iothread.c:59
#12 0x00007f6abf710e25 in start_thread () at /lib64/libpthread.so.0
#13 0x00007f6abf43e34d in clone () at /lib64/libc.so.6
It also reproduced with backend of NBD on P9 host host : 4.14.0-24.el7a.ppc64le guest : 4.14.0-24.el7a.ppc64le qemu : qemu-kvm-rhev-2.10.0-16.el7.ppc64le [root@ibm-p9z-09 commands]# sh guest-9328.sh QEMU 2.10.0 monitor - type 'help' for more information (qemu) q qemu-kvm: /builddir/build/BUILD/qemu-2.10.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed. guest-9328.sh: line 30: 31098 Aborted /usr/libexec/qemu-kvm -name 'guest' -machine pseries-rhel7.5.0 -m 16G -nodefaults -smp 4,cores=4,threads=1,sockets=1 -boot order=cdn,once=d,menu=off,strict=off -device nec-usb-xhci,id=xhci0 -device usb-tablet,id=usb-tablet0 -device usb-kbd,id=usb-kbd0 -device VGA,id=vga0 -chardev socket,id=qmp_id_qmpmonitor,path=/var/tmp/qmp-cmd-monitor-yhong,server,nowait -mon chardev=qmp_id_qmpmonitor,mode=control -enable-kvm -object iothread,id=iothread0 -object iothread,id=iothread1 -device virtio-scsi-pci,id=scsi0,iothread=iothread0 -device virtio-scsi-pci,id=scsi1,iothread=iothread1 -drive file=nbd:10.19.19.53:10086,format=qcow2,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_system -device scsi-hd,bus=scsi0.0,drive=drive_system,id=system,bootindex=0 -drive file=nbd:10.19.19.53:20000,format=qcow2,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data0 -device scsi-hd,bus=scsi1.0,drive=drive_data0,id=data0 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=40:f2:e9:5d:9c:03 -qmp tcp:0:3000,server,nowait -chardev socket,id=serial_id_serial,path=/var/tmp/serial-yhong,server,nowait -device spapr-vty,reg=0x30000000,chardev=serial_id_serial -monitor stdio -vnc :30 Can reproduce on: kernel:3.10.0-826.el7.x86_64 + qemu-kvm-rhev:qemu-kvm-rhev-2.10.0-16.el7.x86_64 Test steps: same with comment 2. Test result: qemu core dump with core.1094 [root@intel-3323-24-1 home]# gdb -c core.1094 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-109.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. [New LWP 1097] [New LWP 1196] [New LWP 1096] [New LWP 1276] [New LWP 1194] [New LWP 1269] [New LWP 1204] [New LWP 1277] [New LWP 1094] [New LWP 1197] [New LWP 1195] [New LWP 1095] Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done. done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -name avocado-vt-vm1 -machine pc -vga std -object iothrea'. Program terminated with signal 6, Aborted. #0 0x00007f1a6bf941a7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install boost-system-1.53.0-27.el7.x86_64 boost-thread-1.53.0-27.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 elfutils-libelf-0.170-3.el7.x86_64 elfutils-libs-0.170-3.el7.x86_64 glib2-2.54.2-2.el7.x86_64 glibc-2.17-220.el7.x86_64 glusterfs-api-3.8.4-53.el7.x86_64 glusterfs-libs-3.8.4-53.el7.x86_64 gmp-6.0.0-15.el7.x86_64 gnutls-3.3.26-9.el7.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-18.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-49.el7.x86_64 libcacard-2.5.2-2.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-11.el7.x86_64 libcurl-7.29.0-46.el7.x86_64 libdb-5.3.21-22.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-25.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibverbs-15-2.el7.x86_64 libidn-1.28-4.el7.x86_64 libiscsi-1.9.0-7.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmount-2.23.2-49.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 librados2-0.94.5-2.el7.x86_64 librbd1-0.94.5-2.el7.x86_64 librdmacm-15-2.el7.x86_64 libseccomp-2.3.1-3.el7.x86_64 libselinux-2.5-12.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-25.el7.x86_64 libtasn1-4.10-1.el7.x86_64 libusbx-1.0.21-1.el7.x86_64 libuuid-2.23.2-49.el7.x86_64 lz4-1.7.5-2.el7.x86_64 lzo-2.06-8.el7.x86_64 nettle-2.7.1-8.el7.x86_64 nspr-4.17.0-1.el7.x86_64 nss-3.34.0-1.el7.x86_64 nss-softokn-freebl-3.34.0-1.el7.x86_64 nss-util-3.34.0-1.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 openldap-2.4.44-10.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 opus-1.0.2-6.el7.x86_64 p11-kit-0.23.5-3.el7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 snappy-1.1.0-3.el7.x86_64 spice-server-0.14.0-2.el7.x86_64 systemd-libs-219-51.el7.x86_64 usbredir-0.7.1-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f1a6bf941a7 in raise () at /lib64/libc.so.6 #1 0x00007f1a6bf95898 in abort () at /lib64/libc.so.6 #2 0x00007f1a6bf8cfc8 in __assert_fail_base () at /lib64/libc.so.6 #3 0x00007f1a6bf8d074 in () at /lib64/libc.so.6 #4 0x000055eeaa026447 in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x55eeade85400) at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:246 #5 0x000055eeaa0ab9a6 in virtio_scsi_handle_cmd_vq (s=<optimized out>, s=<optimized out>, d=0x55eeade85400) at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:246 #6 0x000055eeaa0ab9a6 in virtio_scsi_handle_cmd_vq (req=0x55eead80a780, s=0x55eeaea6a170) at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:559 #7 0x000055eeaa0ab9a6 in virtio_scsi_handle_cmd_vq (s=s@entry=0x55eeaea6a170, vq=vq@entry=0x55eeaea72100) at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:599 #8 0x000055eeaa0ac58a in virtio_scsi_data_plane_handle_cmd (vdev=<optimized out>, vq=0x55eeaea72100) at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi-dataplane.c:60 #9 0x000055eeaa0b8db6 in virtio_queue_host_notifier_aio_poll (vq=0x55eeaea72100) at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:1506 #10 0x000055eeaa0b8db6 in virtio_queue_host_notifier_aio_poll (opaque=0x55eeaea72168) at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:2420 #11 0x000055eeaa34c77e in run_poll_handlers_once (ctx=ctx@entry=0x55eeac69bcc0) at util/aio-posix.c:497 #12 0x000055eeaa34d1c5 in aio_poll (blocking=true, ctx=0x55eeac69bcc0) at util/aio-posix.c:573 #13 0x000055eeaa34d1c5 in aio_poll (ctx=0x55eeac69bcc0, blocking=blocking@entry=true) at util/aio-posix.c:602 #14 0x000055eeaa1407c6 in iothread_run (opaque=0x55eeac6a5340) at iothread.c:59 #15 0x00007f1a6c332dd5 in start_thread () at /lib64/libpthread.so.0 #16 0x00007f1a6c05c94d in clone () at /lib64/libc.so.6 Patch sent upstream: https://patchwork.ozlabs.org/patch/867549/ I hit more race conditions after backporting the fix from Comment#7. After additional debugging I sent another patch upstream. Patch sent upstream: https://patchwork.ozlabs.org/patch/875530/ *** Bug 1550335 has been marked as a duplicate of this bug. *** Verified, the problem has been resolved, set its status to "Verified". Test version: kernel:3.10.0-879.el7.x86_64 qemu-kvm-rhev:qemu-kvm-rhev-2.12.0-1.el7.x86_64 Test Steps: 1. Start guest with qemu cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine pc \ -vga std \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=03,iothread=iothread0 \ -drive id=drive_image1,if=none,snapshot=on,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel75-64-virtio.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1,bus=scsi0.0,lun=0 \ -device virtio-net-pci,mac=9a:b2:b3:b4:b5:b6,id=iduCv1Ln,vectors=4,netdev=idKgexFk,bus=pci.0,addr=05 \ -netdev tap,id=idKgexFk,vhost=on \ -m 4096 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu host \ -vnc :1 \ -enable-kvm \ -monitor stdio \ -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=04,iothread=iothread1 \ -drive id=data_image1,if=none,werror=stop,rerror=stop,cache=none,format=qcow2,file=/home/test.qcow2 \ -device scsi-hd,id=data1,drive=data_image1,bus=scsi1.0,lun=0 \ 2. Run io test on guest (guest)#dd if=/dev/zero of=/dev/sdb bs=4K count=1000000000 oflag=direct status=progress 3. Quit qemu during io testing (qemu)quit Test Result: Qemu quit without any error. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3443 |