Bug 1018537 - qemu core dump after run read/randwr fio in guest with usb or scsi disk
qemu core dump after run read/randwr fio in guest with usb or scsi disk
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.5
Unspecified Unspecified
medium Severity high
: rc
: ---
Assigned To: Fam Zheng
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-12 22:56 EDT by langfang
Modified: 2014-10-14 02:53 EDT (History)
13 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.441.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-10-14 02:53:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description langfang 2013-10-12 22:56:53 EDT
Description of problem:

Guest core dump after run  read/randwr usb disk use fio

Version-Release number of selected component (if applicable):

Host
# uname -r
2.6.32-423.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-28.el6.x86_64

Guest:6.4.z-32

kernel-2.6.32-358.24.1.el6.i686.rpm

How reproducible:

60%

Steps to Reproduce:
1.Boot guest with usb storage
#qemu-img create -f qcow2 usb.qcow2 800M

...-drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/home/test2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,id=scsi-disk -drive file=/home/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1...

2.IN guest 

Install fio 

#fio --filename=/dev/sdc --direct=1 --rw=read --bs=1M --size=10M --name=test --iodepth=1 
#fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=1M --size=10M --name=test --iodepth=1 



Actual results:
Guest core dump

...
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/usb-msd.c:356: usb_msd_cancel_io: Assertion `s->packet == p' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4c93925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.22-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64 db4-4.7.25-18.el6_4.x86_64 dbus-libs-1.2.24-7.el6_3.x86_64 flac-1.2.1-6.1.el6.x86_64 glib2-2.26.1-3.el6.x86_64 glibc-2.12-1.130.el6.x86_64 glusterfs-api-3.4.0.34rhs-1.el6.x86_64 glusterfs-libs-3.4.0.34rhs-1.el6.x86_64 gnutls-2.8.5-10.el6_4.2.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libICE-1.0.6-1.el6.x86_64 libSM-1.2.1-2.el6.x86_64 libX11-1.5.0-4.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libXext-1.3.1-2.el6.x86_64 libXi-1.6.1-3.el6.x86_64 libXtst-1.2.1-2.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libasyncns-0.8-1.1.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-1.el6.x86_64 libogg-1.1.4-2.1.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libsndfile-1.0.20-5.el6.x86_64 libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.14.el6.x86_64 libvorbis-1.2.3-4.el6_2.1.x86_64 libxcb-1.8.1-1.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 openssl-1.0.1e-15.el6.x86_64 pixman-0.26.2-5.el6_4.x86_64 pulseaudio-libs-0.9.21-14.el6_3.x86_64 spice-server-0.12.4-4.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff4c93925 in raise () from /lib64/libc.so.6
#1  0x00007ffff4c95105 in abort () from /lib64/libc.so.6
#2  0x00007ffff4c8ca4e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4c8cb10 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e43b74 in usb_msd_cancel_io (dev=<value optimized out>, p=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-msd.c:356
#5  0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff99cc9a8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#6  0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff99cc930, async=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#7  0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#8  ehci_advance_async_state (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#9  0x00007ffff7df8fc1 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#10 0x00007ffff7e01466 in qemu_aio_wait () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:145
#11 0x00007ffff7e016f5 in qemu_aio_flush () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:113
#12 0x00007ffff7e46fd2 in scsi_cancel_io (req=0x7ffff8da3190) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:105
#13 0x00007ffff7e44ee2 in scsi_req_cancel (req=0x7ffff8da3190) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-bus.c:1424
#14 0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff99cc9a8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#15 0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff99cc930, async=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#16 0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#17 ehci_advance_async_state (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#18 0x00007ffff7f25812 in ehci_frame_timer (opaque=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2213
#19 0x00007ffff7dc16ba in qemu_run_timers (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1339
#20 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4083
#21 0x00007ffff7de440a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
---Type <return> to continue, or q <return> to quit---
#22 0x00007ffff7dc42a9 in main_loop (argc=63, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#23 main (argc=63, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644


Expected results:
Guest work well

Additional info:
1)
MY CLI:
(gdb) r  -M rhel6.5.0 -cpu Opteron_G3 -m 2G -smp 4,sockets=2,cores=2,threads=1 -enable-kvm -usb -device usb-tablet,id=input0 -name rhel6.4-z-32 -uuid 0dc2ab15-843a-4b40-844e-615fd9219236 -rtc base=localtime,clock=host,driftfix=slew -drive file=/dev/vg-flang/lv-flang,format=raw,if=none,id=ide0 -device ide-drive,drive=ide0,bus=ide.0,unit=0,id=ide0-0-0,bootindex=0 -vnc :1 -monitor stdio -boot menu=on  -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,id=virtio-net-pci0,mac=92:31:61:E0:31:26,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0  -qmp tcp:0:4444,server,nowait -drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/home/test2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,id=scsi-disk -drive file=/home/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1
Starting program: /usr/libexec/qemu-kvm -M rhel6.5.0 -cpu Opteron_G3 -m 2G -smp 4,sockets=2,cores=2,threads=1 -enable-kvm -usb -device usb-tablet,id=input0 -name rhel6.4-z-32 -uuid 0dc2ab15-843a-4b40-844e-615fd9219236 -rtc base=localtime,clock=host,driftfix=slew -drive file=/dev/vg-flang/lv-flang,format=raw,if=none,id=ide0 -device ide-drive,drive=ide0,bus=ide.0,unit=0,id=ide0-0-0,bootindex=0 -vnc :1 -monitor stdio -boot menu=on  -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,id=virtio-net-pci0,mac=92:31:61:E0:31:26,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0  -qmp tcp:0:4444,server,nowait -drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/home/test2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,id=scsi-disk -drive file=/home/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1

2)Host
#cat /proc/cpuinfo
..
processor	: 3
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 16
model name	: AMD A10-5800K APU with Radeon(tm) HD Graphics  
stepping	: 1
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 2
apicid		: 19
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core cpb npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bmi1
bogomips	: 7585.93
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
Comment 2 Sibiao Luo 2013-10-13 22:32:40 EDT
Hi flang,

    IIRC, i did not meet it when i run this testing, could you help check if is a regression issue or guest speciecied issue ?
BTW, i did not meet this issue in my intel host with qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64.
host info:
# uname -r && rpm -q qemu-kvm-rhev
2.6.32-422.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
guest info:
kernel-2.6.32-422.el6.x86_64

Best Regards,
sluo
Comment 3 langfang 2013-10-14 00:56:10 EDT
(In reply to Sibiao Luo from comment #2)
> Hi flang,
> 
>     IIRC, i did not meet it when i run this testing, could you help check if
> is a regression issue or guest speciecied issue ?
> BTW, i did not meet this issue in my intel host with
> qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64.
> host info:
> # uname -r && rpm -q qemu-kvm-rhev
> 2.6.32-422.el6.x86_64
> qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
> guest info:
> kernel-2.6.32-422.el6.x86_64
> 
> Best Regards,
> sluo

Reproduce this bug as follow version:
host:
# uname -r
2.6.32-422.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-28.el6.x86_64
Guest

kernel-2.6.32-358.24.1.el6.i686.rpm

Steps:
1.boot guest
2.(qemu)block_set_io_throttle drive-usb-2-0 1000 0 0 0 0 0
3.In guest


#fio --filename=/dev/sdc --direct=1 --rw=read --bs=1M --size=10M --name=test --iodepth=1 
#fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=1M --size=10M --name=test --iodepth=1 

4.If can't reproduce ,please do step3 many times

Results:Guest
...
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/usb-msd.c:356: usb_msd_cancel_io: Assertion `s->packet == p' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4c93925 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff4c93925 in raise () from /lib64/libc.so.6
#1  0x00007ffff4c95105 in abort () from /lib64/libc.so.6
#2  0x00007ffff4c8ca4e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4c8cb10 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e43b74 in usb_msd_cancel_io (dev=<value optimized out>, 
    p=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-msd.c:356
#5  0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff9769918)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#6  0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff97698a0, async=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#7  0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#8  ehci_advance_async_state (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#9  0x00007ffff7df8fc1 in qemu_bh_poll ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#10 0x00007ffff7e01466 in qemu_aio_wait ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:145
#11 0x00007ffff7e016f5 in qemu_aio_flush ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:113
#12 0x00007ffff7e46fd2 in scsi_cancel_io (req=0x7ffff977d6d0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:105
#13 0x00007ffff7e44ee2 in scsi_req_cancel (req=0x7ffff977d6d0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-bus.c:1424
#14 0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff9769918)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#15 0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff97698a0, async=1)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#16 0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#17 ehci_advance_async_state (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#18 0x00007ffff7f25812 in ehci_frame_timer (opaque=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2213
#19 0x00007ffff7dc16ba in qemu_run_timers (timeout=1000)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1339
#20 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4083
#21 0x00007ffff7de440a in kvm_main_loop ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
#22 0x00007ffff7dc42a9 in main_loop (argc=63, argv=<value optimized out>, 
    envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#23 main (argc=63, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644


CLI as same as comment0
Comment 4 langfang 2013-10-15 23:42:02 EDT
Same steps as comment 3 use scsi disk also hit qemu coredump

Steps:
1.BOOt guest with scsi disk 
 ...-drive file=/home/test3.qcow2,if=none,id=drive-scsi-disk-1,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk-1,bus=scsi0.0,scsi-id=0,id=scsi-disk-1

2.(qemu)block_set_io_throttle drive-scsi-disk-1 10000 0 0 0 0 0

3.In guest 

/dev/sdb--->scsi disk 

#fio --filename=/dev/sdb --direct=1 --rw=write --bs=1M --size=10M --name=test --



Resutls:

Wait about 6 min,qemu coredump

...
[New Thread 0x7fffef4c5700 (LWP 16014)]
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/scsi-disk.c:239: scsi_dma_complete: Assertion `r->req.aiocb != ((void *)0)' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4c93925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.22-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64 db4-4.7.25-18.el6_4.x86_64 dbus-libs-1.2.24-7.el6_3.x86_64 flac-1.2.1-6.1.el6.x86_64 glib2-2.26.1-3.el6.x86_64 glibc-2.12-1.130.el6.x86_64 glusterfs-api-3.4.0.34rhs-1.el6.x86_64 glusterfs-libs-3.4.0.34rhs-1.el6.x86_64 gnutls-2.8.5-10.el6_4.2.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libICE-1.0.6-1.el6.x86_64 libSM-1.2.1-2.el6.x86_64 libX11-1.5.0-4.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libXext-1.3.1-2.el6.x86_64 libXi-1.6.1-3.el6.x86_64 libXtst-1.2.1-2.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libasyncns-0.8-1.1.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-1.el6.x86_64 libogg-1.1.4-2.1.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libsndfile-1.0.20-5.el6.x86_64 libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.14.el6.x86_64 libvorbis-1.2.3-4.el6_2.1.x86_64 libxcb-1.8.1-1.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 openssl-1.0.1e-15.el6.x86_64 pixman-0.26.2-5.el6_4.x86_64 pulseaudio-libs-0.9.21-14.el6_3.x86_64 spice-server-0.12.4-4.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff4c93925 in raise () from /lib64/libc.so.6
#1  0x00007ffff4c95105 in abort () from /lib64/libc.so.6
#2  0x00007ffff4c8ca4e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4c8cb10 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e475b1 in scsi_dma_complete (opaque=0x7fffd8000910, ret=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:239
#5  0x00007ffff7f26e81 in dma_complete (dbs=0x7fffdc000db0, ret=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/dma-helpers.c:88
#6  0x00007ffff7f27052 in dma_bdrv_cb (opaque=0x7fffdc000db0, ret=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/dma-helpers.c:114
#7  0x00007ffff7dfc6ce in bdrv_co_em_bh (opaque=0x7fffdc000e50)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4009
#8  0x00007ffff7df8fc1 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#9  0x00007ffff7dc1629 in main_loop_wait (timeout=0) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4096
#10 0x00007ffff7de440a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
#11 0x00007ffff7dc42a9 in main_loop (argc=45, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#12 main (argc=45, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644
(gdb)
Comment 5 Fam Zheng 2013-10-16 04:51:51 EDT
I can reproduce the case in comment #4, but it takes a while like an hour or so to repeat the fio workload as above, with io throttled (bps=10000).

The crashing code is in scsi io completion/cancellation code, don't have any conclusion looking at the backtrace and context code, yet but it seems like an unexpected second run of completion code path on a request timeout, which triggers assertion.

Resetting assignee to get it triaged again.

Fam
Comment 15 Fam Zheng 2014-08-12 04:55:34 EDT
Sorry, I should have provided you the RHEV build, because IO throttling is disabled in RHEL:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7831919

Please try again,
Fam
Comment 16 langfang 2014-08-13 01:49:21 EDT
(In reply to Fam Zheng from comment #15)
> Sorry, I should have provided you the RHEV build, because IO throttling is
> disabled in RHEL:
> 
> http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7831919
> 
> Please try again,
> Fam

Test above build 

Version:
Host:
# uname -r
2.6.32-431.29.2.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.436.el6.test.x86_64

guest:
2.6.32-431.el6.x86_64

Steps:
1.Boot guest with usb storage

#qemu-img create -f qcow2 usb.qcow2 800M
... 
 -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1

2.(qemu)block_set_io_throttle drive-scsi-disk-1 10000 0 0 0 0 0

3.In guest 

/dev/sdb--->usb disk

#fio --filename=/dev/sdb --direct=1 --rw=write --bs=1M --size=10M --name=test --



Resutls:guest run fio about 2 hours,work well,qemu not core dump
Comment 17 Fam Zheng 2014-08-13 04:04:30 EDT
Thanks for the update!

Fam
Comment 18 Jeff Nelson 2014-08-25 14:27:32 EDT
Fix included in qemu-kvm-0.12.1.2-2.441.el6
Comment 20 mazhang 2014-08-25 23:14:35 EDT
Reproduced this bug.

Host:
qemu-kvm-rhev-tools-0.12.1.2-2.438.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.438.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
qemu-img-rhev-0.12.1.2-2.438.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.438.el6.x86_64
kernel-2.6.32-497.el6.x86_64

Guest:
kernel-2.6.32-497.el6.x86_64

Steps:
1. boot vm:
gdb --args /usr/libexec/qemu-kvm \
-machine rhel6.6.0,dump-guest-core=off \
-cpu SandyBridge \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1,maxcpus=160 \
-enable-kvm \
-name rhel6.6 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-monitor unix:/tmp/monitor2,server,nowait \
-vga qxl \
-spice port=5900,disable-ticketing \
-usb \
-device usb-tablet,id=tablet0 \
-device virtio-scsi-pci,id=si0 \
-drive file=/home/RHEL-Server-6.6-64-1.qcow2,if=none,media=disk,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native \
-device scsi-hd,drive=drive-scsi-disk,bus=si0.0,id=scsi-disk0,bootindex=0 \
-device usb-ehci,id=ehci \
-drive file=/home/storage.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none \
-device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1 \
-netdev tap,id=hostnet0,vhost=on \
-device e1000,netdev=hostnet0,id=net0,mac=00:01:02:B6:40:23 \

2. (qemu)block_set_io_throttle drive-scsi-disk-1 10000 0 0 0 0 0

3. Fio test in guest.
#fio --filename=/dev/sdb --direct=1 --rw=read --bs=1M --size=10M --name=test

Result:
qemu-kvm core dumped.
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/usb-msd.c:356: usb_msd_cancel_io: Assertion `s->packet == p' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4836915 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.22-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-gssapi-2.1.23-15.el6.x86_64 cyrus-sasl-lib-2.1.23-15.el6.x86_64 cyrus-sasl-md5-2.1.23-15.el6.x86_64 cyrus-sasl-plain-2.1.23-15.el6.x86_64 db4-4.7.25-18.el6_4.x86_64 dbus-libs-1.2.24-7.el6_3.x86_64 flac-1.2.1-6.1.el6.x86_64 glib2-2.28.8-1.el6.x86_64 glibc-2.12-1.148.el6.x86_64 glusterfs-api-3.6.0.27-1.el6.x86_64 glusterfs-libs-3.6.0.27-1.el6.x86_64 gnutls-2.8.5-14.el6_5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-31.el6.x86_64 libICE-1.0.6-1.el6.x86_64 libSM-1.2.1-2.el6.x86_64 libX11-1.6.0-2.2.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libXext-1.3.2-2.1.el6.x86_64 libXi-1.7.2-2.2.el6.x86_64 libXtst-1.2.2-2.1.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libasyncns-0.8-1.1.el6.x86_64 libcom_err-1.41.12-20.el6.x86_64 libgcc-4.4.7-10.el6.x86_64 libgcrypt-1.4.5-11.el6_4.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-3.el6_5.x86_64 libogg-1.1.4-2.1.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libsndfile-1.0.20-5.el6.x86_64 libstdc++-4.4.7-10.el6.x86_64 libtasn1-2.3-6.el6_5.x86_64 libuuid-2.17.2-12.18.el6.x86_64 libvorbis-1.2.3-4.el6_2.1.x86_64 libxcb-1.9.1-2.el6.x86_64 lzo-2.03-3.1.el6_5.1.x86_64 nss-softokn-freebl-3.14.3-15.el6.x86_64 openssl-1.0.1e-28.el6.x86_64 pixman-0.32.4-4.el6.x86_64 pulseaudio-libs-0.9.21-17.el6.x86_64 snappy-1.1.0-1.el6.x86_64 spice-server-0.12.4-11.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff4836915 in raise () from /lib64/libc.so.6
#1  0x00007ffff48380f5 in abort () from /lib64/libc.so.6
#2  0x00007ffff482fa3e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff482fb00 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e36fd4 in usb_msd_cancel_io (dev=<value optimized out>, p=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-msd.c:356
#5  0x00007ffff7e2fcaa in usb_cancel_packet (p=0x7fffffdd2be8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#6  0x00007ffff7f1d6d9 in ehci_free_queue (q=0x7fffffdd2b70, async=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#7  0x00007ffff7f1f40d in ehci_queues_rip_unseen (ehci=0x7ffff9473920)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#8  ehci_advance_async_state (ehci=0x7ffff9473920) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#9  0x00007ffff7de7101 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#10 0x00007ffff7def756 in qemu_aio_wait () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:145
#11 0x00007ffff7def9e5 in qemu_aio_flush () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:113
#12 0x00007ffff7e3a6c2 in scsi_cancel_io (req=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:105
#13 0x00007ffff7e38702 in scsi_req_cancel (req=0x7ffff91bed70) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-bus.c:1488
#14 0x00007ffff7e2fcaa in usb_cancel_packet (p=0x7fffffdd2be8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#15 0x00007ffff7f1d6d9 in ehci_free_queue (q=0x7fffffdd2b70, async=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#16 0x00007ffff7f1f40d in ehci_queues_rip_unseen (ehci=0x7ffff9473920)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#17 ehci_advance_async_state (ehci=0x7ffff9473920) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#18 0x00007ffff7f1f6f2 in ehci_frame_timer (opaque=0x7ffff9473920)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2213
#19 0x00007ffff7daed4a in qemu_run_timers (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1341
#20 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4085
#21 0x00007ffff7dd24ea in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2258
#22 0x00007ffff7db3767 in main_loop (argc=<value optimized out>, argv=<value optimized out>, 
    envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4268
#23 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6725

scsi disk also hit this problem.
Comment 21 mazhang 2014-08-26 01:26:51 EDT
Verify this bug on qemu-kvm-rhev-0.12.1.2-2.441.el6.x86_64.

Host:
qemu-img-rhev-0.12.1.2-2.441.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.441.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.441.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.441.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
kernel-2.6.32-497.el6.x86_64

Guest:
kernel-2.6.32-497.el6.x86_64

Result:
Both usb-storage and scsi-hd works well, the problem has gone.

So this bug has been fixed.
Comment 22 errata-xmlrpc 2014-10-14 02:53:14 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html

Note You need to log in before you can comment on or make changes to this bug.