RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1018537 - qemu core dump after run read/randwr fio in guest with usb or scsi disk
Summary: qemu core dump after run read/randwr fio in guest with usb or scsi disk
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.5
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: ---
Assignee: Fam Zheng
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-10-13 02:56 UTC by langfang
Modified: 2014-10-14 06:53 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.441.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-14 06:53:14 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1490 0 normal SHIPPED_LIVE qemu-kvm bug fix and enhancement update 2014-10-14 01:28:27 UTC

Description langfang 2013-10-13 02:56:53 UTC 
Description of problem:

Guest core dump after run  read/randwr usb disk use fio

Version-Release number of selected component (if applicable):

Host
# uname -r
2.6.32-423.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-28.el6.x86_64

Guest:6.4.z-32

kernel-2.6.32-358.24.1.el6.i686.rpm

How reproducible:

60%

Steps to Reproduce:
1.Boot guest with usb storage
#qemu-img create -f qcow2 usb.qcow2 800M

...-drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/home/test2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,id=scsi-disk -drive file=/home/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1...

2.IN guest 

Install fio 

#fio --filename=/dev/sdc --direct=1 --rw=read --bs=1M --size=10M --name=test --iodepth=1 
#fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=1M --size=10M --name=test --iodepth=1 



Actual results:
Guest core dump

...
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/usb-msd.c:356: usb_msd_cancel_io: Assertion `s->packet == p' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4c93925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.22-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64 db4-4.7.25-18.el6_4.x86_64 dbus-libs-1.2.24-7.el6_3.x86_64 flac-1.2.1-6.1.el6.x86_64 glib2-2.26.1-3.el6.x86_64 glibc-2.12-1.130.el6.x86_64 glusterfs-api-3.4.0.34rhs-1.el6.x86_64 glusterfs-libs-3.4.0.34rhs-1.el6.x86_64 gnutls-2.8.5-10.el6_4.2.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libICE-1.0.6-1.el6.x86_64 libSM-1.2.1-2.el6.x86_64 libX11-1.5.0-4.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libXext-1.3.1-2.el6.x86_64 libXi-1.6.1-3.el6.x86_64 libXtst-1.2.1-2.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libasyncns-0.8-1.1.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-1.el6.x86_64 libogg-1.1.4-2.1.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libsndfile-1.0.20-5.el6.x86_64 libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.14.el6.x86_64 libvorbis-1.2.3-4.el6_2.1.x86_64 libxcb-1.8.1-1.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 openssl-1.0.1e-15.el6.x86_64 pixman-0.26.2-5.el6_4.x86_64 pulseaudio-libs-0.9.21-14.el6_3.x86_64 spice-server-0.12.4-4.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff4c93925 in raise () from /lib64/libc.so.6
#1  0x00007ffff4c95105 in abort () from /lib64/libc.so.6
#2  0x00007ffff4c8ca4e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4c8cb10 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e43b74 in usb_msd_cancel_io (dev=<value optimized out>, p=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-msd.c:356
#5  0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff99cc9a8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#6  0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff99cc930, async=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#7  0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#8  ehci_advance_async_state (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#9  0x00007ffff7df8fc1 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#10 0x00007ffff7e01466 in qemu_aio_wait () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:145
#11 0x00007ffff7e016f5 in qemu_aio_flush () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:113
#12 0x00007ffff7e46fd2 in scsi_cancel_io (req=0x7ffff8da3190) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:105
#13 0x00007ffff7e44ee2 in scsi_req_cancel (req=0x7ffff8da3190) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-bus.c:1424
#14 0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff99cc9a8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#15 0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff99cc930, async=1) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#16 0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#17 ehci_advance_async_state (ehci=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#18 0x00007ffff7f25812 in ehci_frame_timer (opaque=0x7ffff9713430) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2213
#19 0x00007ffff7dc16ba in qemu_run_timers (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1339
#20 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4083
#21 0x00007ffff7de440a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
---Type <return> to continue, or q <return> to quit---
#22 0x00007ffff7dc42a9 in main_loop (argc=63, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#23 main (argc=63, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644


Expected results:
Guest work well

Additional info:
1)
MY CLI:
(gdb) r  -M rhel6.5.0 -cpu Opteron_G3 -m 2G -smp 4,sockets=2,cores=2,threads=1 -enable-kvm -usb -device usb-tablet,id=input0 -name rhel6.4-z-32 -uuid 0dc2ab15-843a-4b40-844e-615fd9219236 -rtc base=localtime,clock=host,driftfix=slew -drive file=/dev/vg-flang/lv-flang,format=raw,if=none,id=ide0 -device ide-drive,drive=ide0,bus=ide.0,unit=0,id=ide0-0-0,bootindex=0 -vnc :1 -monitor stdio -boot menu=on  -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,id=virtio-net-pci0,mac=92:31:61:E0:31:26,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0  -qmp tcp:0:4444,server,nowait -drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/home/test2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,id=scsi-disk -drive file=/home/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1
Starting program: /usr/libexec/qemu-kvm -M rhel6.5.0 -cpu Opteron_G3 -m 2G -smp 4,sockets=2,cores=2,threads=1 -enable-kvm -usb -device usb-tablet,id=input0 -name rhel6.4-z-32 -uuid 0dc2ab15-843a-4b40-844e-615fd9219236 -rtc base=localtime,clock=host,driftfix=slew -drive file=/dev/vg-flang/lv-flang,format=raw,if=none,id=ide0 -device ide-drive,drive=ide0,bus=ide.0,unit=0,id=ide0-0-0,bootindex=0 -vnc :1 -monitor stdio -boot menu=on  -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,id=virtio-net-pci0,mac=92:31:61:E0:31:26,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0  -qmp tcp:0:4444,server,nowait -drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/home/test2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,id=scsi-disk -drive file=/home/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/home/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1

2)Host
#cat /proc/cpuinfo
..
processor	: 3
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 16
model name	: AMD A10-5800K APU with Radeon(tm) HD Graphics  
stepping	: 1
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 2
apicid		: 19
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core cpb npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bmi1
bogomips	: 7585.93
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
Comment 2 Sibiao Luo 2013-10-14 02:32:40 UTC 
Hi flang,

    IIRC, i did not meet it when i run this testing, could you help check if is a regression issue or guest speciecied issue ?
BTW, i did not meet this issue in my intel host with qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64.
host info:
# uname -r && rpm -q qemu-kvm-rhev
2.6.32-422.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
guest info:
kernel-2.6.32-422.el6.x86_64

Best Regards,
sluo
Comment 3 langfang 2013-10-14 04:56:10 UTC 
(In reply to Sibiao Luo from comment #2)
> Hi flang,
> 
>     IIRC, i did not meet it when i run this testing, could you help check if
> is a regression issue or guest speciecied issue ?
> BTW, i did not meet this issue in my intel host with
> qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64.
> host info:
> # uname -r && rpm -q qemu-kvm-rhev
> 2.6.32-422.el6.x86_64
> qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
> guest info:
> kernel-2.6.32-422.el6.x86_64
> 
> Best Regards,
> sluo

Reproduce this bug as follow version:
host:
# uname -r
2.6.32-422.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.412.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-28.el6.x86_64
Guest

kernel-2.6.32-358.24.1.el6.i686.rpm

Steps:
1.boot guest
2.(qemu)block_set_io_throttle drive-usb-2-0 1000 0 0 0 0 0
3.In guest


#fio --filename=/dev/sdc --direct=1 --rw=read --bs=1M --size=10M --name=test --iodepth=1 
#fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=1M --size=10M --name=test --iodepth=1 

4.If can't reproduce ,please do step3 many times

Results:Guest
...
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/usb-msd.c:356: usb_msd_cancel_io: Assertion `s->packet == p' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4c93925 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff4c93925 in raise () from /lib64/libc.so.6
#1  0x00007ffff4c95105 in abort () from /lib64/libc.so.6
#2  0x00007ffff4c8ca4e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4c8cb10 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e43b74 in usb_msd_cancel_io (dev=<value optimized out>, 
    p=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-msd.c:356
#5  0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff9769918)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#6  0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff97698a0, async=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#7  0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#8  ehci_advance_async_state (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#9  0x00007ffff7df8fc1 in qemu_bh_poll ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#10 0x00007ffff7e01466 in qemu_aio_wait ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:145
#11 0x00007ffff7e016f5 in qemu_aio_flush ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:113
#12 0x00007ffff7e46fd2 in scsi_cancel_io (req=0x7ffff977d6d0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:105
#13 0x00007ffff7e44ee2 in scsi_req_cancel (req=0x7ffff977d6d0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-bus.c:1424
#14 0x00007ffff7e3cc0a in usb_cancel_packet (p=0x7ffff9769918)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#15 0x00007ffff7f237f9 in ehci_free_queue (q=0x7ffff97698a0, async=1)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#16 0x00007ffff7f2552d in ehci_queues_rip_unseen (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#17 ehci_advance_async_state (ehci=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#18 0x00007ffff7f25812 in ehci_frame_timer (opaque=0x7ffff9713430)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2213
#19 0x00007ffff7dc16ba in qemu_run_timers (timeout=1000)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1339
#20 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4083
#21 0x00007ffff7de440a in kvm_main_loop ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
#22 0x00007ffff7dc42a9 in main_loop (argc=63, argv=<value optimized out>, 
    envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#23 main (argc=63, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644


CLI as same as comment0
Comment 4 langfang 2013-10-16 03:42:02 UTC 
Same steps as comment 3 use scsi disk also hit qemu coredump

Steps:
1.BOOt guest with scsi disk 
 ...-drive file=/home/test3.qcow2,if=none,id=drive-scsi-disk-1,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk-1,bus=scsi0.0,scsi-id=0,id=scsi-disk-1

2.(qemu)block_set_io_throttle drive-scsi-disk-1 10000 0 0 0 0 0

3.In guest 

/dev/sdb--->scsi disk 

#fio --filename=/dev/sdb --direct=1 --rw=write --bs=1M --size=10M --name=test --



Resutls:

Wait about 6 min,qemu coredump

...
[New Thread 0x7fffef4c5700 (LWP 16014)]
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/scsi-disk.c:239: scsi_dma_complete: Assertion `r->req.aiocb != ((void *)0)' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4c93925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.22-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64 db4-4.7.25-18.el6_4.x86_64 dbus-libs-1.2.24-7.el6_3.x86_64 flac-1.2.1-6.1.el6.x86_64 glib2-2.26.1-3.el6.x86_64 glibc-2.12-1.130.el6.x86_64 glusterfs-api-3.4.0.34rhs-1.el6.x86_64 glusterfs-libs-3.4.0.34rhs-1.el6.x86_64 gnutls-2.8.5-10.el6_4.2.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libICE-1.0.6-1.el6.x86_64 libSM-1.2.1-2.el6.x86_64 libX11-1.5.0-4.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libXext-1.3.1-2.el6.x86_64 libXi-1.6.1-3.el6.x86_64 libXtst-1.2.1-2.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libasyncns-0.8-1.1.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-1.el6.x86_64 libogg-1.1.4-2.1.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libsndfile-1.0.20-5.el6.x86_64 libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.14.el6.x86_64 libvorbis-1.2.3-4.el6_2.1.x86_64 libxcb-1.8.1-1.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 openssl-1.0.1e-15.el6.x86_64 pixman-0.26.2-5.el6_4.x86_64 pulseaudio-libs-0.9.21-14.el6_3.x86_64 spice-server-0.12.4-4.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff4c93925 in raise () from /lib64/libc.so.6
#1  0x00007ffff4c95105 in abort () from /lib64/libc.so.6
#2  0x00007ffff4c8ca4e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4c8cb10 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e475b1 in scsi_dma_complete (opaque=0x7fffd8000910, ret=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:239
#5  0x00007ffff7f26e81 in dma_complete (dbs=0x7fffdc000db0, ret=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/dma-helpers.c:88
#6  0x00007ffff7f27052 in dma_bdrv_cb (opaque=0x7fffdc000db0, ret=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/dma-helpers.c:114
#7  0x00007ffff7dfc6ce in bdrv_co_em_bh (opaque=0x7fffdc000e50)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4009
#8  0x00007ffff7df8fc1 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#9  0x00007ffff7dc1629 in main_loop_wait (timeout=0) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4096
#10 0x00007ffff7de440a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
#11 0x00007ffff7dc42a9 in main_loop (argc=45, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#12 main (argc=45, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644
(gdb)
Comment 5 Fam Zheng 2013-10-16 08:51:51 UTC 
I can reproduce the case in comment #4, but it takes a while like an hour or so to repeat the fio workload as above, with io throttled (bps=10000).

The crashing code is in scsi io completion/cancellation code, don't have any conclusion looking at the backtrace and context code, yet but it seems like an unexpected second run of completion code path on a request timeout, which triggers assertion.

Resetting assignee to get it triaged again.

Fam
Comment 15 Fam Zheng 2014-08-12 08:55:34 UTC 
Sorry, I should have provided you the RHEV build, because IO throttling is disabled in RHEL:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7831919

Please try again,
Fam
Comment 16 langfang 2014-08-13 05:49:21 UTC 
(In reply to Fam Zheng from comment #15)
> Sorry, I should have provided you the RHEV build, because IO throttling is
> disabled in RHEL:
> 
> http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7831919
> 
> Please try again,
> Fam

Test above build 

Version:
Host:
# uname -r
2.6.32-431.29.2.el6.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.436.el6.test.x86_64

guest:
2.6.32-431.el6.x86_64

Steps:
1.Boot guest with usb storage

#qemu-img create -f qcow2 usb.qcow2 800M
... 
 -drive file=/home/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none, -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1

2.(qemu)block_set_io_throttle drive-scsi-disk-1 10000 0 0 0 0 0

3.In guest 

/dev/sdb--->usb disk

#fio --filename=/dev/sdb --direct=1 --rw=write --bs=1M --size=10M --name=test --



Resutls:guest run fio about 2 hours,work well,qemu not core dump
Comment 17 Fam Zheng 2014-08-13 08:04:30 UTC 
Thanks for the update!

Fam
Comment 18 Jeff Nelson 2014-08-25 18:27:32 UTC 
Fix included in qemu-kvm-0.12.1.2-2.441.el6
Comment 20 mazhang 2014-08-26 03:14:35 UTC 
Reproduced this bug.

Host:
qemu-kvm-rhev-tools-0.12.1.2-2.438.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.438.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
qemu-img-rhev-0.12.1.2-2.438.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.438.el6.x86_64
kernel-2.6.32-497.el6.x86_64

Guest:
kernel-2.6.32-497.el6.x86_64

Steps:
1. boot vm:
gdb --args /usr/libexec/qemu-kvm \
-machine rhel6.6.0,dump-guest-core=off \
-cpu SandyBridge \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1,maxcpus=160 \
-enable-kvm \
-name rhel6.6 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-monitor unix:/tmp/monitor2,server,nowait \
-vga qxl \
-spice port=5900,disable-ticketing \
-usb \
-device usb-tablet,id=tablet0 \
-device virtio-scsi-pci,id=si0 \
-drive file=/home/RHEL-Server-6.6-64-1.qcow2,if=none,media=disk,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native \
-device scsi-hd,drive=drive-scsi-disk,bus=si0.0,id=scsi-disk0,bootindex=0 \
-device usb-ehci,id=ehci \
-drive file=/home/storage.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none \
-device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1 \
-netdev tap,id=hostnet0,vhost=on \
-device e1000,netdev=hostnet0,id=net0,mac=00:01:02:B6:40:23 \

2. (qemu)block_set_io_throttle drive-scsi-disk-1 10000 0 0 0 0 0

3. Fio test in guest.
#fio --filename=/dev/sdb --direct=1 --rw=read --bs=1M --size=10M --name=test

Result:
qemu-kvm core dumped.
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/usb-msd.c:356: usb_msd_cancel_io: Assertion `s->packet == p' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff4836915 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install alsa-lib-1.0.22-3.el6.x86_64 celt051-0.5.1.3-0.el6.x86_64 cyrus-sasl-gssapi-2.1.23-15.el6.x86_64 cyrus-sasl-lib-2.1.23-15.el6.x86_64 cyrus-sasl-md5-2.1.23-15.el6.x86_64 cyrus-sasl-plain-2.1.23-15.el6.x86_64 db4-4.7.25-18.el6_4.x86_64 dbus-libs-1.2.24-7.el6_3.x86_64 flac-1.2.1-6.1.el6.x86_64 glib2-2.28.8-1.el6.x86_64 glibc-2.12-1.148.el6.x86_64 glusterfs-api-3.6.0.27-1.el6.x86_64 glusterfs-libs-3.6.0.27-1.el6.x86_64 gnutls-2.8.5-14.el6_5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-31.el6.x86_64 libICE-1.0.6-1.el6.x86_64 libSM-1.2.1-2.el6.x86_64 libX11-1.6.0-2.2.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libXext-1.3.2-2.1.el6.x86_64 libXi-1.7.2-2.2.el6.x86_64 libXtst-1.2.2-2.1.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libasyncns-0.8-1.1.el6.x86_64 libcom_err-1.41.12-20.el6.x86_64 libgcc-4.4.7-10.el6.x86_64 libgcrypt-1.4.5-11.el6_4.x86_64 libgpg-error-1.7-4.el6.x86_64 libjpeg-turbo-1.2.1-3.el6_5.x86_64 libogg-1.1.4-2.1.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libsndfile-1.0.20-5.el6.x86_64 libstdc++-4.4.7-10.el6.x86_64 libtasn1-2.3-6.el6_5.x86_64 libuuid-2.17.2-12.18.el6.x86_64 libvorbis-1.2.3-4.el6_2.1.x86_64 libxcb-1.9.1-2.el6.x86_64 lzo-2.03-3.1.el6_5.1.x86_64 nss-softokn-freebl-3.14.3-15.el6.x86_64 openssl-1.0.1e-28.el6.x86_64 pixman-0.32.4-4.el6.x86_64 pulseaudio-libs-0.9.21-17.el6.x86_64 snappy-1.1.0-1.el6.x86_64 spice-server-0.12.4-11.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 usbredir-0.5.1-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff4836915 in raise () from /lib64/libc.so.6
#1  0x00007ffff48380f5 in abort () from /lib64/libc.so.6
#2  0x00007ffff482fa3e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff482fb00 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7e36fd4 in usb_msd_cancel_io (dev=<value optimized out>, p=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-msd.c:356
#5  0x00007ffff7e2fcaa in usb_cancel_packet (p=0x7fffffdd2be8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#6  0x00007ffff7f1d6d9 in ehci_free_queue (q=0x7fffffdd2b70, async=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#7  0x00007ffff7f1f40d in ehci_queues_rip_unseen (ehci=0x7ffff9473920)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#8  ehci_advance_async_state (ehci=0x7ffff9473920) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#9  0x00007ffff7de7101 in qemu_bh_poll () at /usr/src/debug/qemu-kvm-0.12.1.2/async.c:70
#10 0x00007ffff7def756 in qemu_aio_wait () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:145
#11 0x00007ffff7def9e5 in qemu_aio_flush () at /usr/src/debug/qemu-kvm-0.12.1.2/aio.c:113
#12 0x00007ffff7e3a6c2 in scsi_cancel_io (req=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-disk.c:105
#13 0x00007ffff7e38702 in scsi_req_cancel (req=0x7ffff91bed70) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/scsi-bus.c:1488
#14 0x00007ffff7e2fcaa in usb_cancel_packet (p=0x7fffffdd2be8) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb.c:356
#15 0x00007ffff7f1d6d9 in ehci_free_queue (q=0x7fffffdd2b70, async=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:681
#16 0x00007ffff7f1f40d in ehci_queues_rip_unseen (ehci=0x7ffff9473920)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:735
#17 ehci_advance_async_state (ehci=0x7ffff9473920) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2075
#18 0x00007ffff7f1f6f2 in ehci_frame_timer (opaque=0x7ffff9473920)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/usb-ehci.c:2213
#19 0x00007ffff7daed4a in qemu_run_timers (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1341
#20 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4085
#21 0x00007ffff7dd24ea in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2258
#22 0x00007ffff7db3767 in main_loop (argc=<value optimized out>, argv=<value optimized out>, 
    envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4268
#23 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6725

scsi disk also hit this problem.
Comment 21 mazhang 2014-08-26 05:26:51 UTC 
Verify this bug on qemu-kvm-rhev-0.12.1.2-2.441.el6.x86_64.

Host:
qemu-img-rhev-0.12.1.2-2.441.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.441.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.441.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.441.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
kernel-2.6.32-497.el6.x86_64

Guest:
kernel-2.6.32-497.el6.x86_64

Result:
Both usb-storage and scsi-hd works well, the problem has gone.

So this bug has been fixed.
Comment 22 errata-xmlrpc 2014-10-14 06:53:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html
Note You need to log in before you can comment on or make changes to this bug.