Bug 1075846

Summary: qemu-kvm core dumped when hotplug/unhotplug USB3.0 device multi times
Product: Red Hat Enterprise Linux 7 Reporter: ShupingCui <scui>
Component: qemu-kvmAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: chayang, hhuang, huding, juzhang, knoel, michen, mrezanin, qiguo, rbalakri, scui, sluo, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-1.5.3-76.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 08:05:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1103193, 1146483, 1146486    

Description ShupingCui 2014-03-13 02:17:35 UTC
Description of problem:
qemu-kvm core dumped when hotplug/unhotplug USB3.0 multi times

Version-Release number of selected component (if applicable):
[root@localhost ~]# uname -r
3.10.0-105.el7.x86_64
[root@localhost ~]# rpm -qa qemu-kvm
qemu-kvm-1.5.3-52.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. boot rhel7 guest
/usr/libexec/qemu-kvm \
    -name 'virt-tests-vm1'  \
    -sandbox off  \
    -M pc  \
    -nodefaults  \
    -vga cirrus  \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
    -device nec-usb-xhci,id=usbtest,bus=pci.0,addr=04 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.0-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=05 \
    -device virtio-net-pci,mac=9a:69:6a:6b:6c:6d,id=idrW0T8P,netdev=idAiGqdM,bus=pci.0,addr=06  \
    -netdev tap,id=idAiGqdM,vhost=on,vhostfd=26,fd=25  \
    -m 4096  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'SandyBridge',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device usb-hub,id=usb-hub1,bus=usbtest.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off  \
    -no-kvm-pit-reinjection \
    -enable-kvm
2. Plugin usb device
(qemu) device_add usb-kbd,bus=usbtest.0,id=usbplugdev
3. Verify usb device is pluged on guest
# udevadm settle
# dmesg -c
 [   24.458130] usb 2-2: new full-speed USB device number 3 using xhci_hcd
 [   24.470611] usb 2-2: New USB device found, idVendor=0627, idProduct=0001
 [   24.472033] usb 2-2: New USB device strings: Mfr=1, Product=4, SerialNumber=5
 [   24.473532] usb 2-2: Product: QEMU USB Keyboard
 [   24.474483] usb 2-2: Manufacturer: QEMU
 [   24.475287] usb 2-2: SerialNumber: 42
 [   24.476499] usb 2-2: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
 [   24.478778] input: QEMU QEMU USB Keyboard as /devices/pci0000:00/0000:00:04.0/usb2/2-2/2-2:1.0/input/input5
 [   24.480972] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:04.0-2/input0
4. Unplug usb device
(qemu) device_del usbplugdev
5. repeat step2 - step4 multi times(>5)

Actual results:
qemu-kvm core dump
(gdb) bt
#0  0x00007fab4498f3da in object_class_dynamic_cast_assert (class=0x7fab489b47f0, typename=typename@entry=0x7fab44af1949 "usb-device", 
    file=file@entry=0x7fab44af1929 "hw/usb/bus.c", line=line@entry=197, func=func@entry=0x7fab44af1e10 <__func__.27231> "usb_device_ep_stopped") at qom/object.c:528
#1  0x00007fab4492ecc3 in usb_device_ep_stopped (dev=0x7fab488a9820, ep=0x7fab488aa950) at hw/usb/bus.c:197
#2  0x00007fab4493e788 in xhci_ep_nuke_xfers (xhci=xhci@entry=0x7faa1fd38010, slotid=slotid@entry=2, epid=<optimized out>) at hw/usb/hcd-xhci.c:1336
#3  0x00007fab44942479 in xhci_stop_ep (epid=<optimized out>, slotid=2, xhci=0x7faa1fd38010) at hw/usb/hcd-xhci.c:1396
#4  xhci_process_commands (xhci=0x7faa1fd38010) at hw/usb/hcd-xhci.c:2571
#5  0x00007fab44a0dc83 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fab371bda40, size=size@entry=4, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7fab44a0e1a0 <memory_region_write_accessor>, opaque=opaque@entry=0x7faa1fd38a30)
    at /usr/src/debug/qemu-1.5.3/memory.c:365
#6  0x00007fab44a121cb in memory_region_dispatch_write (size=4, data=0, addr=0, mr=0x7faa1fd38a30) at /usr/src/debug/qemu-1.5.3/memory.c:917
#7  io_mem_write (mr=0x7faa1fd38a30, addr=0, val=<optimized out>, size=4) at /usr/src/debug/qemu-1.5.3/memory.c:1592
#8  0x00007fab44a0dc83 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fab371bdae0, size=size@entry=4, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7fab44a0e1a0 <memory_region_write_accessor>, opaque=opaque@entry=0x7fab2c007340)
    at /usr/src/debug/qemu-1.5.3/memory.c:365
#9  0x00007fab44a121cb in memory_region_dispatch_write (size=4, data=0, addr=0, mr=0x7fab2c007340) at /usr/src/debug/qemu-1.5.3/memory.c:917
#10 io_mem_write (mr=0x7fab2c007340, addr=0, val=<optimized out>, size=size@entry=4) at /usr/src/debug/qemu-1.5.3/memory.c:1592
#11 0x00007fab449c978d in address_space_rw (as=as@entry=0x7fab456e2740 <address_space_memory>, addr=4273807360, 
    buf=buf@entry=0x7fab44696028 <Address 0x7fab44696028 out of bounds>, len=4, is_write=true) at /usr/src/debug/qemu-1.5.3/exec.c:1987
#12 0x00007fab449c97fd in cpu_physical_memory_rw (addr=<optimized out>, buf=buf@entry=0x7fab44696028 <Address 0x7fab44696028 out of bounds>, len=<optimized out>, 
    is_write=<optimized out>) at /usr/src/debug/qemu-1.5.3/exec.c:2069
#13 0x00007fab44a0cc65 in kvm_cpu_exec (env=env@entry=0x7fab472f5160) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1680
#14 0x00007fab449c1365 in qemu_kvm_cpu_thread_fn (arg=0x7fab472f5160) at /usr/src/debug/qemu-1.5.3/cpus.c:793
#15 0x00007fab4281cdf3 in start_thread (arg=0x7fab371be700) at pthread_create.c:308
#16 0x00007fab3f5283cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113


Expected results:
hotplug/unhotplug is ok, and no core dump

Additional info:
(gdb) bt full
#0  0x00007fab4498f3da in object_class_dynamic_cast_assert (class=0x7fab489b47f0, typename=typename@entry=0x7fab44af1949 "usb-device", 
    file=file@entry=0x7fab44af1929 "hw/usb/bus.c", line=line@entry=197, func=func@entry=0x7fab44af1e10 <__func__.27231> "usb_device_ep_stopped") at qom/object.c:528
        ret = <optimized out>
#1  0x00007fab4492ecc3 in usb_device_ep_stopped (dev=0x7fab488a9820, ep=0x7fab488aa950) at hw/usb/bus.c:197
        klass = <optimized out>
        __func__ = "usb_device_ep_stopped"
#2  0x00007fab4493e788 in xhci_ep_nuke_xfers (xhci=xhci@entry=0x7faa1fd38010, slotid=slotid@entry=2, epid=<optimized out>) at hw/usb/hcd-xhci.c:1336
        slot = 0x7faa1fd3aaa0
        epctx = 0x7fab2002bc70
        i = <optimized out>
        xferi = 0
        killed = 0
        ep = 0x7fab488aa950
        __PRETTY_FUNCTION__ = "xhci_ep_nuke_xfers"
#3  0x00007fab44942479 in xhci_stop_ep (epid=<optimized out>, slotid=2, xhci=0x7faa1fd38010) at hw/usb/hcd-xhci.c:1396
        slot = 0x7faa1fd3aaa0
        epctx = <optimized out>
#4  xhci_process_commands (xhci=0x7faa1fd38010) at hw/usb/hcd-xhci.c:2571
        epid = <optimized out>
        trb = {parameter = 0, status = 0, control = 33766401, addr = 5219795872, ccs = true}
        type = <optimized out>
        event = {type = ER_COMMAND_COMPLETE, ccode = CC_SUCCESS, ptr = 5219795872, length = 0, flags = 0, slotid = 0 '\000', epid = 0 '\000'}
        addr = 5219795872
        i = <optimized out>
        slotid = 2
#5  0x00007fab44a0dc83 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fab371bda40, size=size@entry=4, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7fab44a0e1a0 <memory_region_write_accessor>, opaque=opaque@entry=0x7faa1fd38a30)
    at /usr/src/debug/qemu-1.5.3/memory.c:365
        access_mask = 4294967295
        access_size = 4
        i = <optimized out>
#6  0x00007fab44a121cb in memory_region_dispatch_write (size=4, data=0, addr=0, mr=0x7faa1fd38a30) at /usr/src/debug/qemu-1.5.3/memory.c:917
No locals.
#7  io_mem_write (mr=0x7faa1fd38a30, addr=0, val=<optimized out>, size=4) at /usr/src/debug/qemu-1.5.3/memory.c:1592
No locals.
#8  0x00007fab44a0dc83 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fab371bdae0, size=size@entry=4, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7fab44a0e1a0 <memory_region_write_accessor>, opaque=opaque@entry=0x7fab2c007340)
    at /usr/src/debug/qemu-1.5.3/memory.c:365
        access_mask = 4294967295
        access_size = 4
        i = <optimized out>
#9  0x00007fab44a121cb in memory_region_dispatch_write (size=4, data=0, addr=0, mr=0x7fab2c007340) at /usr/src/debug/qemu-1.5.3/memory.c:917
No locals.
#10 io_mem_write (mr=0x7fab2c007340, addr=0, val=<optimized out>, size=size@entry=4) at /usr/src/debug/qemu-1.5.3/memory.c:1592
No locals.
#11 0x00007fab449c978d in address_space_rw (as=as@entry=0x7fab456e2740 <address_space_memory>, addr=4273807360, 
---Type <return> to continue, or q <return> to quit---
    buf=buf@entry=0x7fab44696028 <Address 0x7fab44696028 out of bounds>, len=4, is_write=true) at /usr/src/debug/qemu-1.5.3/exec.c:1987
        addr1 = <optimized out>
        d = 0x7fab47154a80
        l = 4
        ptr = <optimized out>
        val = <optimized out>
        page = 4273807360
        section = <optimized out>
#12 0x00007fab449c97fd in cpu_physical_memory_rw (addr=<optimized out>, buf=buf@entry=0x7fab44696028 <Address 0x7fab44696028 out of bounds>, len=<optimized out>, 
    is_write=<optimized out>) at /usr/src/debug/qemu-1.5.3/exec.c:2069
No locals.
#13 0x00007fab44a0cc65 in kvm_cpu_exec (env=env@entry=0x7fab472f5160) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1680
        cpu = 0x7fab472f5050
        __func__ = "kvm_cpu_exec"
        run = 0x7fab44696000
        ret = <optimized out>
        run_ret = 0
#14 0x00007fab449c1365 in qemu_kvm_cpu_thread_fn (arg=0x7fab472f5160) at /usr/src/debug/qemu-1.5.3/cpus.c:793
        env = 0x7fab472f5160
        cpu = 0x7fab472f5050
        __func__ = "qemu_kvm_cpu_thread_fn"
        r = <optimized out>
#15 0x00007fab4281cdf3 in start_thread (arg=0x7fab371be700) at pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fab371be700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140373340710656, 2616551943445414848, 0, 140373340711360, 140373340710656, 140373610418528, 
                -2658898294789635136, -2658996099232693312}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, 
              canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#16 0x00007fab3f5283cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Comment 5 Gerd Hoffmann 2014-05-12 12:51:15 UTC
http://patchwork.ozlabs.org/patch/348006/

Comment 6 Gerd Hoffmann 2014-07-03 06:53:36 UTC
upstream commit 463c534db516701ac732ac606c33c7101cf22e56

Comment 7 Gerd Hoffmann 2014-07-04 11:31:36 UTC
Please retest with this test build:
http://people.redhat.com/ghoffman/bz1103193/

Comment 8 ShupingCui 2014-07-07 02:25:33 UTC
(In reply to Gerd Hoffmann from comment #7)
> Please retest with this test build:
> http://people.redhat.com/ghoffman/bz1103193/

Retest with this build, case passed and not core dump found.

# uname -r
3.10.0-123.el7.x86_64
# rpm -qa | grep qemu-kvm
qemu-kvm-common-1.5.3-65.el7.bz1103193.2.x86_64
qemu-kvm-1.5.3-65.el7.bz1103193.2.x86_64
qemu-kvm-tools-1.5.3-65.el7.bz1103193.2.x86_64

Thanks,
Shuping

Comment 9 Miroslav Rezanina 2014-10-21 14:52:38 UTC
Fix included in qemu-kvm-1.5.3-76.el7

Comment 10 Miroslav Rezanina 2014-10-21 14:52:48 UTC
Fix included in qemu-kvm-1.5.3-76.el7

Comment 11 Miroslav Rezanina 2014-10-21 14:53:28 UTC
Fix included in qemu-kvm-1.5.3-76.el7

Comment 13 Qian Guo 2014-10-30 07:40:48 UTC
Reproduce this bug with qemu-kvm-1.5.3-75.el7.x86_64

Steps:
1.Boot guest:
/usr/libexec/qemu-kvm -cpu SandyBridge -enable-kvm -m 4G -smp 4,sockets=1,cores=4,threads=1 -name test -rtc base=localtime,clock=host,driftfix=slew  -k en-us  -boot menu=on -spice disable-ticketing,port=5901 -vga qxl -usb -device usb-tablet -monitor stdio -drive file=/home/rhel7u1/rhel7u1cp1.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk,addr=0x3 -qmp unix:/tmp/q1,server,nowait  -netdev tap,id=hostnet0,vhost=on,queues=4,script=/etc/qemu-ifup -device virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=vnet0,mac=52:54:00:1a:2b:02 -serial unix:/tmp/s2,server,nowait     -device ich9-usb-uhci1,id=usb1  -device nec-usb-xhci,id=usbtest -device usb-hub,id=usb-hub1,bus=usbtest.0,port=1 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1

2.Hotplug and hotunplug usb3.0 device repeatedly:

Result, when repeated the 5th time, qemu crashed:
(gdb) bt
#0  type_is_ancestor (type=0xbfd66348f685e063, target_type=0x5555564ee5f0) at qom/object.c:189
#1  0x0000555555716c8b in object_class_dynamic_cast (class=class@entry=0x7ffff300cf28 <main_arena+1992>, typename=typename@entry=0x55555587af49 "usb-device") at qom/object.c:514
#2  0x0000555555716da5 in object_class_dynamic_cast_assert (class=0x7ffff300cf28 <main_arena+1992>, typename=typename@entry=0x55555587af49 "usb-device", file=file@entry=0x55555587af29 "hw/usb/bus.c", 
    line=line@entry=203, func=func@entry=0x55555587b410 <__func__.27251> "usb_device_ep_stopped") at qom/object.c:546
#3  0x00005555556b63a3 in usb_device_ep_stopped (dev=0x555557382000, ep=0x555557383130) at hw/usb/bus.c:203
#4  0x00005555556c5e98 in xhci_ep_nuke_xfers (xhci=xhci@entry=0x7fffda338010, slotid=slotid@entry=2, epid=<optimized out>) at hw/usb/hcd-xhci.c:1338
#5  0x00005555556c9b89 in xhci_stop_ep (epid=<optimized out>, slotid=2, xhci=0x7fffda338010) at hw/usb/hcd-xhci.c:1398
#6  xhci_process_commands (xhci=0x7fffda338010) at hw/usb/hcd-xhci.c:2573
#7  0x0000555555795ea3 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fffe8823a80, size=size@entry=4, access_size_min=<optimized out>, access_size_max=<optimized out>, 
    access=access@entry=0x5555557963c0 <memory_region_write_accessor>, opaque=opaque@entry=0x7fffda338a30) at /usr/src/debug/qemu-1.5.3/memory.c:365
#8  0x000055555579a3eb in memory_region_dispatch_write (size=4, data=0, addr=0, mr=0x7fffda338a30) at /usr/src/debug/qemu-1.5.3/memory.c:917
#9  io_mem_write (mr=0x7fffda338a30, addr=0, val=<optimized out>, size=4) at /usr/src/debug/qemu-1.5.3/memory.c:1597
#10 0x0000555555795ea3 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fffe8823b20, size=size@entry=4, access_size_min=<optimized out>, access_size_max=<optimized out>, 
    access=access@entry=0x5555557963c0 <memory_region_write_accessor>, opaque=opaque@entry=0x7fffdc002010) at /usr/src/debug/qemu-1.5.3/memory.c:365
#11 0x000055555579a3eb in memory_region_dispatch_write (size=4, data=0, addr=0, mr=0x7fffdc002010) at /usr/src/debug/qemu-1.5.3/memory.c:917
#12 io_mem_write (mr=0x7fffdc002010, addr=0, val=<optimized out>, size=size@entry=4) at /usr/src/debug/qemu-1.5.3/memory.c:1597
#13 0x00005555557513ed in address_space_rw (as=as@entry=0x55555646dbe0 <address_space_memory>, addr=4228194304, buf=buf@entry=0x7ffff7feb028 "", len=4, is_write=true)
    at /usr/src/debug/qemu-1.5.3/exec.c:1987
#14 0x000055555575145d in cpu_physical_memory_rw (addr=<optimized out>, buf=buf@entry=0x7ffff7feb028 "", len=<optimized out>, is_write=<optimized out>) at /usr/src/debug/qemu-1.5.3/exec.c:2069
#15 0x0000555555794e85 in kvm_cpu_exec (env=env@entry=0x555556710b40) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1678
#16 0x0000555555748fc5 in qemu_kvm_cpu_thread_fn (arg=0x555556710b40) at /usr/src/debug/qemu-1.5.3/cpus.c:793
#17 0x00007ffff5604df3 in start_thread () from /lib64/libpthread.so.0
#18 0x00007ffff2d4805d in clone () from /lib64/libc.so.6


So this bug is reproduced:

Verify this bug with qemu-kvm-1.5.3-77.el7.x86_64 

Steps as above

Result, hotplug and unplug for long time(more than 100 times), no crash and guest works well, from dmesg, only the plug and unplug infos:
...
[  144.372861] usb 3-1.3: USB disconnect, device number 23
[  144.670111] usb 3-1.4: new full-speed USB device number 24 using xhci_hcd
[  144.743809] usb 3-1.4: New USB device found, idVendor=0627, idProduct=0001
[  144.743813] usb 3-1.4: New USB device strings: Mfr=1, Product=4, SerialNumber=5
[  144.743815] usb 3-1.4: Product: QEMU USB Keyboard
[  144.743817] usb 3-1.4: Manufacturer: QEMU
[  144.743819] usb 3-1.4: SerialNumber: 42
[  144.744027] usb 3-1.4: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[  144.745053] input: QEMU QEMU USB Keyboard as /devices/pci0000:00/0000:00:06.0/usb3/3-1/3-1.4/3-1.4:1.0/input/input27
[  144.745316] hid-generic 0003:0627:0001.0018: input,hidraw2: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:06.0-1.4/input0
...

So this bug is fixed by qemu-kvm-1.5.3-77.el7.x86_64

Verify with qemu-kvm-rhev-2.1.2-5.el7.x86_64

Same steps and results as qemu-kvm-1.5.3-77.el7.x86_64, so this bug is fixed by both qemu-kvm and qemu-kvm-rhev latest builds.

Comment 15 errata-xmlrpc 2015-03-05 08:05:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0349.html