Bug 1088116

Summary: qemu crash when device_del usb-redir
Product: Red Hat Enterprise Linux 7 Reporter: Xiaoqing Wei <xwei>
Component: qemu-kvmAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: hdegoede, hhuang, juzhang, knoel, michen, rbalakri, shuang, sluo, virt-maint, xwei
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-1.5.3-76.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 08:06:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
xz none

Description Xiaoqing Wei 2014-04-16 04:58:32 UTC
Description of problem:
qemu crash when device_del usb-redir

Version-Release number of selected component (if applicable):
kernel-3.10.0-121.el7.x86_64
qemu-kvm-1.5.3-60.el7.x86_64
spice-server-0.12.4-5.el7.x86_64
# rpm -qa | grep -i usb
usb_modeswitch-1.2.7-5.el7.x86_64
libusbx-1.0.15-4.el7.x86_64
libusb-0.1.4-3.el7.x86_64
libgusb-0.1.6-3.el7.x86_64
usb_modeswitch-data-20130807-2.el7.noarch
libusbx-debuginfo-1.0.15-4.el7.x86_64
usbutils-007-4.el7.x86_64
libertas-usb8388-firmware-20140213-0.3.git4164c23.el7.noarch
usbredir-0.6-7.el7.x86_64
usbredir-debuginfo-0.6-7.el7.x86_64


How reproducible:
100%

Steps to Reproduce:

1. boot a vm w/ usb-redir
/root/staf-kvm-devel/autotest-devel/client/tests/virt/qemu/qemu -monitor stdio \
    -name 'virt-tests-vm1'  \
    -sandbox off  \
    -M pc  \
    -nodefaults  \
    -vga qxl  \
    -global qxl-vga.vram_size=33554432 \
    -device intel-hda,bus=pci.0,addr=03 \
    -device hda-duplex  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20140415-193920-0jJPjuvS,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140415-193920-0jJPjuvS,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20140415-193920-0jJPjuvS,path=/tmp/seabios-20140415-193920-0jJPjuvS,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20140415-193920-0jJPjuvS,iobase=0x402 \
    -device usb-ehci,id=usb1,bus=pci.0,addr=04 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.0-64-virtio.qcow2 \
    -device virtio-blk-pci,scsi=off,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=05 \
    -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=idI9p14E,mq=on,vectors=10,netdev=idEKrcbV,bus=pci.0,addr=06  \
    -netdev tap,id=idEKrcbV,vhost=on,vhostfds=26:27:28:29,fds=22:23:24:25 \
    22<>/dev/tap50169 23<>/dev/tap50169 24<>/dev/tap50169 25<>/dev/tap50169 \
    26<>/dev/vhost-net 27<>/dev/vhost-net 28<>/dev/vhost-net 29<>/dev/vhost-net \
    -m 2048  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,disable-ticketing,addr=0,seamless-migration=on,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off  \
    -no-kvm-pit-reinjection \
    -device virtio-balloon-pci,id=ballooning \
    -chardev spicevmc,id=charredir0,name=usbredir \
    -device usb-redir,chardev=charredir0,id=redir0  \
    -enable-kvm \

2. delete the usb-redir

qemu# device_del redir0
3.

Actual results:
qemu crash

Expected results:
qemu should not crash, and device delete success

Additional info:
(gdb) bt
#0  0x00007f3d3eb27cb0 in ?? ()
#1  0x00007f3d3c965216 in qemu_chr_add_handlers (s=s@entry=0x7f3d3e79fd20, fd_can_read=fd_can_read@entry=0x0, fd_read=fd_read@entry=0x0, fd_event=fd_event@entry=0x0, opaque=opaque@entry=0x0)
    at qemu-char.c:219
#2  0x00007f3d3c8c9a57 in release_chr (obj=<optimized out>, name=<optimized out>, opaque=<optimized out>) at hw/core/qdev-properties-system.c:141
#3  0x00007f3d3c972168 in object_property_del_all (obj=<optimized out>) at qom/object.c:343
#4  object_finalize (data=0x7f3d3ea9c340) at qom/object.c:397
#5  object_unref (obj=0x7f3d3ea9c340) at qom/object.c:696
#6  0x00007f3d3c8cd2a9 in qdev_simple_unplug_cb (dev=<optimized out>) at hw/core/qdev.c:261
#7  0x00007f3d3c8cd22b in qdev_unplug (dev=0x7f3d3ea9c340, errp=errp@entry=0x7fffca969bd0) at hw/core/qdev.c:219
#8  0x00007f3d3c961b72 in qmp_device_del (id=<optimized out>, errp=errp@entry=0x7fffca969bd0) at qdev-monitor.c:692
#9  0x00007f3d3c8b0fdb in hmp_device_del (mon=0x7f3d3e7fec70, qdict=<optimized out>) at hmp.c:1187
#10 0x00007f3d3c9fbe29 in handle_user_command (mon=mon@entry=0x7f3d3e7fec70, cmdline=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4008
#11 0x00007f3d3c9fc0f7 in monitor_command_cb (mon=0x7f3d3e7fec70, cmdline=<optimized out>, opaque=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4624
#12 0x00007f3d3c973e24 in readline_handle_byte (rs=0x7f3d3e9668d0, ch=<optimized out>) at readline.c:374
#13 0x00007f3d3c9fc084 in monitor_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-1.5.3/monitor.c:4610
#14 0x00007f3d3c964b6b in qemu_chr_be_write (len=<optimized out>, buf=0x7fffca969ce0 "\r", s=0x7f3d3e79eff0) at qemu-char.c:167
#15 fd_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f3d3e79eff0) at qemu-char.c:850
#16 0x00007f3d3bc88ac6 in g_main_dispatch (context=0x7f3d3e79ee00) at gmain.c:3058
#17 g_main_context_dispatch (context=context@entry=0x7f3d3e79ee00) at gmain.c:3634
#18 0x00007f3d3c937aaa in glib_pollfds_poll () at main-loop.c:187
#19 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
#20 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
#21 0x00007f3d3c85de50 in main_loop () at vl.c:1988
#22 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4360

Comment 1 Xiaoqing Wei 2014-04-16 05:05:53 UTC
Created attachment 886746 [details]
xz

Comment 4 Gerd Hoffmann 2014-04-17 08:20:53 UTC
Looked up in the source.  Looks like CharDriverState->chr_update_read_handler points into nowhere (line 219 calls that function pointer).  Hans, any clue?

Comment 5 Gerd Hoffmann 2014-04-17 09:56:18 UTC
see comment #4

Comment 6 Hans de Goede 2014-04-17 10:08:35 UTC
(In reply to Gerd Hoffmann from comment #4)
> Looked up in the source.  Looks like
> CharDriverState->chr_update_read_handler points into nowhere (line 219 calls
> that function pointer).  Hans, any clue?

No not really, spice-qemu-char.c does not set CharDriverState->chr_update_read_handler and it allocates the CharDriverState using gmalloc0 so it should be NULL. So either something is overriding memory, or something has freed the memory and it has been reused. Maybe qemu_chr_delete is called on it too early ?

Comment 7 Gerd Hoffmann 2014-04-17 11:28:28 UTC
Can you repeat the test with ElectricFence please?
Also this looks like autotest, which of the tests is it?

short instructions:

  yum install -y ElectricFence
  export EF_ALLOW_MALLOC_0=1
  export LD_PRELOAD=libefence.so.0.0
  qemu-kvm $args

Comment 8 juzhang 2014-04-18 01:24:44 UTC
Hi Xwei,

Can you have a look comment7 and update our testing result?

Best Regards,
Junyi

Comment 9 Xiaoqing Wei 2014-04-18 02:29:42 UTC
(In reply to Gerd Hoffmann from comment #7)
> Can you repeat the test with ElectricFence please?
> Also this looks like autotest, which of the tests is it?

just manually booting cmd as above, did one cmd in monitor


> 
> short instructions:
> 
>   yum install -y ElectricFence
>   export EF_ALLOW_MALLOC_0=1
>   export LD_PRELOAD=libefence.so.0.0
>   qemu-kvm $args

crash as well. gdb fail to explain why.

# gdb qemu-kvm /tmp/crash.qemu.31078/core 

  Electric Fence 2.2.2 Copyright (C) 1987-1999 Bruce Perens <bruce>
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-51.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 31078]
[New LWP 31085]
[New LWP 31086]
[New LWP 31087]
[New LWP 31088]
[New LWP 31089]
[New LWP 31090]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

ElectricFence Exiting: mprotect() failed: Cannot allocate memory

Comment 13 Gerd Hoffmann 2014-04-24 06:33:54 UTC
patches posted.

Comment 15 Miroslav Rezanina 2014-10-21 14:52:29 UTC
Fix included in qemu-kvm-1.5.3-76.el7

Comment 17 Xiaoqing Wei 2014-10-31 04:59:39 UTC
Boot VM with usb redir and steps as below:

-----------------------
QEMU 1.5.3 monitor - type 'help' for more information
(qemu) info usb
  Device 0.1, Port 1, Speed 480 Mb/s, Product QEMU USB Tablet
  Device 0.0, Port 2, Speed 1.5 Mb/s, Product USB Redirection Device
(qemu) device_del redir0
(qemu) info usb
  Device 0.1, Port 1, Speed 480 Mb/s, Product QEMU USB Tablet
(qemu) info usbhost 
  Bus 2, Addr 4, Port 1.6.1, Speed 1.5 Mb/s    Class 00: USB device 0557:2213, CS-1734A V4.2.418
(qemu) 
------------------------

==============
QEMU 2.1.2 monitor - type 'help' for more information
(qemu) info usb
  Device 0.1, Port 1, Speed 480 Mb/s, Product QEMU USB Tablet
  Device 0.0, Port 2, Speed 1.5 Mb/s, Product USB Redirection Device
(qemu)  device_del redir0
(qemu) info usb
  Device 0.1, Port 1, Speed 480 Mb/s, Product QEMU USB Tablet

==============

[root@dhcp-10-208 ~]# rpm -q qemu-kvm
qemu-kvm-1.5.3-77.el7.x86_64   or qemu-kvm-rhev-2.1.2-1.el7.x86_64
[root@dhcp-10-208 ~]# rpm -qa | grep -i usb
libusbx-1.0.15-4.el7.x86_64
usbredir-0.6-7.el7.x86_64
usb_modeswitch-data-20130807-2.el7.noarch
usbutils-007-4.el7.x86_64
libgusb-0.1.6-3.el7.x86_64
libusb-0.1.4-3.el7.x86_64
usb_modeswitch-1.2.7-5.el7.x86_64



Set to Verified.

Comment 19 errata-xmlrpc 2015-03-05 08:06:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0349.html