Bug 672191

Summary: spicevmc: flow control on the spice agent channel is missing in both directions
Product: Red Hat Enterprise Linux 6 Reporter: Pavel Stehlik <pstehlik>
Component: qemu-kvmAssignee: Alon Levy <alevy>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: high    
Version: 6.1CC: dblechte, ehabkost, hdegoede, juzhang, khong, lcapitulino, mkenneth, mkrcmari, szhou, tburke, virt-maint, yeylon
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.153.el6 Doc Type: Bug Fix
Doc Text:
Cause: flow control on the spice agent channel was missing in both directions Consequence: copy & paste of large amounts of text makes qemu-kvm hangs. Fix: added throttling support to spicevmc Consequence: copy & paste of large amounts of text works.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 11:23:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Stehlik 2011-01-24 11:00:51 UTC
Created attachment 474948 [details]
Video

Description of problem:
 If guest tools are installed & RHEVM Spice agent is running & VM is opened for the 1st time since start, then VM is killed. evt msg: "CM is down, Exit mesasge Lost connection withj kvm process."

See att logs &  video.

Version-Release number of selected component (if applicable):
ic79
kernel-2.6.32-71.14.1.el6.x86_64
lvm2-2.02.72-8.el6_0.4.x86_64
qemu-kvm-0.12.1.2-2.113.el6_0.6.x86_64
libvirt-0.8.7-2.el6.x86_64
vdsm-4.9-42.el6.x86_64
fence-agents-3.0.12-8.el6_0.2.x86_64
bridge-utils-1.2-9.el6.x86_64
udev-147-2.29.el6.x86_64
seabios-0.5.1-3.el6.x86_64
spice-server-0.4.2-15.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1. start VM with tools & wait since Spice agent runs (can be checked by rdp)
2. open spice console
3. 
  
Actual results:
VM is killed & there aren't much info in logs


Additional info:
attaching to qemu before crash shows abort in spicevmc
#2  0x000000362e62b935 in __assert_fail (
assertion=0x5adfdb "svc->datalen == 0", file=<value optimized out>, 
line=210, function=<value optimized out>) at assert.c:81
#3  0x0000000000474cbb in ?? ()

Comment 3 Shirley Zhou 2011-03-07 11:29:11 UTC
Reproduce this bug with packages as following steps:
ic99
qemu-kvm-0.12.1.2-2.113.el6_0.8.x86_64
spice-server-0.4.2-15.el6.x86_64

CLI:/usr/libexec/qemu-kvm -enable-kvm -m 1G -smp 4,sockets=1,cores=4,threads=1 -name bug -uuid 34541105-50b0-1234-921b-5c360a941234 -monitor stdio -rtc base=localtime -boot c -drive file=/mnt/winXp.s1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none,werror=stop,rerror=stop -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:03:12:21,bus=pci.0,addr=0x7 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -vga qxl -spice port=8001,disable-ticketing  -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -device spicevmc,nr=1,bus=virtio-serial0.0

After install RHEV-toolsSetup in guest, it will prompt need restart. After guest restart, quit guest. Then boot guest with above same cli, core dumped.

****
Breakpoint 1, spice_vmc_have_data (vserport=0x29b1e30, buf=0x7f16b80008c0 "\001", len=36) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/spice-vmc.c:210
210	    assert(svc->datalen == 0);
(gdb) p *(SpiceVMChannel*)svc
$14 = {vserport = {dev = {id = 0x0, state = DEV_STATE_INITIALIZED, opts = 0x289aed0, hotplugged = 0, info = 0x84e6c0, parent_bus = 0x29acfb0, num_gpio_out = 0, gpio_out = 0x0, num_gpio_in = 0, 
      gpio_in = 0x0, child_bus = {lh_first = 0x0}, num_child_bus = 0, sibling = {le_next = 0x0, le_prev = 0x29acfd0}}, info = 0x84e6c0, next = {tqe_next = 0x0, tqe_prev = 0x29b1898}, vser = 0x29b17f0, 
    ivq = 0x2edf660, ovq = 0x2edf6b0, name = 0x5adffd "com.redhat.spice.0", id = 1, is_console = 0 '\000', guest_connected = true, host_connected = true, throttled = false}, running = true, 
  active_interface = true, active_interface_vmstate = 0 '\000', interface = {base = {base_version = 1, type = 0x5adfb2 "vdi_port", id = 1, description = 0x5ae038 "spice virtual channel vdi port", 
      minor_version = 1, major_version = 1}, plug = 0x474a30 <spice_vmc_interface_plug>, unplug = 0x474a60 <spice_vmc_interface_unplug>, write = 0x474da0 <spice_vmc_interface_write>, 
    read = 0x474cc0 <spice_vmc_interface_read>}, plug = 0x291eb60, datapos = 0x7f16c4000b08 "\001", datalen = 28, debug = 0}
(gdb) n
207	{
(gdb) 
210	    assert(svc->datalen == 0);
(gdb) 
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/hw/spice-vmc.c:210: spice_vmc_have_data: Assertion `svc->datalen == 0' failed.

Program received signal SIGABRT, Aborted.
****

Comment 4 Alon Levy 2011-03-07 12:16:23 UTC
I have a fix for this, just haven't gotten around to making a scratchbuild for you to test with it yet, the patch is here:

 http://patchwork.freedesktop.org/patch/4375/

Also I want to try to fix it in the correct way (using flow control hooks in virtio-serial), but since this is a high priority bug this fix is good enough.

Alon

Comment 5 Alon Levy 2011-03-20 10:44:28 UTC
Hi,

 Can you please test http://brewweb.devel.redhat.com/brew/taskinfo?taskID=3188963 it should fix the problem.

Alon

Comment 9 Hans de Goede 2011-03-22 15:52:21 UTC
Further testing has revealed that since the spicevmc device -> chardev conversion there is no flow control in the other (client -> guest) direction either. I've a nice and simple patch fixing this. I'll post version 3 of the agent channel / spicevmc chardev flowcontrol patchset including this fix to rhvirt-patches right away.

Comment 10 Hans de Goede 2011-03-22 17:25:40 UTC
*** Bug 689562 has been marked as a duplicate of this bug. ***

Comment 18 juzhang 2011-04-10 07:36:19 UTC
Reproduced using qemu-kvm directly with qemu-kvm-tools-0.12.1.2-2.152.el6.x86_64.

steps:
1.create empty file and write 933 KB plain text
2.boot guest 
#/usr/libexec/qemu-kvm -m 2G -smp 4 -drive file=/root/zhangjunyi/rhel6.1-64.qcow2,if=none,id=test,cache=none,format=qcow2,werror=stop,rerror=stop -device virtio-blk-pci,drive=test -cpu qemu64,+sse2,+x2apic -boot c -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:94 -spice port=5930,disable-ticketing -vga qxl -serial stdio -qmp tcp:0:4444,server,nowait -usb -device virtio-serial-pci,id=virtio-serial1 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial1.0,nr=3,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -soundhw ac97
3.Copy the 933 KB text into clipboard in client
4.Paste the 933 KB text into empty file in Guest

Results
copy failed,Nothing is copied,2 mins later,agent got stuck.guest hang,can not connect via qmp.

Additional infos:
copy form guest to client,works well.

Verified using qemu-kvm directly with qemu-kvm-tools-0.12.1.2-2.156.el6.x86_64.

Results:
copy form guest to client or from client to guest,both work well

Hi,Hans/Alon

   Would you please tell me do we support drag copy file between guest and client?I means copy via "click file->drag"rather than "open file->copy contents->open file-> paste file"?I tired,failed.

Comment 19 Hans de Goede 2011-04-10 07:54:02 UTC
(In reply to comment #18)
> Reproduced using qemu-kvm directly with
> qemu-kvm-tools-0.12.1.2-2.152.el6.x86_64.
> 
> steps:
> 1.create empty file and write 933 KB plain text
> 2.boot guest 
> #/usr/libexec/qemu-kvm -m 2G -smp 4 -drive
> file=/root/zhangjunyi/rhel6.1-64.qcow2,if=none,id=test,cache=none,format=qcow2,werror=stop,rerror=stop
> -device virtio-blk-pci,drive=test -cpu qemu64,+sse2,+x2apic -boot c -netdev
> tap,id=hostnet0,vhost=on -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:94 -spice
> port=5930,disable-ticketing -vga qxl -serial stdio -qmp
> tcp:0:4444,server,nowait -usb -device virtio-serial-pci,id=virtio-serial1
> -chardev spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial1.0,nr=3,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
> -soundhw ac97
> 3.Copy the 933 KB text into clipboard in client
> 4.Paste the 933 KB text into empty file in Guest
> 
> Results
> copy failed,Nothing is copied,2 mins later,agent got stuck.guest hang,can not
> connect via qmp.
> 

As expected with -152.

> Additional infos:
> copy form guest to client,works well.
> 

This should fail as well with -152, changes are you need something bigger, confirm the guest to run as 1920x1200 and then press printscreen, copy to clipboard, followed by a paste in gimp in the client that did the trick for me.

> Verified using qemu-kvm directly with qemu-kvm-tools-0.12.1.2-2.156.el6.x86_64.
> 
> Results:
> copy form guest to client or from client to guest,both work well
> 

Good :)

> Hi,Hans/Alon
> 
>    Would you please tell me do we support drag copy file between guest and
> client?I means copy via "click file->drag"rather than "open file->copy
> contents->open file-> paste file"?I tired,failed.

No we don't support that. I've thought about it in the past, and it would be something interesting to support. But atm we don't support it.

Comment 20 juzhang 2011-04-10 08:44:22 UTC
> 
> This should fail as well with -152, changes are you need something bigger,
> confirm the guest to run as 1920x1200 and then press printscreen, copy to
> clipboard, followed by a paste in gimp in the client that did the trick for me.
> 

Thanks for your tips,hit too.
I also tried more bigger plant(10M) text. qemu-kvm quit with the following messages.
qemu-kvm: spice-qemu-char.c:108: spice_chr_write: Assertion `s->datalen == 0' failed.

And retested -156,works fine.


>No we don't support that. I've thought about it in the past, and it would be
>something interesting to support. But atm we don't support it.
ok,got it.Do we have RFE bug about it?thanks for your confirmation

Comment 21 juzhang 2011-04-10 08:55:15 UTC
According to comment18,comment19 and comment20,set this issue status as verified.

Comment 22 Hans de Goede 2011-04-10 09:03:33 UTC
(In reply to comment #20)

<snip>

> >No we don't support that. I've thought about it in the past, and it would be
> >something interesting to support. But atm we don't support it.
> ok,got it.Do we have RFE bug about it?

No, we don't have an RFE bug for this.

Comment 23 juzhang 2011-04-10 09:40:37 UTC
(In reply to comment #22)
> (In reply to comment #20)
> 
> <snip>
> 
> > >No we don't support that. I've thought about it in the past, and it would be
> > >something interesting to support. But atm we don't support it.
> > ok,got it.Do we have RFE bug about it?
> 
> No, we don't have an RFE bug for this.
I have filed a RFE Bug 695060 named RFE: file copy paste support which is proposed to rhel6.2 against spice-vdagent.I cced hans and alon.

Comment 24 Eduardo Habkost 2011-05-05 19:02:22 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: flow control on the spice agent channel was missing in both directions

Consequence: copy & paste of large amounts of text makes qemu-kvm hangs.

Fix: added throttling support to spicevmc

Consequence: copy & paste of large amounts of text works.

Comment 25 errata-xmlrpc 2011-05-19 11:23:20 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html

Comment 26 errata-xmlrpc 2011-05-19 13:01:17 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html