Bug 846427 - unstable vdagent (and mouse loss) with copy-pastes larger than 1MB from the guest to the client
unstable vdagent (and mouse loss) with copy-pastes larger than 1MB from the ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: spice-vdagent-win (Show other bugs)
3.0.0
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Arnon Gilboa
Desktop QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-07 14:42 EDT by Yonit Halperin
Modified: 2013-04-10 11:44 EDT (History)
10 users (show)

See Also:
Fixed In Version: vdagent-win-0.1-12
Doc Type: Bug Fix
Doc Text:
When attempting to copy and past large amounts of text from the client to the guest, the spice-agent restarted too many times and finally stopped. This caused the agent to crash. Updates to both the vdagent and virtio-serial driver have imposed a limit on the amount of data able to be copied and pasted, meaning the text will not copy but neither will it crash the agent.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 13:19:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdagent log for outcome 1 (71.62 KB, text/x-log)
2012-08-07 14:42 EDT, Yonit Halperin
no flags Details
vdagent: add message_queue for messages written to pipe (13.27 KB, patch)
2012-09-16 07:07 EDT, Arnon Gilboa
no flags Details | Diff

  None (edit)
Description Yonit Halperin 2012-08-07 14:42:08 EDT
Created attachment 602828 [details]
vdagent log for outcome 1

Description of problem:
copy-paste of data > 1MB from the guest to the client

2 outcomes may occur (in addition to the expected one):
(1) agent is killed by the guest, mouse is lost.
vdagent.log (with messages I added to the code for debugging) is attached
(2) corrupted msgs reach the server, and it detaches itself from the vdagent and switches to server mouse mode:
a message of "dispatch_vdi_port_data: invalid port" appears on the server, and
I added special code to verify that the message that precede the "invalid port" message is also corrupted. 

Cause of the bug: the agent code doesn't maintain a queue of messages directed to the server/client. Instead, it holds a buffer of size 1MB. If other message than clipboard don't have enough space in the buffer, it is lost, 
VDAgent::dispatch_message fails, and the agent is killed.
For clipboard msg, the clipboard data is stored till it is fully sent.

Detailed cause of outcome (1) : the write buffer was full with clipboard data;  Then there was an attempt to write another message to the buffer, which failed.
In the case of the log attached, the message was a response to another VD_AGENT_CLIPBOARD_REQUEST - it looks like spice-gtk is sensative to key repeats, and my ctrl-v was sent twice.

Detailed cause of outcome (2): interleaving msgs
In the middle of writing a large clipboard data to the buffer (in several chunks), another message was successfully written to the buffer ==> data corruption.


Version-Release number of selected component (if applicable):


How reproducible:

I was able to always reproduce with copying pictures of size > 6MB
guest: Windows7-32 bit
client: remote-viewer (linux)

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Arnon Gilboa 2012-08-08 03:03:16 EDT
Thanks Yonit. I really appreciate the time you spent to locate this buggy code. It's strange the bug was a bit hidden when using the old spicec / vioserial. I'll add a protected msg queue as mentioned.

(In reply to comment #0)
> Created attachment 602828 [details]
> vdagent log for outcome 1
> 
> Description of problem:
> copy-paste of data > 1MB from the guest to the client
> 
> 2 outcomes may occur (in addition to the expected one):
> (1) agent is killed by the guest, mouse is lost.
> vdagent.log (with messages I added to the code for debugging) is attached
> (2) corrupted msgs reach the server, and it detaches itself from the vdagent
> and switches to server mouse mode:
> a message of "dispatch_vdi_port_data: invalid port" appears on the server,
> and
> I added special code to verify that the message that precede the "invalid
> port" message is also corrupted. 
> 
> Cause of the bug: the agent code doesn't maintain a queue of messages
> directed to the server/client. Instead, it holds a buffer of size 1MB. If
> other message than clipboard don't have enough space in the buffer, it is
> lost, 
> VDAgent::dispatch_message fails, and the agent is killed.
> For clipboard msg, the clipboard data is stored till it is fully sent.
> 
> Detailed cause of outcome (1) : the write buffer was full with clipboard
> data;  Then there was an attempt to write another message to the buffer,
> which failed.
> In the case of the log attached, the message was a response to another
> VD_AGENT_CLIPBOARD_REQUEST - it looks like spice-gtk is sensative to key
> repeats, and my ctrl-v was sent twice.
> 
> Detailed cause of outcome (2): interleaving msgs
> In the middle of writing a large clipboard data to the buffer (in several
> chunks), another message was successfully written to the buffer ==> data
> corruption.
> 
> 
> How reproducible:
> 
> I was able to always reproduce with copying pictures of size > 6MB
> guest: Windows7-32 bit
> client: remote-viewer (linux)
Comment 3 Arnon Gilboa 2012-09-16 07:07:41 EDT
Created attachment 613397 [details]
vdagent: add message_queue for messages written to pipe
Comment 4 Arnon Gilboa 2012-09-20 10:56:38 EDT
The attached patch is only part of the message corruption solution, which is still reproduced. It seems like the main suspect is virtio-serial windows driver, similar to bzs 858674 752338 858687.

Notes:
1. Corruption was unreproducible on Fedora 17 guest with virtio serial & spice agent.

2. Spice-server-only test (no qemu at all) generating stream of messages could not reproduce the corruption.
http://cgit.freedesktop.org/spice/spice/tree/server/tests/test_vdagent.c

3. Debug printfs added in qemu/spice-qemu-char.c spice_chr_write() shows a corrupted msgs
dbg spice_chr_write: 2048 port 1 size 2040 proto 1 type 10 op 0 sz 2020
dbg spice_chr_write: 2048 port 1 size 2040 proto 1 type 10 op 0 sz 2020
dbg spice_chr_write: 2048 port 30828432 size 4294965888 proto 1 type 10 op 0 sz 2020
dbg spice_chr_write: 2048 port 1 size 2040 proto 1 type 10 op 0 sz 2020
(./x86_64-softmmu/qemu-system-x86_64:19068): Spice-Warning **: reds.c:841:vdi_port_read_buf_process: invalid port

To isolate the issue we have also written a minimal dummy_agent which does nothing but simply emulate spice guest vdagent without any real functionality. It continuously reads (async) all incoming messages (e.g. mouse moves from spice client) while writing (async) an infinite stream of chunks, each chunk containing a Spice agent message of an illegal type. For each received message, a "." is printed to stdout. For each [batch_count, default 1000] * 2KB chunks written "w%u" is printed to stdout and we Sleep for [interval_ms, default 10] ms, not to starve the reads. With dummy_agent, corruption is easily reproduced after few hundreds of MBs are written.

http://git.engineering.redhat.com/?p=users/agilboa/dummy_agent.git
Comment 5 Arnon Gilboa 2012-10-02 12:13:44 EDT
seems to be ok with vdagent-win-0.1-12 and newer virtio-serial driver:
http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/37/win/virtio-win-prewhql-0.1.zip
Comment 9 errata-xmlrpc 2012-12-04 13:19:10 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-1502.html

Note You need to log in before you can comment on or make changes to this bug.