Bug 972957

Summary: Admin Portal error "Unable to evaluate payload" in Internet Explorer
Product: Red Hat Enterprise Virtualization Manager Reporter: Bryan Yount <byount>
Component: ovirt-engine-webadmin-portalAssignee: Einav Cohen <ecohen>
Status: CLOSED DEFERRED QA Contact: Jiri Belka <jbelka>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.3CC: acathrow, byount, ecohen, iheim, jbelka, jkt, mketchio, pstehlik, Rhev-m-bugs, ssekidde, vszocs, ykaul
Target Milestone: ---Keywords: Reopened
Target Release: 3.3.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: ux
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1053395 (view as bug list) Environment:
Last Closed: 2013-12-12 19:33:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Unable to evaluate payload error none

Description Bryan Yount 2013-06-10 22:21:01 UTC
Created attachment 759385 [details]
Unable to evaluate payload error

Description of problem:
Intermittently, when using Internet Explorer 9 on the Admin Portal, a user will see the error message "Error: A request to the Server failed: Unable to evaluate payload" pop up. The only way to dismiss it is to completely reload the Admin Portal or close the browser and reopen.

Version-Release number of selected component (if applicable):
3.1.0-50 (3.1.3)

How reproducible:
Intermittently

Steps to Reproduce:
Unsure of steps to reproduce. Seems to just pop up when the page is sitting there.


Additional info:
See attached screenshot

Comment 7 Andrew Cathrow 2013-07-08 16:04:40 UTC
Closing since we can't reproduce.
Please re-open if we can on 3.2

Comment 11 Vojtech Szocs 2013-11-27 11:41:21 UTC
Hi, this is my analysis of the problem.

The "Unable to evaluate payload" error comes from CommandClientSerializationStreamReader#prepareToRead method. CommandClientSerializationStreamReader is a GWT client-side class responsible for processing (reading) GWT RPC response payload received from server. In GWT 2.5.1 the code is following:

 public void prepareToRead(String js) throws RemoteException {
     try {
         payload = eval("(function(){var " + BACKREF_IDENT + "={};" + js + "})()");
         assert payload != null : "Payload evaluated to null";
     } catch (JavaScriptException e) {
         throw new IncompatibleRemoteServiceException(
             "Unable to evaluate payload", e);
     } catch (Throwable e) {
         throw new RemoteException("Unable to evaluate payload", e);
     }
 }

The code above attempts to eval() [1] the string representing GWT RPC response payload.

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval

The "Unable to evaluate payload" error therefore occurs if eval() invocation fails. There many potential root causes:
* server's GWT RPC response payload processing is buggy and sends bad data to client [a], -OR-
* data (objects) embedded within the payload have buggy serialization policy (i.e. bad custom serializer), -OR-
* IE9's eval() implementation is buggy (i.e. some data might cause it to fail)

[a] we're using GWT direct-eval RPC (aka deRPC) implementation which is considered "experimental", we're working on moving to REST API in long term

The "Large memory leak when using RPC with IE9" problem [2] mentioned by Simon & Einav isn't really related to "Unable to evaluate payload" error. In case of "Large memory leak" problem, the error seems to come from ClientSerializationStreamReader#prepareToRead [3] and is caused by buggy IE9 eval() implementation that causes memory leak [4].

[2] https://code.google.com/p/google-web-toolkit/issues/detail?id=5736
[3] https://code.google.com/p/google-web-toolkit/issues/detail?id=5736#c11 /"It is not in fact the CommandClientSerializationStreamReader::prepareToRead()'s eval() which is causing this leak but instead the super sourced ClientSerializationStreamReader::prepareToRead()'s eval()."/
[4] http://support.microsoft.com/kb/2572253

As mentioned in [3] the memory leak generated by IE9's eval() implementation is proportional to amount of GWT RPC payload data processed. Discussion & patch at [3] suggests replacing eval() with JSON.parse() via bumping up GWT RPC protocol version.

To summarize:
* The "Unable to evaluate payload" error boils down to IE9's eval() implementation failing on certain GWT RPC payload data.
* If someone CAN reliably reproduce this, it means the problem is on our side -> server-side GWT RPC response payload processing.
* If someone CAN NOT reliably reproduce this, it means IE9's eval() implementation is buggy and we can do nothing about that.
* Using WebAdmin + IE9 will cause memory leaks proportional to amount of GWT RPC payload data processed. In practice, 32-bit IE9 will crash when reaching ~2 GB memory limit [b], 64-bit IE9 will waste all memory.

[b] http://stackoverflow.com/questions/11891593/the-maximum-amount-of-memory-any-single-process-on-windows-can-address/11892191#11892191

(I know it sounds scary, especially the memory leak part. We can do nothing about buggy browsers such as IE. People shouldn't use buggy browsers unless they have no other alternative.)

I recommend upgrading to IE10 if possible. If stuck with IE9 please try to reliably reproduce this problem so we can investigate further.

Comment 15 Itamar Heim 2013-12-12 19:33:00 UTC
per comment 11 which provides all the technical backgorund - as this is a bug in IE9, solved in IE10 and we can't debug this without a reliable reproducer.
closing. please re-open if IE10 is not an option and there is a use case reproducing this leak in IE9.
thanks