Bug 722548 (rhq-gui-timeouts)

my 2c, 
this is not an issue of making the user wait longer
the issue is the length of time the work is taking

either
a) do less work, e.g. return fewer rows, or do the work async
b) do the work faster, e.g. optimize the query

In general the solution is not to have the user sit there and wait minutes for the result.

Obviously Monitor>Graphs tab for large groups is not amenable to a) and has historically been optimized so there may not be much more than can be squeezed out of it given the current architecture. Don't these graphs still live in iframes? Can't that be used to overcome the connection problem. The desire here specifically would not to regress over prior releases.

After discussing with some other devs, the consensus here is to try to address this as follows:

1) make the default GWT RPC timeout a configurable Server setting; make sure to grab this setting from the Server right at GUI startup, so the GUI starts using it right away; this will be very valuable in the perf env, where we can set it very high so nothing times out, allowing us to see exactly how long various calls are taking
2) also add an optional query string parameter (e.g. rpcTimeout=100) that can be added to any URL to tell the GUI to override the default timeout; this will be valuable, because it will allow a user who is hitting a timeout for a particular view to increase the timeout enough to allow the view to load
3) identify the calls that are taking more than 10s and investigate whether their performance can be tuned to a more acceptable level - use Oracle EM, hibernate perf monitor, code analysis, etc.
4) for certain RPC calls (e.g. group metric queries) increase the timeout for those particular calls to a value higher than 10s; we should make sure that the query string parameter described by 2) takes precedence over per-call timeouts, so user still has the ability to override them if they're not high enough
5) recommend in our docs that users increase their max simultaneous http connections browser setting to some value that allows for more concurrency in the GUI; note, we can not *require* this, since it's a global browser setting and users' corporate policies may not allow changing such settings; we also need to be careful not to recommend too high of a value, since more simultaneous connections from multiple clients will also put more stress on the Server and could possibly hit simultaneous connection limits configured in the RHQ Server's JBoss Web Server.

We could also consider adding a user preference in the future, but I think 1) and 2) will should be sufficient.

Hopefully, the above will be enough to get the situation under control, and we will not have to undertake the major refactoring of making all RPC fully async (i.e. where the RPC calls immediately return void and then the Server pushes the responses to the client via Errai or some other web push library once the server-side SLSB method returns).

(In reply to comment #1)
> my 2c, 
> this is not an issue of making the user wait longer
> the issue is the length of time the work is taking
> 
> either
> a) do less work, e.g. return fewer rows, or do the work async

all of these queries *should* already be paged. however, i already discovered one, the autodiscovery list view, that is not being paged (and hence took 65s to load w/ 1500 NEW Resources). so we should definitely check all slow list queries (well, all list queries really) and make sure they are actually doing paging.

async is last resort since it will require adding something like Errai, which will be a pretty big effort.

> b) do the work faster, e.g. optimize the query
> 
> In general the solution is not to have the user sit there and wait minutes for
> the result.

agreed. however, if they have a group w/ 100k resources, they might have to accept that certain views for that group will take more than 10s to load, especially if we've already done all we can to optimize the underlying queries.

Currently if an RPC call is in progress and the user decides navigate to some other view, that call will continue to consume one of the precious http connections until it completes or times out, even though the user has abandoned the view that initiated the call. So we should consider adding something that would cancel any *user-initiated* calls that are still in progress when the user navigates to another view.

(In reply to comment #3)
> 
> all of these queries *should* already be paged. however, i already discovered
> one, the autodiscovery list view, that is not being paged (and hence took 65s
> to load w/ 1500 NEW Resources). so we should definitely check all slow list
> queries (well, all list queries really) and make sure they are actually doing
> paging.
> 

Is this fixed, is there another BZ for this?

Answering my own question I see
https://bugzilla.redhat.com/show_bug.cgi?id=720791
was raised to cover the autodiscovery queue issue

Merged rpcTimeout parameter to master in:

4b7240fe1fbd0a169a4ad0663ecc3602f100dcaa
3268a40274ada82656d2b5ea095280c5985efde6

So far we've only addressed 2) from Comment 2 above. 3) and 4) also need to be addressed before this issue should be moved to ON_QA.

[master f4fdec7] includes the following:

a) add new #Test/Rpc view that can be used to invoke a new sleep() RPC method that sleeps for a specified number of seconds
b) increase default RPC timeout from 10s to 30s
c) use custom timeouts for autodiscovery queue, config history list, group schedules list views, which may exceed even the new default timeout

a) allows us to test how the app behaves when the browser's max http connections setting has been exceeded,
b) gives us a little more breathing room in the default timeout to handle calls that take more than 10s but not ridiculously long,
and c) are temporary measures for specific calls that are known to take unacceptably long; the longer term plan is to optimize the performance of these calls so they take no more than 10s

removing superfluous blocks bz's

All dependent bugs completed.