Bug 986145 - Erratic behaviour during concurrent or cancelled TMX exports
Erratic behaviour during concurrent or cancelled TMX exports
Status: CLOSED CURRENTRELEASE
Product: Zanata
Classification: Community
Component: Performance (Show other bugs)
development
All Linux
high Severity high
: ---
: 3.0
Assigned To: Sean Flanigan
Zanata-QA Mailling List
:
Depends On:
Blocks: 960786
  Show dependency treegraph
 
Reported: 2013-07-19 00:43 EDT by Damian Jansen
Modified: 2013-11-26 22:35 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-26 22:24:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Thread dump from jboss (128.96 KB, text/plain)
2013-07-19 01:20 EDT, Damian Jansen
no flags Details

  None (edit)
Description Damian Jansen 2013-07-19 00:43:18 EDT
Description of problem:
If an admin starts five or more concurrent TMX exports, Zanata will stop responding, and the exports will not complete.

Version-Release number of selected component (if applicable):
Dev

How reproducible:
Easily, appears to be always

Steps to Reproduce:
1. Log in as an administrator
2. Press Projects button
3. Middle click the Export All to TMX button five times

Actual results:
- Zanata does not respond to the browser actions (connecting indefinitely)
- Download streams will not complete

Expected results:
Five downloaded TMX files with no system performance degradation.

Additional info:
Four seemed to work ok, if not a bit heavy on the system resources.
Comment 1 Damian Jansen 2013-07-19 00:54:36 EDT
Serious enough to schedule for release
Comment 2 Damian Jansen 2013-07-19 01:20:49 EDT
Created attachment 775626 [details]
Thread dump from jboss
Comment 3 Sean Flanigan 2013-07-21 21:57:38 EDT
Due to an Okapi bug / API flaw (https://code.google.com/p/okapi/issues/detail?id=352), write errors due to the cancelled download are not being detected.  After the socket is closed, the server continues trying to write data, and reading from the database, until it has written the entire thing.  This takes much longer than usual due to all the exceptions being thrown and caught.
Comment 4 Damian Jansen 2013-07-21 22:04:16 EDT
The major issue with this bug is the IOExceptions causing a long delay on finishing the process.

The need for "semaphoring" of downloads is another issue (https://bugzilla.redhat.com/show_bug.cgi?id=986741), which is somewhat compounded by this bug.
Comment 5 Sean Flanigan 2013-07-22 00:38:29 EDT
The reason that Zanata stopped responding is that the JDBC connection pool (default size 10) was exhausted.  This happened because the cancelled TMX downloads continued to use connections for quite a long time.

https://github.com/zanata/zanata-server/pull/61 should ensure that cancelled downloads are detected promptly.  A cancelled download should now finish just as quickly as if the download were not cancelled, hopefully a little faster.

However, due to http://bugs.mysql.com/bug.php?id=42929 the JDBC driver still has to consume the entire result set.  Now we could split up the queries so that the result set is smaller, but this would complicate the implementation quite a bit, especially with regards to transactions, and I believe it would really hurt the performance of TMX downloads.

The implemented speed improvement should be enough to reduce the likelihood of say 10 concurrent cancelled TMX exports happening, which would cause Zanata to become non-responsive.  A fix for bug 986741 should be able to prevent it entirely.
Comment 6 Damian Jansen 2013-07-22 20:59:23 EDT
Major improvement. Mysqld still tops out for about 15 seconds but zanata stops the process almost immediately.
Tested at 81583983d079566b5f6b0c18c7c028e4aef59c77
Comment 7 Sean Flanigan 2013-11-26 22:15:33 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 8 Sean Flanigan 2013-11-26 22:16:50 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 9 Sean Flanigan 2013-11-26 22:18:50 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 10 Sean Flanigan 2013-11-26 22:24:56 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 11 Sean Flanigan 2013-11-26 22:33:30 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 12 Sean Flanigan 2013-11-26 22:35:46 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.

Note You need to log in before you can comment on or make changes to this bug.