Bug 1018633 - Connection leak in production system
Connection leak in production system
Status: CLOSED CURRENTRELEASE
Product: Zanata
Classification: Community
Component: Component-Logic (Show other bugs)
3.0
Unspecified Unspecified
urgent Severity urgent
: ---
: 3.1
Assigned To: Sean Flanigan
Zanata-QA Mailling List
:
: 1020585 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-13 21:46 EDT by Ding-Yi Chen
Modified: 2014-03-04 03:17 EST (History)
4 users (show)

See Also:
Fixed In Version: 3.1.1-SNAPSHOT (20131017-1642)
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 959115
Environment:
Last Closed: 2013-11-26 22:23:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
When the connections were created (according to JavaMelody) (157.25 KB, text/plain)
2013-10-22 19:43 EDT, Sean Flanigan
no flags Details

  None (edit)
Comment 2 Sean Flanigan 2013-10-13 22:00:42 EDT
Ding-Yi Chen 2013-10-14 11:46:02 EST:

Connection leak is still in production system.

Version-Release number of selected component (if applicable):
Zanata version 3.1.1-SNAPSHOT (20131011-1233)

How reproducible:
Requires several days of production load.

Steps to Reproduce:
1. Boot Zanata
2. Allow users to log in to the editor and generate load

Expected:
No connection leak.

Actual:
Connection leak still happening, which stops new connections.
Comment 3 Sean Flanigan 2013-10-13 22:06:00 EDT
Correction, the affected version is:

Zanata version 3.0.3 (20130919-2218)
Comment 4 Ding-Yi Chen 2013-10-13 22:32:00 EDT
Correction, the https://zanata.example.com/admin/monitoring?part=graph&graph=usedConnections&period=semaine still works.

Steps to get to the monitoring graph:
1. Log in with admin role.
2. Go to Administration menu.
3. Click server monitoring
4. Click the diagram "Used JDBC connection"
Comment 6 Sean Flanigan 2013-10-16 23:22:27 EDT
Simultaneously launching two instances of the same editor (same language) from a project with many documents seems to be fairly reliable at reproducing the exception, as long as the relevant documents/statistics haven't been cached already.  (You can use JavaMelody to clear all data caches.)

07:07:12,839 INFO  [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (http-/0.0.0.0:8080-7) IJ000311: Throwable from unregister connection: java.lang.IllegalStateException: Trying to return an unknown connection2! org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@b0da531

Every time that exception is logged, another connection is leaked (or already was), although the cause is probably a little earlier, possibly in a different thread.
Comment 7 Sean Flanigan 2013-10-16 23:26:43 EDT
Pull request: https://github.com/zanata/zanata-server/pull/229
Comment 8 Ding-Yi Chen 2013-10-17 03:53:35 EDT
VERIFIED Zanata version 3.1.1-SNAPSHOT (20131017-1642)
Comment 9 Ding-Yi Chen 2013-10-22 02:44:43 EDT
*** Bug 1020585 has been marked as a duplicate of this bug. ***
Comment 11 Sean Flanigan 2013-10-22 19:43:23 EDT
Created attachment 815210 [details]
When the connections were created (according to JavaMelody)
Comment 12 Sean Flanigan 2013-10-23 23:17:21 EDT
False alarm (I think).

It appears that the connections are being retained deliberately by the DataSource, not leaked, because there are no warning messages in the log such as "Trying to return an unknown connection2!".  Also, several of the connections were created at the same time, which may suggest that they were created to expand a connection pool (unless they *all* happened to leak at the same time).

Looking at the stack traces, I can't find any suspicous code (eg ResultSet handling) that would explain such a leak.  Unless we actually run out of connections (or get the above warning message again), I think we'll have to assume it's behaving normally.  I think it's just normal DataSource connection pool behaviour.

However, if we get the above log warning, or a server failure like "javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout" or "javax.resource.ResourceException: IJ000453: Unable to get managed connection for ..." we should create a new bug.
Comment 13 Sean Flanigan 2013-11-26 22:14:13 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 14 Sean Flanigan 2013-11-26 22:16:03 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 15 Sean Flanigan 2013-11-26 22:19:53 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 16 Sean Flanigan 2013-11-26 22:23:58 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 17 Sean Flanigan 2013-11-26 22:34:44 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 18 Sean Flanigan 2013-11-26 22:36:51 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.

Note You need to log in before you can comment on or make changes to this bug.