Bug 1018633 - Connection leak in production system
Summary: Connection leak in production system
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Zanata
Classification: Retired
Component: Component-Logic
Version: 3.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.1
Assignee: Sean Flanigan
QA Contact: Zanata-QA Mailling List
URL:
Whiteboard:
: 1020585 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-10-14 01:46 UTC by Ding-Yi Chen
Modified: 2014-03-04 08:17 UTC (History)
4 users (show)

Fixed In Version: 3.1.1-SNAPSHOT (20131017-1642)
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 959115
Environment:
Last Closed: 2013-11-27 03:23:58 UTC


Attachments (Terms of Use)
When the connections were created (according to JavaMelody) (157.25 KB, text/plain)
2013-10-22 23:43 UTC, Sean Flanigan
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 959115 0 urgent CLOSED Database connection pool leaks under load 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 988202 0 unspecified CLOSED RFE: REST API rate limiting 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1069428 0 unspecified CLOSED Various concurrency problems due to unsafe Seam injections 2021-02-22 00:41:40 UTC

Internal Links: 959115 988202 1069428

Comment 2 Sean Flanigan 2013-10-14 02:00:42 UTC
Ding-Yi Chen 2013-10-14 11:46:02 EST:

Connection leak is still in production system.

Version-Release number of selected component (if applicable):
Zanata version 3.1.1-SNAPSHOT (20131011-1233)

How reproducible:
Requires several days of production load.

Steps to Reproduce:
1. Boot Zanata
2. Allow users to log in to the editor and generate load

Expected:
No connection leak.

Actual:
Connection leak still happening, which stops new connections.

Comment 3 Sean Flanigan 2013-10-14 02:06:00 UTC
Correction, the affected version is:

Zanata version 3.0.3 (20130919-2218)

Comment 4 Ding-Yi Chen 2013-10-14 02:32:00 UTC
Correction, the https://zanata.example.com/admin/monitoring?part=graph&graph=usedConnections&period=semaine still works.

Steps to get to the monitoring graph:
1. Log in with admin role.
2. Go to Administration menu.
3. Click server monitoring
4. Click the diagram "Used JDBC connection"

Comment 6 Sean Flanigan 2013-10-17 03:22:27 UTC
Simultaneously launching two instances of the same editor (same language) from a project with many documents seems to be fairly reliable at reproducing the exception, as long as the relevant documents/statistics haven't been cached already.  (You can use JavaMelody to clear all data caches.)

07:07:12,839 INFO  [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (http-/0.0.0.0:8080-7) IJ000311: Throwable from unregister connection: java.lang.IllegalStateException: Trying to return an unknown connection2! org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@b0da531

Every time that exception is logged, another connection is leaked (or already was), although the cause is probably a little earlier, possibly in a different thread.

Comment 7 Sean Flanigan 2013-10-17 03:26:43 UTC
Pull request: https://github.com/zanata/zanata-server/pull/229

Comment 8 Ding-Yi Chen 2013-10-17 07:53:35 UTC
VERIFIED Zanata version 3.1.1-SNAPSHOT (20131017-1642)

Comment 9 Ding-Yi Chen 2013-10-22 06:44:43 UTC
*** Bug 1020585 has been marked as a duplicate of this bug. ***

Comment 11 Sean Flanigan 2013-10-22 23:43:23 UTC
Created attachment 815210 [details]
When the connections were created (according to JavaMelody)

Comment 12 Sean Flanigan 2013-10-24 03:17:21 UTC
False alarm (I think).

It appears that the connections are being retained deliberately by the DataSource, not leaked, because there are no warning messages in the log such as "Trying to return an unknown connection2!".  Also, several of the connections were created at the same time, which may suggest that they were created to expand a connection pool (unless they *all* happened to leak at the same time).

Looking at the stack traces, I can't find any suspicous code (eg ResultSet handling) that would explain such a leak.  Unless we actually run out of connections (or get the above warning message again), I think we'll have to assume it's behaving normally.  I think it's just normal DataSource connection pool behaviour.

However, if we get the above log warning, or a server failure like "javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout" or "javax.resource.ResourceException: IJ000453: Unable to get managed connection for ..." we should create a new bug.

Comment 13 Sean Flanigan 2013-11-27 03:14:13 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 14 Sean Flanigan 2013-11-27 03:16:03 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 15 Sean Flanigan 2013-11-27 03:19:53 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 16 Sean Flanigan 2013-11-27 03:23:58 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 17 Sean Flanigan 2013-11-27 03:34:44 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 18 Sean Flanigan 2013-11-27 03:36:51 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.


Note You need to log in before you can comment on or make changes to this bug.