Bug 959115 - Database connection pool leaks under load
Summary: Database connection pool leaks under load
Alias: None
Product: Zanata
Classification: Retired
Component: Component-Logic
Version: 2.3
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 2.3
Assignee: Sean Flanigan
QA Contact: Ding-Yi Chen
Depends On:
TreeView+ depends on / blocked
Reported: 2013-05-03 07:51 UTC by Sean Flanigan
Modified: 2013-11-27 03:35 UTC (History)
1 user (show)

Fixed In Version: 2.3.2-SNAPSHOT (20130514-0010)
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1018633 (view as bug list)
Last Closed: 2013-11-27 03:24:36 UTC

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1018633 0 urgent CLOSED Connection leak in production system 2021-02-22 00:41:40 UTC

Internal Links: 1018633

Description Sean Flanigan 2013-05-03 07:51:46 UTC
Description of problem:
Connections are being leaked occasionally in production, until the pool is exhausted.

Version-Release number of selected component (if applicable):
version 2.3.0 (20130426-0119)

How reproducible:
Requires several days of production load.

Steps to Reproduce:
1. Boot Zanata
2. Allow users to log in to the editor and generate load
3. Keep watching https://zanata.example.com/admin/monitoring?part=graph&graph=usedConnections&period=semaine every few hours

Actual results:
The number of used connections goes up and down depending on load, but gradually creeps higher and higher (most connections are returned to the pool, only some of them are leaked), until the pool is exhausted and any database operation causes an error.

Expected results:
During quiet periods, the number of used connections should drop.  If the server is completely idle, it should drop to 1 (the minimum).

Additional info:

Comment 1 Sean Flanigan 2013-05-03 08:06:13 UTC
Fixed in https://github.com/zanata/zanata/commit/6bf6c409f24cf4c688a666d6955f39f66b66d9ea in release branch.

Comment 2 Sean Flanigan 2013-05-13 06:54:02 UTC
Fixed another potential culprit in release branch: https://github.com/zanata/zanata/commit/c4fb479347a6391c40bf39d540d34a9eed2db6b3

Comment 4 Sean Flanigan 2013-05-13 07:15:47 UTC
When testing this, please verify the functionality of the translation state cache: 
  validation status, last translated date and last translator in the doc list 
  and also up to date TM results when TextFlows have just become translated or untranslated in the editor.

Other than that, please watch out for errors like this in the server log:

[org.jboss.resource.connectionmanager.CachedConnectionManager] (http- Closing a connection for you.  Please close them yourself: org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6@1a4f4082

Note that this error message will only be generated if the debug option is true in jboss-as-web/server/production/deploy/jca-jboss-beans.xml (which it is on our test machines)

Comment 5 Ding-Yi Chen 2013-05-14 07:17:30 UTC
VERIFIED with Zanata version 3.0.0-SNAPSHOT (20130514-1433)
and Zanata version 2.3.2-SNAPSHOT (20130514-0010).

Comment 6 Sean Flanigan 2013-11-27 03:15:04 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 7 Sean Flanigan 2013-11-27 03:16:37 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 8 Sean Flanigan 2013-11-27 03:18:31 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 9 Sean Flanigan 2013-11-27 03:24:36 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 10 Sean Flanigan 2013-11-27 03:33:11 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 11 Sean Flanigan 2013-11-27 03:35:27 UTC
Closing VERIFIED bugs for Zanata versions <= 3.1.

Note You need to log in before you can comment on or make changes to this bug.