Bug 959115

Summary:	Database connection pool leaks under load
Product:	[Retired] Zanata	Reporter:	Sean Flanigan <sflaniga>
Component:	Component-Logic	Assignee:	Sean Flanigan <sflaniga>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ding-Yi Chen <dchen>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	2.3	CC:	zanata-bugs
Target Milestone:	---
Target Release:	2.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	2.3.2-SNAPSHOT (20130514-0010)	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1018633 (view as bug list)		Environment:
Last Closed:	2013-11-27 03:24:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sean Flanigan 2013-05-03 07:51:46 UTC

Description of problem:
Connections are being leaked occasionally in production, until the pool is exhausted.

Version-Release number of selected component (if applicable):
version 2.3.0 (20130426-0119)

How reproducible:
Requires several days of production load.

Steps to Reproduce:
1. Boot Zanata
2. Allow users to log in to the editor and generate load
3. Keep watching https://zanata.example.com/admin/monitoring?part=graph&graph=usedConnections&period=semaine every few hours

Actual results:
The number of used connections goes up and down depending on load, but gradually creeps higher and higher (most connections are returned to the pool, only some of them are leaked), until the pool is exhausted and any database operation causes an error.

Expected results:
During quiet periods, the number of used connections should drop.  If the server is completely idle, it should drop to 1 (the minimum).

Additional info:

Comment 1 Sean Flanigan 2013-05-03 08:06:13 UTC

Fixed in https://github.com/zanata/zanata/commit/6bf6c409f24cf4c688a666d6955f39f66b66d9ea in release branch.

Comment 2 Sean Flanigan 2013-05-13 06:54:02 UTC

Fixed another potential culprit in release branch: https://github.com/zanata/zanata/commit/c4fb479347a6391c40bf39d540d34a9eed2db6b3

Comment 4 Sean Flanigan 2013-05-13 07:15:47 UTC

When testing this, please verify the functionality of the translation state cache: 
  validation status, last translated date and last translator in the doc list 
  and also up to date TM results when TextFlows have just become translated or untranslated in the editor.

Other than that, please watch out for errors like this in the server log:

[org.jboss.resource.connectionmanager.CachedConnectionManager] (http-0.0.0.0-8080-23) Closing a connection for you.  Please close them yourself: org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6@1a4f4082

Note that this error message will only be generated if the debug option is true in jboss-as-web/server/production/deploy/jca-jboss-beans.xml (which it is on our test machines)

Comment 5 Ding-Yi Chen 2013-05-14 07:17:30 UTC

VERIFIED with Zanata version 3.0.0-SNAPSHOT (20130514-1433)
and Zanata version 2.3.2-SNAPSHOT (20130514-0010).

Comment 6 Sean Flanigan 2013-11-27 03:15:04 UTC

Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 7 Sean Flanigan 2013-11-27 03:16:37 UTC

Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 8 Sean Flanigan 2013-11-27 03:18:31 UTC

Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 9 Sean Flanigan 2013-11-27 03:24:36 UTC

Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 10 Sean Flanigan 2013-11-27 03:33:11 UTC

Closing VERIFIED bugs for Zanata versions <= 3.1.

Comment 11 Sean Flanigan 2013-11-27 03:35:27 UTC

Closing VERIFIED bugs for Zanata versions <= 3.1.