Bug 1054224

Created attachment 851073 [details]
trace logs with bug reproduced

Description of problem:
Customer case (https://c.na7.visual.force.com/apex/Case_View?id=500A000000HlfkX&sfdc.override=1) in which they said connections weren't being properly torn down if they deleted or disabled their Oracle datasource.

Version-Release number of selected component (if applicable):
JBoss EAP 6.2

How reproducible:
Your application must have all connections from a datasource InUse and then disable or delete that datasource.  This will cause all of the connections to be destroyed but will also throw a 'RetryableException' causing the datasource to be re-created while its being destroyed.  This leaves you with a non-existent datasource having X number of open connections to your database where X is the min pool size setting in your config.

If you do not have ALL connections available InUse and your disable or delete the datasource, all connections are properly torn down and a 'RetryableException' is not throw resulting in no pool re-creation.

Steps to Reproduce:
1) I created a standard Oracle Datasource with min connections of 5 and max connections of 30
2) I modified the default helloworld quickstart with a JSP that opens a Database connection, does a select and prints out the results.  I put this into a basic loop of 25 iterations and purposefully did not close the Database connection in order to create a bunch of InUse connections. 
3) I ran the JSP until it completed and it produced 25 InUse connections.
4) netstat -anp shows the 25 ESTABLISHED connections from my JBoss process to my Oracle DB.
5) select * from v$session in Oracle shows 25 open connections from JBoss.
6) I ran the JSP a second time until my server hit 30 (max) InUse connections.
7) netstat -anp shows the 30 ESTABLISHED connections from my JBoss process to my Oracle DB.
8) select * from v$session in Oracle shows 30 open connections from JBoss.
9) I stopped the datasource and I see 30 instances of 'WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (MSC service thread 1-7) IJ000615: Destroying active connection in pool: testpool' that indicates all the connections were destroyed,
10) netstat -anp now shows 5 connections from my JBoss process to my Oracle DB still in the ESTABLISHED state.
11) select * from v$session in Oracle now shows 5 open connections from JBoss.

Actual results:
The datasource is re-created and establishes X number of connections to the database where X is the value of min connection for your pool.

Expected results:
The datasource should not be re-created due to the exception and there should be no open connections left to the Database.

Additional info:

Log with bug reproduced:
01007809bug.trace.log
01007809bug.stats.log

Log from a passing scenario:
01007809pass.trace.log
01007809pass.stats.log