Bug 1231658

Summary: [GSS] (6.4.z) Prefill race condition in flush
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Takayoshi Kimura <tkimura>
Component: JCAAssignee: Dominik Pospisil <dpospisi>
Status: CLOSED CURRENTRELEASE QA Contact: Jan Martiska <jmartisk>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.1CC: bmaxwell, dpospisi, dstephan, jawilson, jmartisk, jpederse, sfikes, vtunka
Target Milestone: CR2   
Target Release: EAP 6.4.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1235745, 1278301    
Attachments:
Description Flags
server.log none

Description Takayoshi Kimura 2015-06-15 07:45:30 UTC 
Created attachment 1038802 [details]
server.log

There is a race condition between AbstractPool and SemaphoreArrayListManagedConnectionPool.

https://github.com/ironjacamar/ironjacamar/blob/ironjacamar-1.0.31.Final/core/src/main/java/org/jboss/jca/core/connectionmanager/pool/AbstractPool.java#L354
https://github.com/ironjacamar/ironjacamar/blob/ironjacamar-1.0.31.Final/core/src/main/java/org/jboss/jca/core/connectionmanager/pool/mcp/SemaphoreArrayListManagedConnectionPool.java#L625

AbstractPool flushes the mcp, mcp invokes PoolFiller which works in a separate thread. However AbstractPool is going to unregister the empty mcp and this mcp is no longer used. Then the dead mcp is prefilled in a separate thread by PoolFiller.

At next getConnection(), the mcp is re-initialized and prefilled.

I've attached logs, prefill enabled and min-size is 10. Boot server, invoke flush-all-connection-in-pool, getConnection() then killed. The prefill is done 3 times where it should be twice.
Comment 1 Takayoshi Kimura 2015-06-15 07:59:18 UTC 
In 1.0.31.Final:

> if (mcp.isEmpty())
>    clearMcpPools.add(mcp);

In 1.2 branch, commit 956af09c8494d4187db24a253e7e37b5ba5d9779 for JBJCA-1135:

> if (mcp.isEmpty() && !isPrefill() && size > 1)
>    clearMcpPools.add(mcp);

We probably need to backport "!isPrefill()" to the 1.0 branch.
Comment 2 Dominik Pospisil 2015-07-09 09:19:56 UTC 
Upstream PR sent: https://github.com/ironjacamar/ironjacamar/pull/366
Comment 4 Dominik Pospisil 2015-08-18 14:04:43 UTC 
Merged.
Comment 5 JBoss JIRA Server 2015-10-15 19:15:29 UTC 
Jesper Pedersen <jpederse> updated the status of jira JBJCA-1276 to Resolved
Comment 6 JBoss JIRA Server 2015-10-26 11:46:47 UTC 
Jesper Pedersen <jpederse> updated the status of jira JBJCA-1276 to Closed
Comment 8 Jan Martiska 2015-11-04 06:59:05 UTC 
Verified with EAP 6.4.5.CR1.
Comment 9 Jan Martiska 2015-11-06 09:46:15 UTC 
The fix for this caused a regression: https://bugzilla.redhat.com/show_bug.cgi?id=1277919
Therefore, keeping this reopened for now.
Comment 10 Vaclav Tunka 2015-11-09 16:12:21 UTC 
Fix is in 1.0.34, so marking this as MODIFIED:
https://github.com/ironjacamar/ironjacamar/commit/1511519f23450e06dd84faaf512ebe37416a8ad4
Comment 11 Jan Martiska 2015-11-12 10:54:45 UTC 
Verified with EAP 6.4.5.CR2, including the regression previously introduced.
Comment 12 Petr Penicka 2017-01-17 11:42:51 UTC 
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.
Comment 13 Petr Penicka 2017-01-17 11:43:05 UTC 
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.