Bug 1014454 - Reopenning connections in ClientGIOPConnection.closeAllowReopen() causes thread pool exhausts and NO_RESOURCES exception
Reopenning connections in ClientGIOPConnection.closeAllowReopen() causes thre...
Status: CLOSED INSUFFICIENT_DATA
Product: JBoss Enterprise Application Platform 5
Classification: JBoss
Component: jbossas (Show other bugs)
5.2.0
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Fernando Nasser
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-02 01:29 EDT by Lyle Wang
Modified: 2013-10-02 10:07 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-02 10:07:25 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Customer proposed fix (5.06 KB, text/x-java)
2013-10-02 01:29 EDT, Lyle Wang
no flags Details

  None (edit)
Description Lyle Wang 2013-10-02 01:29:41 EDT
Created attachment 806267 [details]
Customer proposed fix

Description of problem:

Reopenning connections in ClientGIOPConnection.closeAllowReopen() causes thread pool exhausts and NO_RESOURCES exception.
The problem is due to a non-customizable optimization in jacorb, by way of which it recreates a ClientConnection to the remote host:port once it receives a SocketException (connect reset) from the remote peer.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.	Our client CORBA application (that talks to our server-side CORBA application) shuts down

2.	jacorb ClientMessageReceptor thread receives a SocketException: connect reset

3.	Optimization in jacorb closes connection and reopens another in anticipation that a client will try to connect to the remote host again on the same port. Note that an actual connect is not attempted immediately, rather the thread goes into a wait() state

4.	A new instance of the client application is started that most probably binds to some other port.

5.	Subsequent connections to new client application instance happen via a new ClientMessageReceptor thread

6.	1-5 are repeated each time the client application is recycled, thus leaving behind a waiting thread every time

7.	Eventually after around a 1000 recycles cumulatively, jacorb’s thread pool exhausts and we get the NO_RESOURCES exception


Actual results:

jacorb’s thread pool exhausts and we get the NO_RESOURCES exception

Expected results:

Add customized configuration support on connection reopen.

Additional info:

The optimization lies in org.jacorb.orb.giop.ClientConnection.closeConnectionReceived()

Here you see that

((ClientGIOPConnection) connection).closeAllowReopen();

is called instead of close(), in anticipation that a new connection to the same host:port will come up. In our case, it doesn't, because the client application is respawned and binds to a different port. Therefore, the corresponding ClientMessageReceptor thread on the server waits forever in GIOPConnection.waitUntilConnected()

It is possible to customize this behavior by modifying ClientGIOPConnection.closeAllowReopen(). Please see the modified implementation I am attaching alongside. I read in jacorb's ChangeLog that it used to be this way earlier [i.e. always call close()] but then this optimization was added to allow reopening the connection on the same port.

Note You need to log in before you can comment on or make changes to this bug.