Red Hat Bugzilla – Bug 1014454
Reopenning connections in ClientGIOPConnection.closeAllowReopen() causes thread pool exhausts and NO_RESOURCES exception
Last modified: 2013-10-02 10:07:25 EDT
Created attachment 806267 [details]
Customer proposed fix
Description of problem:
Reopenning connections in ClientGIOPConnection.closeAllowReopen() causes thread pool exhausts and NO_RESOURCES exception.
The problem is due to a non-customizable optimization in jacorb, by way of which it recreates a ClientConnection to the remote host:port once it receives a SocketException (connect reset) from the remote peer.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Our client CORBA application (that talks to our server-side CORBA application) shuts down
2. jacorb ClientMessageReceptor thread receives a SocketException: connect reset
3. Optimization in jacorb closes connection and reopens another in anticipation that a client will try to connect to the remote host again on the same port. Note that an actual connect is not attempted immediately, rather the thread goes into a wait() state
4. A new instance of the client application is started that most probably binds to some other port.
5. Subsequent connections to new client application instance happen via a new ClientMessageReceptor thread
6. 1-5 are repeated each time the client application is recycled, thus leaving behind a waiting thread every time
7. Eventually after around a 1000 recycles cumulatively, jacorb’s thread pool exhausts and we get the NO_RESOURCES exception
jacorb’s thread pool exhausts and we get the NO_RESOURCES exception
Add customized configuration support on connection reopen.
The optimization lies in org.jacorb.orb.giop.ClientConnection.closeConnectionReceived()
Here you see that
is called instead of close(), in anticipation that a new connection to the same host:port will come up. In our case, it doesn't, because the client application is respawned and binds to a different port. Therefore, the corresponding ClientMessageReceptor thread on the server waits forever in GIOPConnection.waitUntilConnected()
It is possible to customize this behavior by modifying ClientGIOPConnection.closeAllowReopen(). Please see the modified implementation I am attaching alongside. I read in jacorb's ChangeLog that it used to be this way earlier [i.e. always call close()] but then this optimization was added to allow reopening the connection on the same port.