Bug 1259892

Description of problem:
We have a multi-threaded Java application which makes dozens of connections to the Amazon SQS web service to read messages every 15 minutes.  Occasionally -- about once a day or so now -- one of the threads will hang, preventing the application from terminating and blocking subsequent runs until we find out about it and kill the process.

After taking some Java stack dumps while hung and some research we found that the Apache httpclient library had a bug where the socket timeout was ignored during the SSL handshake:

https://issues.apache.org/jira/browse/HTTPCLIENT-1478

This was reportedly fixed in version 4.3.6 of httpcomponents-client, but RHEL6 only has commons-httpclient 3.1 which is no longer supported.  I don't see anything showing whether the fix was backported to the older library.

Version-Release number of selected component (if applicable):
3.1-0.9.el6_5

How reproducible:
Low -- roughly 0.03% probability

Steps to Reproduce:
1. Create a MultiThreadedHttpConnectionManager.
2. getConnectionWithTimeout (e.g. 60 seconds)
3. In the HttpConnectionParams, setConnectionTimeout(60 seconds), setSoTimeout(60 seconds), setBooleanParameter(STALE_CONNECTION_CHECK, true)
4. Create a TransparentPostMethod and fill it in with the required parameters for an Amazon SQS request.  Use the AWSUtils library to sign the request.
5. Open a connection, then execute the request.

Actual results:
Thread has a small chance of hanging indefinitely in the following call (jstack dump):
Thread 15254: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=87, line=152 (Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=122 (Compiled frame)
 - sun.security.ssl.InputRecord.readFully(java.io.InputStream, byte[], int, int) @bci=21, line=442 (Compiled frame)
 - sun.security.ssl.InputRecord.read(java.io.InputStream, java.io.OutputStream) @bci=32, line=480 (Interpreted frame)
 - sun.security.ssl.SSLSocketImpl.readRecord(sun.security.ssl.InputRecord, boolean) @bci=44, line=927 (Interpreted frame)
 - sun.security.ssl.SSLSocketImpl.performInitialHandshake() @bci=84, line=1312 (Interpreted frame)
 - sun.security.ssl.SSLSocketImpl.startHandshake(boolean) @bci=13, line=1339 (Interpreted frame)
 - sun.security.ssl.SSLSocketImpl.getSession() @bci=10, line=2171 (Interpreted frame)
 - org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.verifyHostName(java.lang.String, javax.net.ssl.SSLSocket) @bci=15, line=213 (Interpreted frame)
 - org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(java.lang.String, int, java.net.InetAddress, int, org.apache.commons.httpclient.params.HttpConnectionParams) @bci=88, line=159 (Interpreted frame)
 - org.apache.commons.httpclient.HttpConnection.open() @bci=182, line=710 (Interpreted frame)
 - org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open() @bci=11, line=1361 (Interpreted frame)
 - com.boingo.cloud.aws.sqs.AmazonSimpleQueueServiceImplementation.pollMessagesInternal(int, org.apache.commons.httpclient.HttpClient, long) @bci=246 (Compiled frame)
 - com.boingo.cloud.aws.sqs.AmazonSimpleQueueServiceImplementation.pollMessages(int, org.apache.commons.httpclient.HttpClient, long) @bci=57 (Interpreted frame)
 - com.boingo.sqs.MessageReader.run() @bci=24, line=59 (Interpreted frame)


Expected results:
Connection times out after 60 seconds, throwing a SocketTimeoutException.

Additional info:
http://insidecoffe.blogspot.com/2011/12/when-timeout-fails-in-threadjoin.html