Created attachment 1264452 [details]
A bash script to make the requests with

Description of problem:
Right now I have a RH SSO server and 3 REST services deployed to the https://console.engint.openshift.com/console/ environment. I access them via a java command line client that uses a bearer token it obtains from the RH SSO server. Every other request to any of the 3 REST services fails with a 503 error.

The every other request problem had been isolated to just the 3 REST services recently, but now it is also happening with the RH SSO server when requesting a bearer token as well.

Version-Release number of selected component (if applicable):
OpenShift Master:
v3.4.1.8
Kubernetes Master:
v1.4.0+776c994

How reproducible:
Currently, very.

Steps to Reproduce:


Actual results:
[sso 842]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]

Requesting greeting...
{
  "id" : 5,
  "content" : "Hello, World!"
}
[sso 843]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]
Exception in thread "main" java.lang.RuntimeException: Failed to request token
        at client.authz.AuthzClient.obtainAccessToken(AuthzClient.java:70)
        at client.GreetingAuthzClient.init(GreetingAuthzClient.java:109)
        at client.GreetingAuthzClient.main(GreetingAuthzClient.java:62)
Caused by: javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:289)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:454)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:490)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.post(ClientInvocationBuilder.java:199)
        at client.authz.AuthzClient.obtainAccessToken(AuthzClient.java:66)
        ... 2 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
        at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:285)
        ... 6 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.InputRecord.read(InputRecord.java:505)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        ... 22 more
[sso 844]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]

Requesting greeting...
Exception in thread "main" javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:228)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:189)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.extractors.BodyEntityExtractor.extractEntity(BodyEntityExtractor.java:60)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:107)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
        at com.sun.proxy.$Proxy21.greeting(Unknown Source)
        at client.GreetingAuthzClient.getGreeting(GreetingAuthzClient.java:156)
        at client.GreetingAuthzClient.main(GreetingAuthzClient.java:64)
[sso 845]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]
Exception in thread "main" java.lang.RuntimeException: Failed to request token
        at client.authz.AuthzClient.obtainAccessToken(AuthzClient.java:70)
        at client.GreetingAuthzClient.init(GreetingAuthzClient.java:109)
        at client.GreetingAuthzClient.main(GreetingAuthzClient.java:62)
Caused by: javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:289)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:454)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:490)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.post(ClientInvocationBuilder.java:199)
        at client.authz.AuthzClient.obtainAccessToken(AuthzClient.java:66)
        ... 2 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
        at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:285)
        ... 6 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.InputRecord.read(InputRecord.java:505)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        ... 22 more
[sso 846]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]

Requesting greeting...
{
  "id" : 6,
  "content" : "Hello, World!"
}
[sso 847]$ 


Expected results:
[sso 846]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]

Requesting greeting...
{
  "id" : 6,
  "content" : "Hello, World!"
}
[sso 847]$ 
[sso 850]$ java -jar target/sso-client.jar --app secured-vertx-rest
Successful oc get routes: Yes
Using auth server URL: https://secure-sso-sso.e8ca.engint.openshiftapps.com/auth
Available application endpoint names: [secured-vertx-rest, secured-swarm-rest, secured-springboot-rest]

Requesting greeting...
{
  "id" : 7,
  "content" : "Hello, World!"
}
[

Additional info:

You can generate the same types of requests using the attached doReq.sh script. Run it using:
./doReq.sh secured-vertx-rest