Backport BZ1410899 to 3.3 as customers are hitting it in production environment.
The original issue that is this is cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1410899 is because people are using invalid tokens. The root cause of the first issue was determined that Cloudforms was not properly configured for authentication to OpenShift. Are you sure this is not just a similar issue? I am trying to determine if we really need to backport this or not. The original fix to this just closes the connection more quickly instead of it timing out.
It seems this happens in other scenario as well, this time the request aborts connection (it looks like healthchecks, but couldn't be confirmed because of https://bugzilla.redhat.com/show_bug.cgi?id=1449022): 2017-05-08 02:47:44,679 DEBUG [io.undertow.request.io] (default I/O-5) UT005013: An IOException occurred: java.io.IOException: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack? at io.undertow.protocols.ssl.SslConduit.notifyReadClosed(SslConduit.java:606) at io.undertow.protocols.ssl.SslConduit.closed(SslConduit.java:971) at io.undertow.protocols.ssl.SslConduit.close(SslConduit.java:1066) at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:889) at io.undertow.protocols.ssl.SslConduit.write(SslConduit.java:377) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:247) at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:588) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:106) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:120) at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:154) at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:187) at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:1976) at io.undertow.io.AsyncSenderImpl.send(AsyncSenderImpl.java:208) at io.undertow.io.AsyncSenderImpl.send(AsyncSenderImpl.java:299) at io.undertow.io.AsyncSenderImpl.send(AsyncSenderImpl.java:271) at io.undertow.io.AsyncSenderImpl.send(AsyncSenderImpl.java:305) at io.undertow.server.handlers.error.SimpleErrorPageHandler$1.handleDefaultResponse(SimpleErrorPageHandler.java:70) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1545) at org.hawkular.openshift.auth.Utils.endExchange(Utils.java:56) at org.hawkular.openshift.auth.TokenAuthenticator.onPooledConnectionWaitTimeout(TokenAuthenticator.java:249) at org.hawkular.openshift.auth.TokenAuthenticator.lambda$createWaiter$3(TokenAuthenticator.java:229) at org.hawkular.openshift.auth.TokenAuthenticator$ConnectionPool.removeTimedOutWaiters(TokenAuthenticator.java:621) at org.hawkular.openshift.auth.TokenAuthenticator$ConnectionPool.offer(TokenAuthenticator.java:555) at org.hawkular.openshift.auth.TokenAuthenticator$ConnectionPool.access$300(TokenAuthenticator.java:495) at org.hawkular.openshift.auth.TokenAuthenticator.handleRequest(TokenAuthenticator.java:200) at org.hawkular.openshift.auth.OpenshiftAuthHandler.handleRequest(OpenshiftAuthHandler.java:106) at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:65) at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94) at org.wildfly.extension.undertow.Host$OptionsHandler.handleRequest(Host.java:285) at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:65) at io.undertow.server.handlers.SetHeaderHandler.handleRequest(SetHeaderHandler.java:90) at io.undertow.server.handlers.SetHeaderHandler.handleRequest(SetHeaderHandler.java:90) at org.wildfly.extension.undertow.Host$HostRootHandler.handleRequest(Host.java:293) at io.undertow.server.handlers.NameVirtualHostHandler.handleRequest(NameVirtualHostHandler.java:64) at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:76) at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49) at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:158) at io.undertow.server.handlers.DisallowedMethodsHandler.handleRequest(DisallowedMethodsHandler.java:61) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:233) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:131) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:57) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1116) at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88) at org.xnio.nio.WorkerThread.run(WorkerThread.java:559) Caused by: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack? at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561) at io.undertow.protocols.ssl.SslConduit.notifyReadClosed(SslConduit.java:604) ... 46 more And curl always returns: $ curl -kv -H "Authorization: Bearer *******************************************" -H "Hawkular-Tenant: openshift-infra" https://172.30.149.83/hawkular/metrics/metrics?type=gauge * About to connect() to 172.30.149.83 port 443 (#0) * Trying 172.30.149.83... * Connected to 172.30.149.83 (172.30.149.83) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * skipping SSL peer certificate verification * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: * subject: CN=hawkular-metrics * start date: Apr 28 16:37:44 2017 GMT * expire date: Apr 28 16:37:45 2019 GMT * common name: hawkular-metrics * issuer: CN=metrics-signer@1493397464 > GET /hawkular/metrics/metrics?type=gauge HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 172.30.149.83 > Accept: */* > Authorization: Bearer ******************************************* > Hawkular-Tenant: openshift-infra > < HTTP/1.1 500 Could not acquire a Kubernetes client connection < Connection: keep-alive < X-Powered-By: Undertow/1 < Server: WildFly/10 < Content-Length: 86 < Content-Type: text/html < Date: Tue, 02 May 2017 08:17:53 GMT < * Connection #0 to host 172.30.149.83 left intact <html><head><title>Error</title></head><body>500 - Internal Server Error</body></html>
*** This bug has been marked as a duplicate of bug 1461635 ***