Bug 1225403

Summary: Multiple concurrent requests (non keep-alive) via HTTPS with server using NIO2 connector results in Exceptions written in debug log during handshaking.
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Radim Hatlapatka <rhatlapa>
Component: WebAssignee: Rémy Maucherat <rmaucher>
Status: CLOSED EOL QA Contact: Michael Cada <mcada>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.4.1CC: cdewolf, rmaucher, rpelisse
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-19 12:46:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Radim Hatlapatka 2015-05-27 10:05:00 UTC
Description of problem:
When having defined https with NIO2 connector [1] and doing multiple requests in parallel it results in Exceptions being written in debug log [2]. The exception is usually ClosedChannelException [3] or IOException [4].

Version-Release number of selected component (if applicable):
EAP 6.4.1.CP.CR2


How reproducible: always


Steps to Reproduce (for the applications and certificates see attachments in https://bugzilla.redhat.com/show_bug.cgi?id=1200276:
1. Unzip EAP distro
2. Deploy test.war (contains some static files with different byte length)
3. Load certificate and keystore to $JBOSS_HOME/standalone
4. Configure https and set protocol to NIO2 ([1])
5. Enable debug logging for "org.apache" ([2])
6. Do concurrent requests via ApacheBench (ab -n 3 -c 2 -Z DHE-RSA-AES256-SHA -v 3  https://127.0.0.1:8443/test/f256)


Actual results:
Requests are processed and exception [1] is shown in debug (all exceptions which occur on the channel are logged as debug)

Expected results:
Requests are processed without the exception [3] being thrown


Additional info:
This behaviour was detected only with enabled ssl (https) and only with non keep-alive requests.


[1]
 <connector name="https" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" socket-binding="https" enabled="true">
                <ssl name="http-perf" key-alias="jboss" password="jbossas" certificate-key-file="${jboss.server.base.dir}/server.keystore" cipher-suite="ALL" protocol="TLSv1" verify-client="none" certificate-file="${jboss.server.base.dir}/server.crt"/>
            </connector>

[2]
To enable debug logging for org.apache:

<logger category="org.apache">
      <level name="DEBUG"/>
</logger>

[3]
11:20:33,412 DEBUG [org.apache.tomcat.util] (http-/127.0.0.1:8443-9) JBWEB003023: Channel processing failed: javax.net.ssl.SSLException: java.util.concurrent.ExecutionException: java.io.IOException: Broken pipe
	at org.apache.tomcat.util.net.jsse.SecureNioChannel.handshake(SecureNioChannel.java:642) [jbossweb-7.5.7.Final.jar:7.5.7.Final]
	at org.apache.tomcat.util.net.jsse.NioJSSESocketChannelFactory.handshake(NioJSSESocketChannelFactory.java:227) [jbossweb-7.5.7.Final.jar:7.5.7.Final]
	at org.apache.tomcat.util.net.NioEndpoint$HandshakeHandler.run(NioEndpoint.java:635) [jbossweb-7.5.7.Final.jar:7.5.7.Final]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]
	at org.apache.tomcat.util.net.NioEndpoint$DefaultThreadFactory$1$1.run(NioEndpoint.java:1249) [jbossweb-7.5.7.Final.jar:7.5.7.Final]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Broken pipe
	at sun.nio.ch.CompletedFuture.get(CompletedFuture.java:69) [rt.jar:1.7.0_75]
	at org.apache.tomcat.util.net.jsse.SecureNioChannel.doHandshake(SecureNioChannel.java:760) [jbossweb-7.5.7.Final.jar:7.5.7.Final]
	at org.apache.tomcat.util.net.jsse.SecureNioChannel.handshake(SecureNioChannel.java:640) [jbossweb-7.5.7.Final.jar:7.5.7.Final]
	... 6 more
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.7.0_75]
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.7.0_75]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.7.0_75]
	at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.7.0_75]
	at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:691) [rt.jar:1.7.0_75]
	at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:381) [rt.jar:1.7.0_75]
	at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:386) [rt.jar:1.7.0_75]
	... 8 more

Comment 1 Rémy Maucherat 2015-05-27 12:40:16 UTC
It is possible, since this is an ab test, that incomplete writes occur. I'll review the code.