Description of problem: We are configuring 2 SSL connectors in JBoss 6. When JBoss starts up, we sometimes get an error initializing one of the SSL web connectors. this occurs like 1 out of 20 times on average and appears randomly. The error message: 10:20:28,755 ERROR [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-1) Error initializing endpoint: java.io.IOException: SSL configuration is invalid due to No available certificate or key corresponds to the SSL cipher suites which are enabled. at org.apache.tomcat.util.net.jsse.JSSESocketFactory.checkConfig(JSSESocketFactory.java:788) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:493) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:168) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:978) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:190) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.apache.catalina.connector.Connector.init(Connector.java:1001) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:268) [jboss-as-web-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA-redhat-2.jar:1.0.2.GA-redhat-2] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA-redhat-2.jar:1.0.2.GA-redhat-2] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38] Caused by: javax.net.ssl.SSLException: No available certificate or key corresponds to the SSL cipher suites which are enabled. at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.checkEnabledSuites(SSLServerSocketImpl.java:310) [jsse.jar:1.6] at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:255) [jsse.jar:1.6] at org.apache.tomcat.util.net.jsse.JSSESocketFactory.checkConfig(JSSESocketFactory.java:784) [jbossweb-7.0.17.Final-redhat-1.jar:] ... 11 more Version-Release number of selected component (if applicable): How reproducible: 1. Configure 2 connection as follow:- <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" max-connections="100"> <ssl name="ssl" password="password" cipher-suite="TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_RC4_128_MD5, TLS_RSA_WITH_RC4_128_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA" verify-client="true" keystore-type="PKCS11"/> </connector> <connector name="common-login" protocol="HTTP/1.1" scheme="https" socket-binding="common-login" secure="true"> <ssl name="common-login" password="password" verify-client="false" keystore-type="PKCS11"/> </connector> 2. Start EAP6 20+ times Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I implemented a fix in a test build on EAP 6.0.1/JBossWeb 7.0.17.GA. This pretty much just allows for synchronized connector initialization through a system property. If that property is set to true, then Http11Protocol.init operations will be done synchronized on a static object. Testing shows the issue is successfully avoided altogether here with multiple https connector set up with that fix applied. I've attached the modified Http11Protocol class for a reference. What would you guys think about implementing this or something similar to it in future releases as an option for protection against this issue in the future?
Created attachment 767982 [details] test fix
For EAP 6.3.0.ER10 configured as described above, the following error still occurs randomly (approx. for 1 startup of 10): 17:20:39,281 ERROR [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-6) JBWEB003043: Error initializing endpoint: java.io.IOException: java.security.KeyStoreException: invalid KeyStore state: found 2 private keys sharing CKA_ID 0x9d1b9651c5675fbcee9b91a10c06e6118379fbc1 at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:440) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4] at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:162) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4] at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:973) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4] at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:174) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4] at org.apache.catalina.connector.Connector.init(Connector.java:985) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4] at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:318) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_55] 17:20:39,298 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.web.connector.https: org.jboss.msc.service.StartException in service jboss.web.connector.https: JBAS018007: Error starting web connector at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:376) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_55] Caused by: LifecycleException: JBWEB000023: Protocol handler initialization failed at org.apache.catalina.connector.Connector.init(Connector.java:987) at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:318) ... 5 more In my opinion (also confirmed by Radim), this error message is related to the same problem as described above.
After reviewing, I think it is more generic to sync the connector init. r2496 in web.
Fixed by component upgrade to JBoss Web 7.5.0.Beta1: https://bugzilla.redhat.com/show_bug.cgi?id=1138672
Verified that this issue is fixed in EAP 6.4.0.DR3.