Description of problem: Can not trigger jenkins build on RHEL6.6. "Could not generate DH keypair" was thrown out. Version-Release number of selected component (if applicable): puddle [2.1.z/2014-08-06.1] RHEL [RHEL6.6/2014-07-31.1] java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1. Install env on RHEL66 2. Create an app with jenkins #rhc app create phps php-5.4 -s --enable-jenkins 3. Do some change, trigger jenkins build #git commit -amx && git push Actual results: Can not trigger jenkins build Expected results: Deploy the app successfully Additional info: INFO: Initiating Java Client Service - Configured for OpenShift Server https://broker.ose21z-auto.com.cn Aug 07, 2014 2:53:39 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://broker.ose21z-auto.com.cn/broker/rest/api Aug 07, 2014 2:53:40 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://broker.ose21z-auto.com.cn/broker/rest/api: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair. Will retry 0 more times before canceling build. Aug 07, 2014 2:53:45 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Cancelling build due to earlier exceptions java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://broker.ose21z-auto.com.cn/broker/rest/api: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair at hudson.plugins.openshift.OpenShiftCloud.getOpenShiftConnection(OpenShiftCloud.java:186) at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:877) at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:451) at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:413) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.openshift.client.OpenShiftEndpointException: Could not request https://broker.ose21z-auto.com.cn/broker/rest/api: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
Can deploy app successfully with jenkins on OSE1.2 and OSE2.1
(In reply to Ma xiaoqiang from comment #0) > Can not trigger jenkins build on RHEL6.6. "Could not generate DH keypair" > was thrown out. Wow, nice catch. Is this specific to RHEL 6.6 java update? > Version-Release number of selected component (if applicable): > java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64 I suspect java-1.7.0-openjdk is what is actually being used. > Additional info: > java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could > not request https://broker.ose21z-auto.com.cn/broker/rest/api: > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate > DH keypair > at > hudson.plugins.openshift.OpenShiftCloud. > getOpenShiftConnection(OpenShiftCloud.java:186) This is probably just where the exception was caught and re-thrown, so doesn't really indicate anything wrong in plugin. [...] > Caused by: com.openshift.client.OpenShiftEndpointException: Could not > request https://broker.ose21z-auto.com.cn/broker/rest/api: > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate > DH keypair Was there more exception after this? I would expect the real exception to have a stack trace here. Probably not too important in this case, though, since this is probably enough to find the problem. I don't have a clear solution but it seems like a bug in the OpenJDK JVM (so it's kind of important to determine which version Jenkins is using) - see http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-could-not-generate-dh-keypair-exception (first answer indicates it's fixed for some versions of Java). Basically Java SSL breaks when it tries to create a session with a server using a DHE cipher with "too large" a key. Could likely be addressed using a different JVM (perhaps we can get RHEL 6.6 to address this bug). It also depends on the cipher being used for the broker httpd SSL/TLS, so could perhaps be worked around by changing the CipherSuite settings there to exclude DHE ciphers.
(In reply to Luke Meyer from comment #2) > (In reply to Ma xiaoqiang from comment #0) > > > Can not trigger jenkins build on RHEL6.6. "Could not generate DH keypair" > > was thrown out. > > Wow, nice catch. Is this specific to RHEL 6.6 java update? Maybe it is not only relative to RHEL 6.6 java update, because RHEL 6.6 + ose-2.0 RHEL 6.6 + ose-1.2 are working well. > > > Version-Release number of selected component (if applicable): > > java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64 > > I suspect java-1.7.0-openjdk is what is actually being used. Yes, you are right, it is using java-1.7.0-openjdk, maybe xiaoqaing did not paste correct java version. > > Caused by: com.openshift.client.OpenShiftEndpointException: Could not > > request https://broker.ose21z-auto.com.cn/broker/rest/api: > > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate > > DH keypair > > Was there more exception after this? I would expect the real exception to > have a stack trace here. Probably not too important in this case, though, > since this is probably enough to find the problem. No stack trace is seen in jenkins log, only this exception could be seen. > > I don't have a clear solution but it seems like a bug in the OpenJDK JVM (so > it's kind of important to determine which version Jenkins is using) - see > http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give- > could-not-generate-dh-keypair-exception (first answer indicates it's fixed > for some versions of Java). > > Basically Java SSL breaks when it tries to create a session with a server > using a DHE cipher with "too large" a key. > > Could likely be addressed using a different JVM (perhaps we can get RHEL 6.6 > to address this bug). It also depends on the cipher being used for the > broker httpd SSL/TLS, so could perhaps be worked around by changing the > CipherSuite settings there to exclude DHE ciphers. Maybe it is also relative with jenkins cartridge, just like what I said, RHEL 6.6 + ose-2.0 RHEL 6.6 + ose-1.2 are working well, only RHEL 6.6 + ose-2.1 has such issue.
(In reply to Johnny Liu from comment #3) > Yes, you are right, it is using java-1.7.0-openjdk, maybe xiaoqaing did not > paste correct java version. Typically both versions are installed on OSE nodes, so it's not always obvious which is actually being used. Typically whatever /etc/alternatives/java points at. > Maybe it is also relative with jenkins cartridge, just like what I said, > RHEL 6.6 + ose-2.0 > RHEL 6.6 + ose-1.2 > are working well, only RHEL 6.6 + ose-2.1 has such issue. With OSE 2.1 we changed the default CipherSuite config to remove a lot of ciphers considered less secure. This probably brings DFE to the top, which combined with the JVM problem triggers this bug. That is my guess at this point. Assuming OSE 2.1 Jenkins builds worked fine with earlier JVM versions then it could be considered a RHEL 6.6 regression and may perhaps be addressed there (not certain how to give RHEL a good test case for a bug report though). Otherwise remove the problematic ciphers as a workaround.
Marián Labuda <mlabuda> updated the status of jira JBIDE-18454 to Closed
Re-test this bug with java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64 and ose-2.1.z/2014-10-08.1 puddle on RHEL-6.6 RC build. Create jbosseap app, and add jenkins-client to it, trigger jenkins build, succeed. But seem like the PR mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1127667#c17 should be merged, am I right?
Yes it should.
The latest puddle came out, so move it to "ON_QA".
Verified this bug with ose-2.1.z/2014-10-09.3 puddle and java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64 on RHEL-6.6, PASS. Scenarios 1: 1. Set up env on RHEL-6.6 (RHEL-6.6-20140926.0) 2. Create jbosseap, and embed jenkin-client 3. Do some change in app repo, then git push to trigger jenkins build, jenkins build is failed which is epxected. 4. In jenkins app git repo, touch .openshift/markers/disable_bad_ciphers_yes, then git push. 5. Check jenkins process, found "-Ddisable_bad_sslciphers=yes" is appended to java process. # ps -ef|grep java 4872 1648 1 22 11:08 ? 00:00:26 /etc/alternatives/jre/bin/java -Xmx168m -XX:MaxPermSize=100m -Djava.awt.headless=true -Dhudson.udp=-1 -Ddisable_bad_sslciphers=yes -DJENKINS_HOME=/var/lib/openshift/54373e7687692b793a00009e/app-root/data/ -Dhudson.slaves.NodeProvisioner.recurrencePeriod=500 -Dhudson.slaves.NodeProvisioner.initialDelay=100 -Dhudson.slaves.NodeProvisioner.MARGIN=100 -Dhudson.model.UpdateCenter.never=true -Dhudson.DNSMultiCast.disabled=true -jar /usr/lib/jenkins/jenkins.war --ajp13Port=-1 --controlPort=-1 --logfile=/dev/stdout --httpPort=8080 --handlerCountMax=45 --handlerCountMaxIdle=20 --httpListenAddress=127.9.132.1 6. Do some change in app repo again, then git push to trigger jenkins build, succeed. 7. Update OpenJDK7 build to java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64, restart node. 8. Do some change in app repo again, then git push to trigger jenkins build, succeed. Scenarios 2: 1. Set up env on RHEL-6.6 (RHEL-6.6-20140926.0), Update OpenJDK7 build to java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64, restart node. 2. Create jbosseap, and embed jenkin-client 3. Do some change in app repo, then git push to trigger jenkins build, succeed. 4. Check jenkins process, there is no "-Ddisable_bad_sslciphers=yes" appended to java process. So this bug is fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1630.html
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Reopened
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Resolved
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Closed