Bug 1127667 - Can not trigger jenkins build on RHEL6.6
Summary: Can not trigger jenkins build on RHEL6.6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 2.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Brenton Leanhardt
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1131221
Blocks: 1145848 1146679 1150287 1150288
TreeView+ depends on / blocked
 
Reported: 2014-08-07 10:22 UTC by Ma xiaoqiang
Modified: 2016-07-04 00:44 UTC (History)
16 users (show)

Fixed In Version: jenkins-plugin-openshift-0.6.40.1-0.el6op jenkins-1.565.3-1.el6op openshift-origin-cartridge-jenkins-1.20.3.5-1.el6op
Doc Type: Bug Fix
Doc Text:
Cause: Changes to the httpd+mod_ssl packages in Red Hat Enterprise Linux 6.6 cause certain ciphers' key sizes offered during TLS/SSL handshaking to be larger than the same ciphers' key sizes in previous versions. These larger key sizes aren't supported by the current release of openjdk-1.7.0, and cause an exception during TLS/SSL handshaking. Consequence: On previously-working OpenShift Enterprise deployments which have been updated to Red Hat Enterprise Linux 6.6, Jenkins builds fail because the Jenkins plugin cannot negotiate an SSL connection with the Broker REST API endpoint. Fix: If an updated (i.e. newer than java-1.7.0-openjdk-1.7.0.65-2.5.1.2) OpenJDK package is available, then the openjdk-1.7.0 package should be updated. On systems where the update is either unavailable or otherwise cannot be installed, the updated jenkins cartridge and dependencies allow the problematic cipher to be disabled. Users can take advantage of this by checking out the jenkins gear repository and adding the "disable_bad_ciphers_yes" marker file. Result: Jenkins builds should work as before. It is important to note that disabling the problematic cipher degrades the security of the REST API connections from the Jenkins gear, and as soon as possible the OpenJDK package should be updated and the marker file removed from all active Jenkins gears.
Clone Of:
: 1145848 1146679 1150287 1150288 (view as bug list)
Environment:
Last Closed: 2014-10-14 13:01:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1035818 0 unspecified CLOSED Backport ECDH support to httpd 2021-02-22 00:41:40 UTC
Red Hat Issue Tracker JBIDE-18454 0 Blocker Closed Cant connect to OpenShift running on RHEL 6.6 (javax.net.ssl.SSLException: Could not generate DH keypair) 2018-11-22 15:37:19 UTC
Red Hat Issue Tracker OSJC-125 0 Major Closed Allow users to remove DHE ciphers 2018-11-22 15:37:19 UTC
Red Hat Product Errata RHBA-2014:1630 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.1 jenkins-plugin-openshift bug fix update 2014-10-14 17:01:14 UTC

Internal Links: 1035818

Description Ma xiaoqiang 2014-08-07 10:22:01 UTC
Description of problem:
Can not trigger jenkins build on RHEL6.6.  "Could not generate DH keypair" was thrown out.

Version-Release number of selected component (if applicable):
puddle [2.1.z/2014-08-06.1]
RHEL [RHEL6.6/2014-07-31.1]
java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Install env on RHEL66
2. Create an app with jenkins
#rhc app create phps php-5.4 -s --enable-jenkins
3. Do some change, trigger jenkins build
#git commit -amx && git push 

Actual results:
Can not trigger jenkins build


Expected results:
Deploy the app successfully

Additional info:
INFO: Initiating Java Client Service - Configured for OpenShift Server https://broker.ose21z-auto.com.cn
Aug 07, 2014 2:53:39 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://broker.ose21z-auto.com.cn/broker/rest/api
Aug 07, 2014 2:53:40 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://broker.ose21z-auto.com.cn/broker/rest/api: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair. Will retry 0 more times before canceling build.
Aug 07, 2014 2:53:45 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Cancelling build due to earlier exceptions
java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://broker.ose21z-auto.com.cn/broker/rest/api: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
        at hudson.plugins.openshift.OpenShiftCloud.getOpenShiftConnection(OpenShiftCloud.java:186)
        at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:877)
        at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:451)
        at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:413)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.openshift.client.OpenShiftEndpointException: Could not request https://broker.ose21z-auto.com.cn/broker/rest/api: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

Comment 1 Ma xiaoqiang 2014-08-07 10:24:30 UTC
Can deploy app successfully with jenkins on OSE1.2 and OSE2.1

Comment 2 Luke Meyer 2014-08-07 13:17:10 UTC
(In reply to Ma xiaoqiang from comment #0)

> Can not trigger jenkins build on RHEL6.6.  "Could not generate DH keypair"
> was thrown out.

Wow, nice catch. Is this specific to RHEL 6.6 java update?

> Version-Release number of selected component (if applicable):
> java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64

I suspect java-1.7.0-openjdk is what is actually being used.

> Additional info:
> java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could
> not request https://broker.ose21z-auto.com.cn/broker/rest/api:
> javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate
> DH keypair
>         at
> hudson.plugins.openshift.OpenShiftCloud.
> getOpenShiftConnection(OpenShiftCloud.java:186)

This is probably just where the exception was caught and re-thrown, so doesn't really indicate anything wrong in plugin.

[...]
> Caused by: com.openshift.client.OpenShiftEndpointException: Could not
> request https://broker.ose21z-auto.com.cn/broker/rest/api:
> javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate
> DH keypair

Was there more exception after this? I would expect the real exception to have a stack trace here. Probably not too important in this case, though, since this is probably enough to find the problem.

I don't have a clear solution but it seems like a bug in the OpenJDK JVM (so it's kind of important to determine which version Jenkins is using) - see http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-could-not-generate-dh-keypair-exception (first answer indicates it's fixed for some versions of Java).

Basically Java SSL breaks when it tries to create a session with a server using a DHE cipher with "too large" a key.

Could likely be addressed using a different JVM (perhaps we can get RHEL 6.6 to address this bug). It also depends on the cipher being used for the broker httpd SSL/TLS, so could perhaps be worked around by changing the CipherSuite settings there to exclude DHE ciphers.

Comment 3 Johnny Liu 2014-08-07 13:38:39 UTC
(In reply to Luke Meyer from comment #2)
> (In reply to Ma xiaoqiang from comment #0)
> 
> > Can not trigger jenkins build on RHEL6.6.  "Could not generate DH keypair"
> > was thrown out.
> 
> Wow, nice catch. Is this specific to RHEL 6.6 java update?

Maybe it is not only relative to RHEL 6.6 java update, because
RHEL 6.6 + ose-2.0
RHEL 6.6 + ose-1.2 
are working well.

> 
> > Version-Release number of selected component (if applicable):
> > java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64
> 
> I suspect java-1.7.0-openjdk is what is actually being used.

Yes, you are right, it is using java-1.7.0-openjdk, maybe xiaoqaing did not paste correct java version.


> > Caused by: com.openshift.client.OpenShiftEndpointException: Could not
> > request https://broker.ose21z-auto.com.cn/broker/rest/api:
> > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate
> > DH keypair
> 
> Was there more exception after this? I would expect the real exception to
> have a stack trace here. Probably not too important in this case, though,
> since this is probably enough to find the problem.

No stack trace is seen in jenkins log, only this exception could be seen.

> 
> I don't have a clear solution but it seems like a bug in the OpenJDK JVM (so
> it's kind of important to determine which version Jenkins is using) - see
> http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-
> could-not-generate-dh-keypair-exception (first answer indicates it's fixed
> for some versions of Java).
> 
> Basically Java SSL breaks when it tries to create a session with a server
> using a DHE cipher with "too large" a key.
> 
> Could likely be addressed using a different JVM (perhaps we can get RHEL 6.6
> to address this bug). It also depends on the cipher being used for the
> broker httpd SSL/TLS, so could perhaps be worked around by changing the
> CipherSuite settings there to exclude DHE ciphers.

Maybe it is also relative with jenkins cartridge, just like what I said,
RHEL 6.6 + ose-2.0
RHEL 6.6 + ose-1.2 
are working well, only RHEL 6.6 + ose-2.1 has such issue.

Comment 4 Luke Meyer 2014-08-07 13:53:31 UTC
(In reply to Johnny Liu from comment #3)
> Yes, you are right, it is using java-1.7.0-openjdk, maybe xiaoqaing did not
> paste correct java version.

Typically both versions are installed on OSE nodes, so it's not always obvious which is actually being used. Typically whatever /etc/alternatives/java points at.

> Maybe it is also relative with jenkins cartridge, just like what I said,
> RHEL 6.6 + ose-2.0
> RHEL 6.6 + ose-1.2 
> are working well, only RHEL 6.6 + ose-2.1 has such issue.

With OSE 2.1 we changed the default CipherSuite config to remove a lot of ciphers considered less secure. This probably brings DFE to the top, which combined with the JVM problem triggers this bug. That is my guess at this point. Assuming OSE 2.1 Jenkins builds worked fine with earlier JVM versions then it could be considered a RHEL 6.6 regression and may perhaps be addressed there (not certain how to give RHEL a good test case for a bug report though). Otherwise remove the problematic ciphers as a workaround.

Comment 20 JBoss JIRA Server 2014-10-08 05:31:20 UTC
Marián Labuda <mlabuda> updated the status of jira JBIDE-18454 to Closed

Comment 22 Johnny Liu 2014-10-09 10:12:57 UTC
Re-test this bug with java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64 and ose-2.1.z/2014-10-08.1 puddle on RHEL-6.6 RC build.

Create jbosseap app, and add jenkins-client to it, trigger jenkins build, succeed.


But seem like the PR mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1127667#c17 should be merged, am I right?

Comment 23 Ben Parees 2014-10-09 12:44:45 UTC
Yes it should.

Comment 24 Johnny Liu 2014-10-10 05:33:34 UTC
The latest puddle came out, so move it to "ON_QA".

Comment 25 Johnny Liu 2014-10-10 05:34:02 UTC
Verified this bug with ose-2.1.z/2014-10-09.3 puddle and java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64 on RHEL-6.6, PASS.

Scenarios 1:
1. Set up env on RHEL-6.6 (RHEL-6.6-20140926.0)
2. Create jbosseap, and embed jenkin-client
3. Do some change in app repo, then git push to trigger jenkins build, jenkins build is failed which is epxected.
4. In jenkins app git repo, touch .openshift/markers/disable_bad_ciphers_yes, then git push.
5. Check jenkins process, found "-Ddisable_bad_sslciphers=yes" is appended to java process.
# ps -ef|grep java
4872      1648     1 22 11:08 ?        00:00:26 /etc/alternatives/jre/bin/java -Xmx168m -XX:MaxPermSize=100m -Djava.awt.headless=true -Dhudson.udp=-1 -Ddisable_bad_sslciphers=yes -DJENKINS_HOME=/var/lib/openshift/54373e7687692b793a00009e/app-root/data/ -Dhudson.slaves.NodeProvisioner.recurrencePeriod=500 -Dhudson.slaves.NodeProvisioner.initialDelay=100 -Dhudson.slaves.NodeProvisioner.MARGIN=100 -Dhudson.model.UpdateCenter.never=true -Dhudson.DNSMultiCast.disabled=true -jar /usr/lib/jenkins/jenkins.war --ajp13Port=-1 --controlPort=-1 --logfile=/dev/stdout --httpPort=8080 --handlerCountMax=45 --handlerCountMaxIdle=20 --httpListenAddress=127.9.132.1
6. Do some change in app repo again, then git push to trigger jenkins build, succeed.
7. Update OpenJDK7 build to java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64, restart node.
8. Do some change in app repo again, then git push to trigger jenkins build, succeed.

Scenarios 2:
1. Set up env on RHEL-6.6 (RHEL-6.6-20140926.0), Update OpenJDK7 build to java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64, restart node.
2. Create jbosseap, and embed jenkin-client
3. Do some change in app repo, then git push to trigger jenkins build, succeed.
4. Check jenkins process, there is no "-Ddisable_bad_sslciphers=yes" appended to java process.


So this bug is fixed.

Comment 27 errata-xmlrpc 2014-10-14 13:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1630.html

Comment 28 JBoss JIRA Server 2014-10-14 19:38:52 UTC
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Reopened

Comment 29 JBoss JIRA Server 2014-10-14 19:39:06 UTC
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Resolved

Comment 30 JBoss JIRA Server 2014-10-14 19:39:15 UTC
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Closed

Comment 31 JBoss JIRA Server 2014-10-16 09:58:09 UTC
Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Reopened


Note You need to log in before you can comment on or make changes to this bug.