Bug 1127667
Summary: | Can not trigger jenkins build on RHEL6.6 | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Ma xiaoqiang <xiama> | |
Component: | Containers | Assignee: | Brenton Leanhardt <bleanhar> | |
Status: | CLOSED ERRATA | QA Contact: | libra bugs <libra-bugs> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 2.1.0 | CC: | adietish, bparees, cww, fweimer, hkario, jialiu, jokerman, jolamb, libra-onpremise-devel, lmeyer, misalunk, mmccomas, pep, salmy, tiwillia, xtian | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | jenkins-plugin-openshift-0.6.40.1-0.el6op jenkins-1.565.3-1.el6op openshift-origin-cartridge-jenkins-1.20.3.5-1.el6op | Doc Type: | Bug Fix | |
Doc Text: |
Cause: Changes to the httpd+mod_ssl packages in Red Hat Enterprise Linux 6.6 cause certain ciphers' key sizes offered during TLS/SSL handshaking to be larger than the same ciphers' key sizes in previous versions. These larger key sizes aren't supported by the current release of openjdk-1.7.0, and cause an exception during TLS/SSL handshaking.
Consequence: On previously-working OpenShift Enterprise deployments which have been updated to Red Hat Enterprise Linux 6.6, Jenkins builds fail because the Jenkins plugin cannot negotiate an SSL connection with the Broker REST API endpoint.
Fix: If an updated (i.e. newer than java-1.7.0-openjdk-1.7.0.65-2.5.1.2) OpenJDK package is available, then the openjdk-1.7.0 package should be updated. On systems where the update is either unavailable or otherwise cannot be installed, the updated jenkins cartridge and dependencies allow the problematic cipher to be disabled. Users can take advantage of this by checking out the jenkins gear repository and adding the "disable_bad_ciphers_yes" marker file.
Result: Jenkins builds should work as before. It is important to note that disabling the problematic cipher degrades the security of the REST API connections from the Jenkins gear, and as soon as possible the OpenJDK package should be updated and the marker file removed from all active Jenkins gears.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1145848 1146679 1150287 1150288 (view as bug list) | Environment: | ||
Last Closed: | 2014-10-14 13:01:20 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1131221 | |||
Bug Blocks: | 1145848, 1146679, 1150287, 1150288 |
Description
Ma xiaoqiang
2014-08-07 10:22:01 UTC
Can deploy app successfully with jenkins on OSE1.2 and OSE2.1 (In reply to Ma xiaoqiang from comment #0) > Can not trigger jenkins build on RHEL6.6. "Could not generate DH keypair" > was thrown out. Wow, nice catch. Is this specific to RHEL 6.6 java update? > Version-Release number of selected component (if applicable): > java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64 I suspect java-1.7.0-openjdk is what is actually being used. > Additional info: > java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could > not request https://broker.ose21z-auto.com.cn/broker/rest/api: > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate > DH keypair > at > hudson.plugins.openshift.OpenShiftCloud. > getOpenShiftConnection(OpenShiftCloud.java:186) This is probably just where the exception was caught and re-thrown, so doesn't really indicate anything wrong in plugin. [...] > Caused by: com.openshift.client.OpenShiftEndpointException: Could not > request https://broker.ose21z-auto.com.cn/broker/rest/api: > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate > DH keypair Was there more exception after this? I would expect the real exception to have a stack trace here. Probably not too important in this case, though, since this is probably enough to find the problem. I don't have a clear solution but it seems like a bug in the OpenJDK JVM (so it's kind of important to determine which version Jenkins is using) - see http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-could-not-generate-dh-keypair-exception (first answer indicates it's fixed for some versions of Java). Basically Java SSL breaks when it tries to create a session with a server using a DHE cipher with "too large" a key. Could likely be addressed using a different JVM (perhaps we can get RHEL 6.6 to address this bug). It also depends on the cipher being used for the broker httpd SSL/TLS, so could perhaps be worked around by changing the CipherSuite settings there to exclude DHE ciphers. (In reply to Luke Meyer from comment #2) > (In reply to Ma xiaoqiang from comment #0) > > > Can not trigger jenkins build on RHEL6.6. "Could not generate DH keypair" > > was thrown out. > > Wow, nice catch. Is this specific to RHEL 6.6 java update? Maybe it is not only relative to RHEL 6.6 java update, because RHEL 6.6 + ose-2.0 RHEL 6.6 + ose-1.2 are working well. > > > Version-Release number of selected component (if applicable): > > java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64 > > I suspect java-1.7.0-openjdk is what is actually being used. Yes, you are right, it is using java-1.7.0-openjdk, maybe xiaoqaing did not paste correct java version. > > Caused by: com.openshift.client.OpenShiftEndpointException: Could not > > request https://broker.ose21z-auto.com.cn/broker/rest/api: > > javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate > > DH keypair > > Was there more exception after this? I would expect the real exception to > have a stack trace here. Probably not too important in this case, though, > since this is probably enough to find the problem. No stack trace is seen in jenkins log, only this exception could be seen. > > I don't have a clear solution but it seems like a bug in the OpenJDK JVM (so > it's kind of important to determine which version Jenkins is using) - see > http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give- > could-not-generate-dh-keypair-exception (first answer indicates it's fixed > for some versions of Java). > > Basically Java SSL breaks when it tries to create a session with a server > using a DHE cipher with "too large" a key. > > Could likely be addressed using a different JVM (perhaps we can get RHEL 6.6 > to address this bug). It also depends on the cipher being used for the > broker httpd SSL/TLS, so could perhaps be worked around by changing the > CipherSuite settings there to exclude DHE ciphers. Maybe it is also relative with jenkins cartridge, just like what I said, RHEL 6.6 + ose-2.0 RHEL 6.6 + ose-1.2 are working well, only RHEL 6.6 + ose-2.1 has such issue. (In reply to Johnny Liu from comment #3) > Yes, you are right, it is using java-1.7.0-openjdk, maybe xiaoqaing did not > paste correct java version. Typically both versions are installed on OSE nodes, so it's not always obvious which is actually being used. Typically whatever /etc/alternatives/java points at. > Maybe it is also relative with jenkins cartridge, just like what I said, > RHEL 6.6 + ose-2.0 > RHEL 6.6 + ose-1.2 > are working well, only RHEL 6.6 + ose-2.1 has such issue. With OSE 2.1 we changed the default CipherSuite config to remove a lot of ciphers considered less secure. This probably brings DFE to the top, which combined with the JVM problem triggers this bug. That is my guess at this point. Assuming OSE 2.1 Jenkins builds worked fine with earlier JVM versions then it could be considered a RHEL 6.6 regression and may perhaps be addressed there (not certain how to give RHEL a good test case for a bug report though). Otherwise remove the problematic ciphers as a workaround. Marián Labuda <mlabuda> updated the status of jira JBIDE-18454 to Closed Re-test this bug with java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64 and ose-2.1.z/2014-10-08.1 puddle on RHEL-6.6 RC build. Create jbosseap app, and add jenkins-client to it, trigger jenkins build, succeed. But seem like the PR mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1127667#c17 should be merged, am I right? Yes it should. The latest puddle came out, so move it to "ON_QA". Verified this bug with ose-2.1.z/2014-10-09.3 puddle and java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64 on RHEL-6.6, PASS. Scenarios 1: 1. Set up env on RHEL-6.6 (RHEL-6.6-20140926.0) 2. Create jbosseap, and embed jenkin-client 3. Do some change in app repo, then git push to trigger jenkins build, jenkins build is failed which is epxected. 4. In jenkins app git repo, touch .openshift/markers/disable_bad_ciphers_yes, then git push. 5. Check jenkins process, found "-Ddisable_bad_sslciphers=yes" is appended to java process. # ps -ef|grep java 4872 1648 1 22 11:08 ? 00:00:26 /etc/alternatives/jre/bin/java -Xmx168m -XX:MaxPermSize=100m -Djava.awt.headless=true -Dhudson.udp=-1 -Ddisable_bad_sslciphers=yes -DJENKINS_HOME=/var/lib/openshift/54373e7687692b793a00009e/app-root/data/ -Dhudson.slaves.NodeProvisioner.recurrencePeriod=500 -Dhudson.slaves.NodeProvisioner.initialDelay=100 -Dhudson.slaves.NodeProvisioner.MARGIN=100 -Dhudson.model.UpdateCenter.never=true -Dhudson.DNSMultiCast.disabled=true -jar /usr/lib/jenkins/jenkins.war --ajp13Port=-1 --controlPort=-1 --logfile=/dev/stdout --httpPort=8080 --handlerCountMax=45 --handlerCountMaxIdle=20 --httpListenAddress=127.9.132.1 6. Do some change in app repo again, then git push to trigger jenkins build, succeed. 7. Update OpenJDK7 build to java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64, restart node. 8. Do some change in app repo again, then git push to trigger jenkins build, succeed. Scenarios 2: 1. Set up env on RHEL-6.6 (RHEL-6.6-20140926.0), Update OpenJDK7 build to java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64, restart node. 2. Create jbosseap, and embed jenkin-client 3. Do some change in app repo, then git push to trigger jenkins build, succeed. 4. Check jenkins process, there is no "-Ddisable_bad_sslciphers=yes" appended to java process. So this bug is fixed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1630.html Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Reopened Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Resolved Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Closed Andre Dietisheim <adietish> updated the status of jira JBIDE-18454 to Reopened |