Bug 1157749

Summary: [Regression] [EL6] Adding Foreman external provider fails with 'Could not generate DH keypair (Failed with error PROVIDER_SSL_FAILURE and code 5052)'
Product: [Retired] oVirt Reporter: Daniel Helgenberger <daniel.helgenberger>
Component: ovirt-engine-coreAssignee: Yaniv Bronhaim <ybronhei>
Status: CLOSED CURRENTRELEASE QA Contact: Pavel Stehlik <pstehlik>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5CC: alonbl, bugs, daniel.helgenberger, ecohen, gklein, iheim, lsurette, oourfali, rbalakri, s.kieske, yeylon
Target Milestone: ---Keywords: Regression
Target Release: 3.5.1   
Hardware: x86_64   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-03 11:27:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
openssl s_client -connect foreman.lab.mbox.loc:443 -showcerts -debug -state
none
CertificateChain.java
none
java -Djavax.net.debug=all CertificateChain https://foreman.lab.mbox.loc none

Description Daniel Helgenberger 2014-10-27 15:45:39 UTC
Description of problem:
Using oVirt Engine 3.5 running on an EL6 host adding Foreman external provider fails. This might be related to a regression in Java 7 (cipher suite order [1]).
However, even using seemingly newer versions of Java 7 effectively prevents EL6-Engines form using Foreman external providers. 

Version-Release number of selected component (if applicable):
Engine:
Linux CentOS6.5 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
ovirt-engine-3.5.0.1-1.el6.noarch
java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64
java-1.6.0-openjdk-1.6.0.0-7.1.13.4.el6_5.x86_64

Foreman:
1.6.2-stable; Host: EL7

How reproducible:
always

Steps to Reproduce:
1. Setup foreman on EL7
2. Setup oVirt Engine 3.5 on EL6
3. Add Foreman as external provider

Actual result:
Execution fails with 'Could not generate DH keypair (Failed with error PROVIDER_SSL_FAILURE and code 5052)'

Expected results:
Foreman is added as external provider

Additional info:
This issue seems be known [2]. However, for a normal user there seems to be no way to add Foreman to oVirt 3.5 atm.
Maybe related to my Foreman running EL7? (unlikely, I think).

[1] http://stackoverflow.com/questions/10687200/java-7-and-could-not-generate-dh-keypair
[2] https://www.mail-archive.com/users%40ovirt.org/msg22066.html

Comment 1 Alon Bar-Lev 2014-10-28 05:48:44 UTC
please attach foreman ssl certificate.

Comment 2 Alon Bar-Lev 2014-10-28 06:06:47 UTC
(In reply to Alon Bar-Lev from comment #1)
> please attach foreman ssl certificate.

Better, please attach the output of:

$ openssl s_client -connect @host@:443 -showcerts -debug -state

Comment 3 Daniel Helgenberger 2014-10-28 10:24:14 UTC
Created attachment 951346 [details]
openssl s_client -connect foreman.lab.mbox.loc:443 -showcerts -debug -state

Comment 4 Daniel Helgenberger 2014-10-28 10:25:32 UTC
Comment on attachment 951346 [details]
openssl s_client -connect foreman.lab.mbox.loc:443 -showcerts -debug -state

This was done form the engine.

Comment 5 Alon Bar-Lev 2014-10-28 10:33:04 UTC
Created attachment 951347 [details]
CertificateChain.java

Please try to run this program.

1. download into /tmp
2. cd /tmp
3. javac CertificateChain.java
4. java CertificateChain https://foreman.lab.mbox.lo
5. paste output.
6. paste java -version output.

Thanks!

Comment 6 Daniel Helgenberger 2014-10-28 11:06:09 UTC
Node, I installed java-1.7.0-openjdk-devel-1.7.0.65-2.5.1.2.el6_5.x86_64 to get javac.

# java CertificateChain https://foreman.lab.mbox.loc
Exception in thread "main" javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
	at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:397)
	at CertificateChain.getSSLPeerCertificates(CertificateChain.java:234)
	at CertificateChain.main(CertificateChain.java:242)

# java -version
java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

Comment 7 Oved Ourfali 2014-10-28 11:37:44 UTC
Reducing severity, as there is a clear and reasonable workaround.

Comment 8 Daniel Helgenberger 2014-10-28 11:46:30 UTC
Oved, can you elaborate how to upgrade jdk to witch version exactly on EL6 engine host without braking it?

Comment 9 Alon Bar-Lev 2014-10-28 12:55:32 UTC
(In reply to Daniel Helgenberger from comment #6)
> Node, I installed java-1.7.0-openjdk-devel-1.7.0.65-2.5.1.2.el6_5.x86_64 to
> get javac.

interesting!

can you please at least confirm it works with https://www.google.com?

and attach a file with the output of:

$ java -Djavax.net.debug=all CertificateChain https://foreman.lab.mbox.loc

thanks!

Comment 10 Daniel Helgenberger 2014-10-28 13:02:46 UTC
(In reply to Alon Bar-Lev from comment #9)
 
> interesting!
I am running engine from the recommanded minimal install. Seems to me jdk-devel was never required.

> 
> can you please at least confirm it works with https://www.google.com?

# java CertificateChain https://google.com
Certificate:
Subject: CN=*.google.com, O=Google Inc, L=Mountain View, ST=California, C=US
Issuer:  CN=Google Internet Authority G2, O=Google Inc, C=US
Certificate:
Subject: CN=Google Internet Authority G2, O=Google Inc, C=US
Issuer:  CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
Certificate:
Subject: CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
Issuer:  CN=GeoTrust Global CA, O=GeoTrust Inc., C=US


> 
> and attach a file with the output of:
> 
> $ java -Djavax.net.debug=all CertificateChain https://foreman.lab.mbox.loc
See attachment

> 
> thanks!

Comment 11 Daniel Helgenberger 2014-10-28 13:03:18 UTC
Created attachment 951400 [details]
java -Djavax.net.debug=all CertificateChain https://foreman.lab.mbox.loc

Comment 12 Alon Bar-Lev 2014-10-28 13:14:34 UTC
(In reply to Daniel Helgenberger from comment #11)
> Created attachment 951400 [details]
> java -Djavax.net.debug=all CertificateChain https://foreman.lab.mbox.loc

thanks!

so we confirm it is not ovirt issue, but java issue, more probably the settings of the server where foreman is installed on.

---
main, handling exception: java.lang.RuntimeException: Could not generate DH keypair
%% Invalidated:  [Session-1, TLS_DHE_RSA_WITH_AES_256_CBC_SHA]
main, SEND TLSv1 ALERT:  fatal, description = internal_error
main, WRITE: TLSv1 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 01 00 02 02 50                               ......P
main, called closeSocket()
main, IOException in getSession():  javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
---

Comment 13 Daniel Helgenberger 2014-10-28 13:30:16 UTC
Bad me, assuming oVirt without investigating Foreman. Turns out it's an open issue in puppet server [1]. Sadly I do not know how to add the ticket here as external bug.

I assume until this is fixed there is no foreman integration?

[1] https://tickets.puppetlabs.com/browse/SERVER-17

Comment 14 Daniel Helgenberger 2014-10-28 16:31:37 UTC
Ok, access seems to be working now with the workaround form [1] basically dropping DH by adding standard DH parameters to the cert. Still, the test fails with (Failed with error PROVIDER_FAILURE and code 5050) Note, I get beyond the authentication stage.

Sample Engine log:
2014-10-28 17:16:03,636 ERROR [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (ajp--127.0.0.1-8702-9) [1b68e5b4] Command org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand throw Vdc Bll exception. With error message VdcBLLException: PROVIDER_FAILURE (Failed with error PROVIDER_FAILURE and code 5050)


I can see access by the engine in foreman:
192.168.50.20 - - [28/Oct/2014:12:18:21 -0400] "GET /api/v2 HTTP/1.1" 401 72 "-" "Jakarta Commons-HttpClient/3.1"
192.168.50.20 - - [28/Oct/2014:12:18:22 -0400] "GET /api/v2 HTTP/1.1" 401 72 "-" "Jakarta Commons-HttpClient/3.1"
192.168.50.20 - - [28/Oct/2014:12:18:39 -0400] "GET /api/v2 HTTP/1.1" 200 13535 "-" "Jakarta Commons-HttpClient/3.1"
192.168.50.20 - - [28/Oct/2014:12:18:39 -0400] "GET /api/v2/discovered_hosts HTTP/1.1" 404 728 "-" "Jakarta Commons-HttpClient/3.1"


[1] http://httpd.apache.org/docs/current/ssl/ssl_faq.html#javadh

Comment 15 Daniel Helgenberger 2014-11-03 11:27:02 UTC
I close this bug because of the release of EL6.6 adds a better java version.

A summary:

Foreman integration dos *not* work on Engines running EL6.5 because of the java version available in EL6.5 repos (to old).

Confirmed working is:
java-1.7.0-openjdk-1.7.0.71-2.5.3.0.fc19.x86_64 [1]

Confirmed not working:
java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64

This was the default java version in the RH / Centos 6 version up to now. As luck has it, the recent release of EL6.6 updated java to:
java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64

Witch is confirmed working without the need of [2].

To get this working on < EL6.6, one needs to drop DH by adding default DH parameters to formemans HTTP server [2].

I close this bug; CURRENTRELEASE meaning EL6 and open-jdk. also it should be re-targeted to RHEL java package (if this makes any sense now).


[1] http://www.mail-archive.com/users@ovirt.org/msg22234.html
[2] http://httpd.apache.org/docs/current/ssl/ssl_faq.html#javadh