Bug 1336838 - engine doesn't trust externally-issued web certificate for internal authentication in spite of issuer being in system (and java) trust store
Summary: engine doesn't trust externally-issued web certificate for internal authentic...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: AAA
Version: 4.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium vote
Target Milestone: ovirt-4.0.1
: 4.0.1.1
Assignee: Ravi Nori
QA Contact: Jiri Belka
URL:
Whiteboard:
: 1385062 (view as bug list)
Depends On:
Blocks: 1336845
TreeView+ depends on / blocked
 
Reported: 2016-05-17 14:32 UTC by David Jaša
Modified: 2017-03-14 17:11 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: oVirt 4.0 introduces Single Sign-On feature, which uses HTTPS connection to communicate between webadmin/userportal and SSO module. Consequence: If custom HTTPS certificate signed by custom CA is used in Apache, then users won't be able to login to webadmin/userportal after upgrade from 3.6 to 4.0. Fix: New configuration variables ENGINE_HTTPS_PKI_TRUST_STORE and ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD have been added to engine PKI configuration. By default those variables are set to use trust store with internal oVirt CA. If customer wants to use custom HTTPS certificate signed by different CA, then he has to perform following steps: 1. Install custom CA (that signed HTTPS certificate) into host wide trustore (more info can be found in update-ca-trust man page) 2. Configure HTTPS certificate in Apache (this step is same as in previous versions) 3. Create new configuration file (for example /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf) with following content: ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" 4. Restart ovirt-engine service Result: New installation: If user wants to use custom HTTPS certificate on Apache, he has to perform above configuration steps 1. - 4. Upgraded installation with custom HTTPS certificate: If user configured custom HTTPS certificate on Apache on previous version, he needs to perform above steps 1., 3. and 4. right after successfully finished upgrade Upgraded installation without custom HTTPS certificate: No special steps are required.
Clone Of:
Environment:
Last Closed: 2016-08-04 13:30:39 UTC
oVirt Team: Infra
rule-engine: ovirt-4.0.z+
rule-engine: planning_ack+
mperina: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 57812 master MERGED aaa: engine doesn't trust externally-issued web certificate 2016-05-31 07:44:33 UTC
oVirt gerrit 58322 ovirt-engine-4.0 MERGED aaa: engine doesn't trust externally-issued web certificate 2016-05-31 07:56:34 UTC
oVirt gerrit 59839 master MERGED aaa: Empty keystore passwd should be passed as null to KeyStore.load 2016-06-27 18:05:45 UTC
oVirt gerrit 59857 ovirt-engine-4.0 MERGED aaa: Empty keystore passwd should be passed as null to KeyStore.load 2016-06-28 07:56:35 UTC

Description David Jaša 2016-05-17 14:32:35 UTC
Description of problem:
engine doesn't trust externally-issued web certificate for internal authentication in spite of issuer being in system (and java) trust store

Version-Release number of selected component (if applicable):
ovirt-engine-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-backend-4.0.0-0.6.master.el7ev.noarch

How reproducible:
always

Steps to Reproduce:
1. get certificate for mod_ssl from external CA
2. add the external CA that signed the mod_ssl certificate to the trust store (trust add /path/to/CA.crt or cp /path/to/CA.crt /etc/pki/ca-trust/source/anchors/ && update-ca-trust)
3. verify that the CA appears in java trust store, e.g.:
keytool -list -keystore /usr/lib/jvm/jre/lib/security/cacerts \
| grep "$(openssl x509 -noout -fingerprint -sha1 -in \
/etc/pki/ca-trust/source/anchors/CA.crt \
| sed -e 's/SHA1 Fingerprint=//')"
4. run engine-setup with Manual SSL (OVESETUP_APACHE/configureSsl=bool:False)
5. try to log in as admin

Actual results:
admin logs in but an error occurs and users are redirected back:
> 2016-05-17 15:54:51,487 INFO  [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-27) [] User admin@internal successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access
> 2016-05-17 15:54:51,510 ERROR [org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-11) [] server_error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Expected results:
admin@internal logs in just fine after the setup

Additional info:

Comment 1 David Jaša 2016-05-18 09:02:00 UTC
Making /etc/pki/ovirt-engine/apache-ca.pem symlink point to web server CA doesn't help.

Comment 2 David Jaša 2016-05-26 11:14:12 UTC
> ... a new conf file needs to be created setting the new variables added
> to point to the system trust store's location and password.

Well, the point of system trust store is well known location and that you can depend on it being there and not bother admins looking up where it exports certificates for java consumption - it's file linked by default java store!
# readlink -f /usr/lib/jvm/jre/lib/security/cacerts
/etc/pki/ca-trust/extracted/java/cacerts

I'd understand knob to actually override this system store by custom store (or just by server certificate fingerprint) so that SSO couldn't be provided by e.g. rogue site signed by legitimate CA (analogous to LDAP server verification in AAA) - but that's not what you describe here.

Comment 3 David Jaša 2016-06-27 11:45:51 UTC
Works when using key store with password. When using system keystore that doesn't use password with no password (setting empty or none ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD variable).

Comment 4 David Jaša 2016-06-27 11:47:29 UTC
... the authentication won't work. This is the case of system trust store at /etc/pki/java/cacerts - I'm not sure however if this is now bug at engine or java or ca-certificates.

Comment 5 Red Hat Bugzilla Rules Engine 2016-06-27 11:53:38 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 6 Konstantin 2016-07-20 07:19:44 UTC
How about /ovirt-engine/services/reports-interface-proxy error when open dashboard?
I try to edit keystore parameter in /var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine.xml but file automaticaly restored after restart

Log:
2016-07-20 09:03:26,834 ERROR [io.undertow.request] (default task-25) UT005023: Exception handling request to /ovirt-engine/services/reports-interface-proxy: javax.servlet.ServletException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at org.ovirt.engine.core.uutils.servlet.ProxyServletBase.doGet(ProxyServletBase.java:163) [uutils.jar:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:]
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [undertow-core-1.3.15.Final.jar:1.3.15.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91]
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_91]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) [jsse.jar:1.8.0_91]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) [jsse.jar:1.8.0_91]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) [jsse.jar:1.8.0_91]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) [jsse.jar:1.8.0_91]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) [jsse.jar:1.8.0_91]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) [jsse.jar:1.8.0_91]
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) [jsse.jar:1.8.0_91]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) [jsse.jar:1.8.0_91]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) [jsse.jar:1.8.0_91]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) [jsse.jar:1.8.0_91]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) [jsse.jar:1.8.0_91]
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_91]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_91]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) [rt.jar:1.8.0_91]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) [rt.jar:1.8.0_91]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) [rt.jar:1.8.0_91]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) [rt.jar:1.8.0_91]
        at org.ovirt.engine.core.uutils.servlet.ProxyServletBase.doGet(ProxyServletBase.java:146) [uutils.jar:]
        ... 34 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) [rt.jar:1.8.0_91]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) [rt.jar:1.8.0_91]
        at sun.security.validator.Validator.validate(Validator.java:260) [rt.jar:1.8.0_91]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_91]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_91]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_91]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) [jsse.jar:1.8.0_91]
        ... 48 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_91]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_91]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_91]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) [rt.jar:1.8.0_91]
        ... 54 more

Comment 7 Martin Perina 2016-07-20 07:47:23 UTC
Could you please open a new bug for that? It seems to me as a bug, but in different part of engine code.

Comment 8 Konstantin 2016-07-20 08:20:21 UTC
Thank you. Open new bug https://bugzilla.redhat.com/show_bug.cgi?id=1358160

Comment 9 Jiri Belka 2016-07-26 13:34:26 UTC
FYI, update-ca-trust did not add my own CA into java truststore on RHEL6 with 3.6 RHEV env, although it added it into /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt.

I did this workaround after update-ca-trust, I hope it will be OK:

keytool -keystore /etc/pki/java/cacerts -importcert -alias foobar -file /etc/pki/ca-trust/source/anchors/CA.crt

# openssl x509 -in /etc/pki/ca-trust/source/anchors/CA.crt -fingerprint -sha1 -noout
SHA1 Fingerprint=1A:C5:CD:72:22:2D:91:44:FB:82:12:BE:72:E9:08:EE:21:1D:D6:30

# keytool -list -keystore /etc/pki/java/cacerts -storepass $passwd | grep $MY_CA_FINGERPRINT
Certificate fingerprint (SHA1): 1A:C5:CD:72:22:2D:91:44:FB:82:12:BE:72:E9:08:EE:21:1D:D6:30

Comment 10 Martin Perina 2016-07-26 13:38:04 UTC
(In reply to Jiri Belka from comment #9)
> FYI, update-ca-trust did not add my own CA into java truststore on RHEL6
> with 3.6 RHEV env, although it added it into
> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt.

This needs to be done on EL7 host after migration, it has no effect to do that on EL6 with RHEV 3.6

Comment 11 Jiri Belka 2016-07-26 14:31:03 UTC
ok, ovirt-engine-4.0.2-0.1.rc.el7ev.noarch

verification based on doctext (i migrated from 3.6)

Comment 12 Jeff Warnica 2016-10-20 13:11:30 UTC
*** Bug 1385062 has been marked as a duplicate of this bug. ***

Comment 13 Nicolas Ecarnot 2016-10-27 14:17:38 UTC
Hello,

TL; DR

Workaround applied, enrolling certificates failing.


Please read http://lists.ovirt.org/pipermail/users/2016-October/043673.html .
I have applied the workaround described above, and everything seemed working (start web page GUI).
But when trying to enroll cert on a host, it is still failing with :
"CA certificate and CA private key do not match".

So far, I can not add any new host to my DC, which situation I consider quite serious.

Do you agree to say this is a severe issue?

Comment 14 Oved Ourfali 2016-10-27 14:25:21 UTC
What version are you working with?

Comment 15 Logan Kuhn 2016-10-31 18:13:18 UTC
I've been able to duplicate this exact situation and, in my case, part user error and part documentation.

When I did this work I moved ca.pem out of the way so the log showed file not found.  I then created a sym link from the new custom cert to ca.pem and did the same for private/ca.pem and was able to enroll hosts without issue.

My guess (correct me if I'm wrong) is that Nicolas is stuck at the second portion.

Comment 16 Martin Perina 2016-11-01 10:55:38 UTC
We don't support replacing oVirt internal CA for engine - hosts communication. We support only usage of HTTPS certificate signed by custom CA since oVirt 3.x and this bug described only additional steps needed for 4.0 to work properly.

I've replied to original email thread with detailed steps how to replace HTTPS certificate in http://lists.ovirt.org/pipermail/users/2016-November/043761.html

For you situation I can suggest you to restore oVirt internal CA you have erroneously replaced, check if everything works as expected and afterwards replaces only HTTPS certificate as supported.

Comment 17 Nicolas Ecarnot 2016-12-05 09:43:58 UTC
(In reply to Oved Ourfali from comment #14)
> What version are you working with?

4.0.5.5-1.el7.centos

Two days ago, I ran into the same try, with this new version.
As previously, all the web access went OK, but I'm still stuck at the cert enrollment issue, so new hosts can not join.

I did not understood what was the workaround provided by Logan Kuhn, so if anyone may describe it, you would make my day.

Comment 18 Martin Perina 2016-12-05 09:51:16 UTC
(In reply to Nicolas Ecarnot from comment #17)
> (In reply to Oved Ourfali from comment #14)
> > What version are you working with?
> 
> 4.0.5.5-1.el7.centos
> 
> Two days ago, I ran into the same try, with this new version.
> As previously, all the web access went OK, but I'm still stuck at the cert
> enrollment issue, so new hosts can not join.
> 
> I did not understood what was the workaround provided by Logan Kuhn, so if
> anyone may describe it, you would make my day.

I'm not aware of any workaround, we simply do not support replacing internal oVirt CA with custom CA to sign certificates for engine <-> VDSM communication. The only way how to fix the error of unable to add new host due to certificate issue  is to restore oVirt internal CA from the backup before actions that replaced original oVirt CA with custom one.

We support replacing HTTPS certificate signed by custom CA and steps describing this configuration are written in the Doc Text of this bug.

Comment 19 Logan Kuhn 2016-12-05 12:49:28 UTC
To be clear since I was referenced.  What I did was in an effort to provide more troubleshooting information for this bug.  

While I had it in that state the host that I made those changes on complained about that certificate at seemingly random times and it was only in that state for a few hours.


Note You need to log in before you can comment on or make changes to this bug.