Description of problem: engine doesn't trust externally-issued web certificate for internal authentication in spite of issuer being in system (and java) trust store Version-Release number of selected component (if applicable): ovirt-engine-4.0.0-0.6.master.el7ev.noarch ovirt-engine-backend-4.0.0-0.6.master.el7ev.noarch How reproducible: always Steps to Reproduce: 1. get certificate for mod_ssl from external CA 2. add the external CA that signed the mod_ssl certificate to the trust store (trust add /path/to/CA.crt or cp /path/to/CA.crt /etc/pki/ca-trust/source/anchors/ && update-ca-trust) 3. verify that the CA appears in java trust store, e.g.: keytool -list -keystore /usr/lib/jvm/jre/lib/security/cacerts \ | grep "$(openssl x509 -noout -fingerprint -sha1 -in \ /etc/pki/ca-trust/source/anchors/CA.crt \ | sed -e 's/SHA1 Fingerprint=//')" 4. run engine-setup with Manual SSL (OVESETUP_APACHE/configureSsl=bool:False) 5. try to log in as admin Actual results: admin logs in but an error occurs and users are redirected back: > 2016-05-17 15:54:51,487 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-27) [] User admin@internal successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access > 2016-05-17 15:54:51,510 ERROR [org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-11) [] server_error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Expected results: admin@internal logs in just fine after the setup Additional info:
Making /etc/pki/ovirt-engine/apache-ca.pem symlink point to web server CA doesn't help.
> ... a new conf file needs to be created setting the new variables added > to point to the system trust store's location and password. Well, the point of system trust store is well known location and that you can depend on it being there and not bother admins looking up where it exports certificates for java consumption - it's file linked by default java store! # readlink -f /usr/lib/jvm/jre/lib/security/cacerts /etc/pki/ca-trust/extracted/java/cacerts I'd understand knob to actually override this system store by custom store (or just by server certificate fingerprint) so that SSO couldn't be provided by e.g. rogue site signed by legitimate CA (analogous to LDAP server verification in AAA) - but that's not what you describe here.
Works when using key store with password. When using system keystore that doesn't use password with no password (setting empty or none ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD variable).
... the authentication won't work. This is the case of system trust store at /etc/pki/java/cacerts - I'm not sure however if this is now bug at engine or java or ca-certificates.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
How about /ovirt-engine/services/reports-interface-proxy error when open dashboard? I try to edit keystore parameter in /var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine.xml but file automaticaly restored after restart Log: 2016-07-20 09:03:26,834 ERROR [io.undertow.request] (default task-25) UT005023: Exception handling request to /ovirt-engine/services/reports-interface-proxy: javax.servlet.ServletException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at org.ovirt.engine.core.uutils.servlet.ProxyServletBase.doGet(ProxyServletBase.java:163) [uutils.jar:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [undertow-servlet-1.3.15.Final.jar:1.3.15.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [undertow-core-1.3.15.Final.jar:1.3.15.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91] Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_91] at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) [jsse.jar:1.8.0_91] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) [jsse.jar:1.8.0_91] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) [jsse.jar:1.8.0_91] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) [jsse.jar:1.8.0_91] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) [jsse.jar:1.8.0_91] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) [jsse.jar:1.8.0_91] at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) [jsse.jar:1.8.0_91] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) [jsse.jar:1.8.0_91] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) [jsse.jar:1.8.0_91] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) [jsse.jar:1.8.0_91] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) [jsse.jar:1.8.0_91] at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_91] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_91] at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) [rt.jar:1.8.0_91] at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) [rt.jar:1.8.0_91] at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) [rt.jar:1.8.0_91] at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) [rt.jar:1.8.0_91] at org.ovirt.engine.core.uutils.servlet.ProxyServletBase.doGet(ProxyServletBase.java:146) [uutils.jar:] ... 34 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) [rt.jar:1.8.0_91] at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) [rt.jar:1.8.0_91] at sun.security.validator.Validator.validate(Validator.java:260) [rt.jar:1.8.0_91] at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_91] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_91] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_91] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) [jsse.jar:1.8.0_91] ... 48 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_91] at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_91] at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_91] at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) [rt.jar:1.8.0_91] ... 54 more
Could you please open a new bug for that? It seems to me as a bug, but in different part of engine code.
Thank you. Open new bug https://bugzilla.redhat.com/show_bug.cgi?id=1358160
FYI, update-ca-trust did not add my own CA into java truststore on RHEL6 with 3.6 RHEV env, although it added it into /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt. I did this workaround after update-ca-trust, I hope it will be OK: keytool -keystore /etc/pki/java/cacerts -importcert -alias foobar -file /etc/pki/ca-trust/source/anchors/CA.crt # openssl x509 -in /etc/pki/ca-trust/source/anchors/CA.crt -fingerprint -sha1 -noout SHA1 Fingerprint=1A:C5:CD:72:22:2D:91:44:FB:82:12:BE:72:E9:08:EE:21:1D:D6:30 # keytool -list -keystore /etc/pki/java/cacerts -storepass $passwd | grep $MY_CA_FINGERPRINT Certificate fingerprint (SHA1): 1A:C5:CD:72:22:2D:91:44:FB:82:12:BE:72:E9:08:EE:21:1D:D6:30
(In reply to Jiri Belka from comment #9) > FYI, update-ca-trust did not add my own CA into java truststore on RHEL6 > with 3.6 RHEV env, although it added it into > /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt. This needs to be done on EL7 host after migration, it has no effect to do that on EL6 with RHEV 3.6
ok, ovirt-engine-4.0.2-0.1.rc.el7ev.noarch verification based on doctext (i migrated from 3.6)
*** Bug 1385062 has been marked as a duplicate of this bug. ***
Hello, TL; DR Workaround applied, enrolling certificates failing. Please read http://lists.ovirt.org/pipermail/users/2016-October/043673.html . I have applied the workaround described above, and everything seemed working (start web page GUI). But when trying to enroll cert on a host, it is still failing with : "CA certificate and CA private key do not match". So far, I can not add any new host to my DC, which situation I consider quite serious. Do you agree to say this is a severe issue?
What version are you working with?
I've been able to duplicate this exact situation and, in my case, part user error and part documentation. When I did this work I moved ca.pem out of the way so the log showed file not found. I then created a sym link from the new custom cert to ca.pem and did the same for private/ca.pem and was able to enroll hosts without issue. My guess (correct me if I'm wrong) is that Nicolas is stuck at the second portion.
We don't support replacing oVirt internal CA for engine - hosts communication. We support only usage of HTTPS certificate signed by custom CA since oVirt 3.x and this bug described only additional steps needed for 4.0 to work properly. I've replied to original email thread with detailed steps how to replace HTTPS certificate in http://lists.ovirt.org/pipermail/users/2016-November/043761.html For you situation I can suggest you to restore oVirt internal CA you have erroneously replaced, check if everything works as expected and afterwards replaces only HTTPS certificate as supported.
(In reply to Oved Ourfali from comment #14) > What version are you working with? 4.0.5.5-1.el7.centos Two days ago, I ran into the same try, with this new version. As previously, all the web access went OK, but I'm still stuck at the cert enrollment issue, so new hosts can not join. I did not understood what was the workaround provided by Logan Kuhn, so if anyone may describe it, you would make my day.
(In reply to Nicolas Ecarnot from comment #17) > (In reply to Oved Ourfali from comment #14) > > What version are you working with? > > 4.0.5.5-1.el7.centos > > Two days ago, I ran into the same try, with this new version. > As previously, all the web access went OK, but I'm still stuck at the cert > enrollment issue, so new hosts can not join. > > I did not understood what was the workaround provided by Logan Kuhn, so if > anyone may describe it, you would make my day. I'm not aware of any workaround, we simply do not support replacing internal oVirt CA with custom CA to sign certificates for engine <-> VDSM communication. The only way how to fix the error of unable to add new host due to certificate issue is to restore oVirt internal CA from the backup before actions that replaced original oVirt CA with custom one. We support replacing HTTPS certificate signed by custom CA and steps describing this configuration are written in the Doc Text of this bug.
To be clear since I was referenced. What I did was in an effort to provide more troubleshooting information for this bug. While I had it in that state the host that I made those changes on complained about that certificate at seemingly random times and it was only in that state for a few hours.