Description of problem: Cannot log into jenkins console which is setup with jenkins-2-rhel7 image with below error: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Version-Release number of selected component (if applicable): Online dev-preview-stg OpenShift Master:v3.4.0.39 (online version 3.4.0.13) Kubernetes Master:v1.4.0+776c994 How reproducible: always Steps to Reproduce: 1.$ oc new-project test 2. $oc policy add-role-to-user admin system:serviceaccount:test:default -n test 3. $oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/jenkins-ephemeral-template.json 4. $oc get route NAME HOST/PORT PATH SERVICE LABELS INSECURE POLICY TLS TERMINATION jenkins jenkins-test.router.default.svc.cluster.local jenkins template=jenkins-ephemeral-template 5. Try to log into jenkins console with openshift account Actual results: Cannot login successfully with errors in additional info. Expected results: Should login Additional info: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77) at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972) at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:283) at com.google.api.client.auth.openidconnect.IdTokenResponse.execute(IdTokenResponse.java:120) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm$6.onSuccess(OpenShiftOAuth2SecurityRealm.java:567) at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:114) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doFinishLogin(OpenShiftOAuth2SecurityRealm.java:719) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:196) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:100) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ... 89 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ... 95 more
Here is jenkins image id: registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:79ad4b35f8764f487ae84ee63ea06b301eb10b2d8999f91557419dc5d75b0698
This is a testblocker for our test against Jenkins in online STG 3.4.0.39
(Sounds like maybe STG is using a bad/invalid cert for the api server? At least, one the jenkins login plugin can't validate)
Yeah Ben - as you speculated, this is really outside the control of the login plugin. Something is out of sync on the api server side wrt the cert, at least based on my level of understanding. The key point in the stack trace is here: at com.google.api.client.auth.openidconnect.IdTokenResponse.execute(IdTokenResponse.java:120) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm$6.onSuccess(OpenShiftOAuth2SecurityRealm.java:567) at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:114) At this point in the oauth redirect ping pong, we have already initiated a AuthorizationCodeFlow to oauth on the master using among other things the cert mounted in the pod, and they have responded to us. The login plugin is now taking the next step and hitting the api server / oauth provider again to get a IdTokenResponse (i.e. the final access token) using the AuthorizationCode we just received. If all is well the api server will redirect the browser back to jenkins, but in this case, the SSL validation obviously fails. I've added Mo and Jordan in case they have something to add. But pending any clarifications from them, I think this bug should be redirected to investigate this ssl cert / api server / oauth mismatch.
Perhaps if the QE contanct could reproduce with Chrome, where they go to "View Developer Tools", then the "network" tab, and then check "preserve log", and then hit the login url, we could infer something important from that trace.
Chris's info in #Comment 6 is very useful information. The same SSL validation error is also seen from the jenkins-plugin on a direct https access to the api server using the cert mounted in the pod. That eliminates oauth and a relevant player in this. I'll see what if anything I can infer from the chrome output Chris gathered, but most likely I'm going to send this to the online team for analysis of the SSL setup (either in general for the api server, or wrt the jenkins template used) in the online environment in question. Miciah - unless you say differently, I'll probably send it over to you (given your involvement in https://bugzilla.redhat.com/show_bug.cgi?id=1399022).
I still see no SSL specifics in the attachment. We are definitely beyond my expertise wrt debugging SSL issues, but I think we've confirmed per Ben's original speculation that this is not a jenkins image or plugin issue. But rather, some inconsistency wrt SSL setup, either for the api master in general or the jenkins template in particular, and with whatever cert is getting mounted into the pod, in this particular online env. Per my earlier comment, and pending any further insight from Mo or others, sending over to Miciah, given his existing involvement in the online setup issues in https://bugzilla.redhat.com/show_bug.cgi?id=1399022.
A bit of a shot in the dark, but one more piece occurred to me ... I wonder if online is running with an older version of the template that is missing https://github.com/openshift/origin/commit/4232ddf19329b042d25ebbe1520b31b47bc89fef
No, the template has been updated in dev-preview-int and dev-preview-stg, and so the jenkins-persistent template's route no longer has the bogus certificate.
The new jenkins image shipped with 3.4 resolved this issue.
Jenkins web console can be successfully login with the new image: jenkins-2-rhel7@sha256:bcdf1028271ac7131863f7e1f44c5627cb4da1eeb8444a4f778a11e43b57305a
Per comment #18, this bug can be verified. Thanks, yasun!