Description of problem: When using custom Ingress router certificate[1] along with custom PKI[2], all routes work as expected, including Jenkins with oAuth option enabled (jenkins-<namespace>.apps.<clusterrid>.example.net). However, if the custom Ingress certificate is signed by an intermediate CA, then, the jenkins pod and the URL both fail with the following error: ~~~ javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1362) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1337) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:108) at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:79) at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:996) at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:322) at com.google.api.client.auth.openidconnect.IdTokenResponse.execute(IdTokenResponse.java:120) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm$11.onSuccess(OpenShiftOAuth2SecurityRealm.java:948) at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:129) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doFinishLogin(OpenShiftOAuth2SecurityRealm.java:1198) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:247) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:629) ... 114 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 120 more ~~~ [1] - https://docs.openshift.com/container-platform/4.3/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress [2] - https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#configuring-a-custom-pki Version-Release number of selected component (if applicable): OCP 4.3.1 jenkins-ephemeral (oAuth enabled) How reproducible: Always (when using intermediate certs) Steps to Reproduce: 1. Create RootCA + Intermediate + Custom Ingress certificate "*.apps.<clusterid>.example.net" 2. Configure the CA as the cluster proxy CA: $ oc -n openshift-config create configmap custom-ca --from-file=ca-bundle.crt=example-ca.crt $ oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}' 3. Configure the certificate as the ingresscontroller's default certificate: $ oc -n openshift-ingress create secret tls custom-default-cert --cert=example_fullchain.crt --key=example.key $ oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"defaultCertificate":{"name":"custom-default-cert"}}}' Actual results: PFA the jenkins pod logs Expected results: A valid oAuth authentication like the rest of routes within OCP 4.3.x. Additional info: - On the test done, the ingress certificate already contains the full chain including endpoint + intermediate + rootCA, and the custom PKI (trustedCAs cluster-wide proxy) also contains both CAs in the correct order.
Hello Pedro, Currently we are trying to reproduce this and are not getting the exact SSLHandshakeException, would it be possible for you to share steps for the creation of the key and certificate files ? We are getting a javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching oauth-openshift.apps.vbobade-230220.devcluster.openshift.com found at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ... Whereas above you have shown that you are getting a javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) Regards, Vibhav
Hi @Vibhav, thanks for checking on this, here you have all details about the creation of the full certificate chain: 0) Pre-steps to hold the entire custom CA structure: ~~~ [RootCA] $ cd /path/to/safe/storage/customCA $ mkdir -p example.net.ca/root-ca/{certreqs,certs,crl,newcerts,private} $ cd example.net.ca/root-ca $ chmod 700 private $ touch root-ca.index $ echo 00 > root-ca.crlnum $ openssl rand -hex 16 > root-ca.serial [IntermediateCA] $ cd /path/to/safe/storage/customCA $ mkdir -p example.net.ca/intermed-ca/{certreqs,certs,crl,newcerts,private} $ cd example.net.ca/intermed-ca $ chmod 700 private $ touch intermed-ca.index $ echo 00 > intermed-ca.crlnum $ openssl rand -hex 16 > intermed-ca.serial ~~~ 1) Create RootCA config file as follows: ~~~ $ cd example.net.ca/root-ca $ cat root-ca.cnf # # OpenSSL configuration for the Root Certification Authority. # # # This definition doesn't work if HOME isn't defined. CA_HOME = . RANDFILE = $ENV::CA_HOME/private/.rnd # # Default Certification Authority [ ca ] default_ca = root_ca # # Root Certification Authority [ root_ca ] dir = $ENV::CA_HOME certs = $dir/certs serial = $dir/root-ca.serial database = $dir/root-ca.index new_certs_dir = $dir/newcerts certificate = $dir/root-ca.cert.pem private_key = $dir/private/root-ca.key.pem default_days = 1826 # Five years crl = $dir/root-ca.crl crl_dir = $dir/crl crlnumber = $dir/root-ca.crlnum name_opt = multiline, align cert_opt = no_pubkey copy_extensions = copy crl_extensions = crl_ext default_crl_days = 180 default_md = sha256 preserve = no email_in_dn = no policy = policy unique_subject = no # # Distinguished Name Policy for CAs [ policy ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = supplied organizationalUnitName = optional commonName = supplied # # Root CA Request Options [ req ] default_bits = 4096 default_keyfile = private/root-ca.key.pem encrypt_key = yes default_md = sha256 string_mask = utf8only utf8 = yes prompt = no req_extensions = root-ca_req_ext distinguished_name = distinguished_name subjectAltName = @subject_alt_name # # Root CA Request Extensions [ root-ca_req_ext ] subjectKeyIdentifier = hash subjectAltName = @subject_alt_name # # Distinguished Name (DN) [ distinguished_name ] organizationName = example.net commonName = example.net Root Certification Authority # # Root CA Certificate Extensions [ root-ca_ext ] basicConstraints = critical, CA:true keyUsage = critical, keyCertSign, cRLSign nameConstraints = critical, @name_constraints subjectKeyIdentifier = hash subjectAltName = @subject_alt_name authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy authorityInfoAccess = @auth_info_access crlDistributionPoints = crl_dist # # Intermediate CA Certificate Extensions [ intermed-ca_ext ] basicConstraints = critical, CA:true, pathlen:0 keyUsage = critical, keyCertSign, cRLSign subjectKeyIdentifier = hash subjectAltName = @subject_alt_name authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy authorityInfoAccess = @auth_info_access crlDistributionPoints = crl_dist # # CRL Certificate Extensions [ crl_ext ] authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy # # Certificate Authorities Alternative Names [ subject_alt_name ] URI = http://ca.example.net/ email = certmaster # # Name Constraints [ name_constraints ] permitted;DNS.1 = example.net permitted;email.1 = example.net # # Certificate download addresses for the root CA [ auth_info_access ] caIssuers;URI = http://ca.example.net/certs/example.net_Root_Certification_Authority.cert.pem # # CRL Download address for the root CA [ crl_dist ] fullname = URI:http://ca.example.net/crl/example.net_Root_Certification_Authority.crl # EOF ~~~ 2) Make sure the config is active: ~~~ $ export OPENSSL_CONF=./root-ca.cnf ~~~ 3) Generate CSR & Key: ~~~ $ openssl req -new -out root-ca.req.pem $ chmod 400 private/root-ca.key.pem ~~~ 4) Self-sign the Root certificate: ~~~ $ openssl rand -hex 16 > root-ca.serial $ openssl ca -selfsign -in root-ca.req.pem -out root-ca.cert.pem -extensions root-ca_ext -startdate `date +%y%m%d000000Z -u -d -1day` -enddate `date +%y%m%d000000Z -u -d +10years+1day` ~~~ 5) (Optional) Verify the cert: ~~~ $ openssl x509 -in ./root-ca.cert.pem -noout -text -certopt no_version,no_pubkey,no_sigdump -nameopt multiline $ openssl verify -verbose -CAfile root-ca.cert.pem root-ca.cert.pem ~~~ 6) Create IntermediateCA config file as follows: ~~~ $ cd example.net.ca/intermed-ca $ cat intermed-ca.cnf # # OpenSSL configuration for the Intermediate Certification Authority. # # # This definition doesn't work if HOME isn't defined. CA_HOME = . RANDFILE = $ENV::CA_HOME/private/.rnd oid_section = new_oids # # XMPP address Support [ new_oids ] xmppAddr = 1.3.6.1.5.5.7.8.5 dnsSRV = 1.3.6.1.5.5.7.8.7 # # Default Certification Authority [ ca ] default_ca = intermed_ca # # Intermediate Certification Authority [ intermed_ca ] dir = $ENV::CA_HOME certs = $dir/certs serial = $dir/intermed-ca.serial database = $dir/intermed-ca.index new_certs_dir = $dir/newcerts certificate = $dir/intermed-ca.cert.pem private_key = $dir/private/intermed-ca.key.pem default_days = 730 # Two years crl = $dir/crl/intermed-ca.crl crl_dir = $dir/crl crlnumber = $dir/intermed-ca.crlnum name_opt = multiline, align cert_opt = no_pubkey copy_extensions = copy crl_extensions = crl_ext default_crl_days = 30 default_md = sha256 preserve = no email_in_dn = no policy = policy unique_subject = no # # Distinguished Name Policy [ policy ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = optional organizationalUnitName = optional commonName = supplied # # Distinguished Name Policy for Personal Certificates [ user_policy ] countryName = supplied stateOrProvinceName = optional localityName = supplied organizationName = optional organizationalUnitName = optional commonName = supplied emailAddress = supplied #xmppAddr = optional # Added to SubjAltName by req # # Intermediate CA request options [ req ] default_bits = 3072 default_keyfile = private/intermed-ca.key.pem encrypt_key = yes default_md = sha256 string_mask = utf8only utf8 = yes prompt = no req_extensions = req_ext distinguished_name = distinguished_name subjectAltName = subject_alt_name # # Intermediate CA Request Extensions [ req_ext ] subjectKeyIdentifier = hash subjectAltName = @subject_alt_name # # Distinguished Name (DN) [ distinguished_name ] organizationName = example.net commonName = example.net Intermediate Certification Authority # # Server Certificate Extensions [ server_ext ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = critical, serverAuth, clientAuth subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy authorityInfoAccess = @auth_info_access crlDistributionPoints = crl_dist # # Client Certificate Extensions [ client_ext ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature extendedKeyUsage = critical, clientAuth subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy authorityInfoAccess = @auth_info_access crlDistributionPoints = crl_dist # # User Certificate Extensions [ user_ext ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature extendedKeyUsage = critical, clientAuth, emailProtection subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy authorityInfoAccess = @auth_info_access crlDistributionPoints = crl_dist # # CRL Certificate Extensions [ crl_ext ] authorityKeyIdentifier = keyid:always issuerAltName = issuer:copy # # Certificate Authorities Alternative Names [ subject_alt_name ] URI = http://ca.example.net/ email = certmaster # # Certificate download addresses for the intermediate CA [ auth_info_access ] caIssuers;URI = http://ca.example.net/certs/example.net_Intermediate_Certification_Authority.cert.pem # # CRL Download address for the intermediate CA [ crl_dist ] fullname = URI:http://ca.example.net/crl/example.net_Intermediate_Certification_Authority.crl # EOF ~~~ 7) Switch to the proper config: ~~~ $ export OPENSSL_CONF=./intermed-ca.cnf ~~~ 8) Generate CSR & Key: ~~~ $ openssl req -new -out intermed-ca.req.pem $ chmod 400 private/intermed-ca.key.pem ~~~ 9) Sign the IntermediateCA with the RootCA: ~~~ $ cp intermed-ca.req.pem /path/to/safe/storage/customCA/example.net.ca/root-ca/certreqs/ $ cd /path/to/safe/storage/customCA/example.net.ca/root-ca/ $ export OPENSSL_CONF=./root-ca.cnf $ openssl rand -hex 16 > root-ca.serial $ openssl ca -in certreqs/intermed-ca.req.pem -out certs/intermed-ca.cert.pem -extensions intermed-ca_ext -startdate `date +%y%m%d000000Z -u -d -1day` -enddate `date +%y%m%d000000Z -u -d +5years+1day` $ cp certs/intermed-ca.cert.pem /path/to/safe/storage/customCA/example.net.ca/intermed-ca/ ~~~ 10) (Optional) Verify the cert: ~~~ $ openssl x509 -in certs/intermed-ca.cert.pem -noout -text -certopt no_version,no_pubkey,no_sigdump -nameopt multiline $ openssl verify -verbose -CAfile root-ca.cert.pem certs/intermed-ca.cert.pem ~~~ 11) Create the custom Ingress wildcard certificate for our cluster: ~~~ $ cd /path/to/safe/storage/customCA/example.net.ca/intermed-ca $ export OPENSSL_CONF=./intermed-ca.cnf $ INGRESS_DOMAIN="$(oc get ingress.config/cluster -o 'jsonpath={.spec.domain}')" $ openssl genrsa -out example.key 2048 $ openssl req -new -key example.key -out example.csr -subj "/C=US/ST=NC/L=Raleigh/O=OCP4/OU=IT/CN=*.$INGRESS_DOMAIN" $ mv example.csr ./certreqs/ ~~~ 12) Sign the Server Cert with the IntermediateCA: ~~~ $ openssl rand -hex 16 > intermed-ca.serial $ openssl ca -in ./certreqs/example.csr -out ./certs/example.pem -extensions server_ext ~~~ NOTE: Previous steps can be simplified but I wanted to replicate a real Corporate CA scenario (like the CU) as better as possible. Best Regards.
Hi again @Vibhav, FWIW, using only a rootCA on the following manner (without IntermediateCA), the Jenkins oAuth works as expected: 1. Generate a CA and certificate: BASE_DOMAIN="$(oc get dns.config/cluster -o 'jsonpath={.spec.baseDomain}')" INGRESS_DOMAIN="$(oc get ingress.config/cluster -o 'jsonpath={.spec.domain}')" openssl genrsa -out example-ca.key 2048 openssl req -x509 -new -key example-ca.key -out example-ca.crt -days 730 -subj "/C=US/ST=NC/L=Raleigh/O=OCP4/OU=IT/CN=$BASE_DOMAIN" openssl genrsa -out example.key 2048 openssl req -new -key example.key -out example.csr -subj "/C=US/ST=NC/L=Raleigh/O=OCP4/OU=IT/CN=*.$INGRESS_DOMAIN" openssl x509 -req -in example.csr -CA example-ca.crt -CAkey example-ca.key -CAcreateserial -out example.crt -days 365 2. Configure the CA as the cluster proxy CA: oc -n openshift-config create configmap custom-ca --from-file=ca-bundle.crt=example-ca.crt oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}' 3. Configure the certificate as the ingresscontroller's default certificate: oc -n openshift-ingress create secret tls custom-default-cert --cert=example.crt --key=example.key oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"defaultCertificate":{"name":"custom-default-cert"}}}' Regards.
Hello @Pedro, Thank you for the detailed explanation and explanation on how the CU might do it, it's very helpful to understand. I was able to reproduce this. Will work on a fix on this ASAP. Regards,
Hello Pedro, Currently Jenkins does not support customCA explicitly, but to make it easier I am working on a feature for the Login Plugin it should make things easier. Please consider the blow as a workaround. The new feature should be able to add the certificate directly to the default JVM TrustStore. 1> Get the default keyStore. oc rsync jenkins-1-8zbx2:/etc/pki/java ./custom-java 2> Add certificate to keystore sudo keytool -keystore ./custom-java/cacerts -import -alias custom-ingress -file ./example.crt 3> Create a ConfigMap from the custom keystore oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts 4> Edit DeploymentConfig for using the custom Keystore with the following changes. spec: template: spec: containers: - env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts" volumeMounts: - mountPath: /etc/pki/java/cacerts name: jenkins-custom-keystore volumes: - name: jenkins-custom-keystore configMap: name: jenkins-custom-keystore Hope this helps, Regards
Hi @Vibhav, thanks for the workaround, I've tried it on my lab but the pod is not able to start and shows the following problem: ~~~ 13s Warning Failed pod/jenkins-2-pwdqg Error: container create failed: time="2020-02-28T08:48:00Z" level=warning msg="exit status 1" time="2020-02-28T08:48:00Z" level=error msg="container_linux.go:346: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/var/lib/kubelet/pods/d0038ae5-9471-42f0-bc70-7872e76572da/volumes/kubernetes.io~configmap/jenkins-custom-keystore\\\\\\\" to rootfs \\\\\\\"/var/lib/containers/storage/overlay/28abf7a1402f9888aea94b233fb75d33c3435cebdcd532aaa01c147d036b2cff/merged\\\\\\\" at \\\\\\\"/var/lib/containers/storage/overlay/28abf7a1402f9888aea94b233fb75d33c3435cebdcd532aaa01c147d036b2cff/merged/etc/pki/ca-trust/extracted/java/cacerts\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"" container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/d0038ae5-9471-42f0-bc70-7872e76572da/volumes/kubernetes.io~configmap/jenkins-custom-keystore\\\" to rootfs \\\"/var/lib/containers/storage/overlay/28abf7a1402f9888aea94b233fb75d33c3435cebdcd532aaa01c147d036b2cff/merged\\\" at \\\"/var/lib/containers/storage/overlay/28abf7a1402f9888aea94b233fb75d33c3435cebdcd532aaa01c147d036b2cff/merged/etc/pki/ca-trust/extracted/java/cacerts\\\" caused \\\"not a directory\\\"\"" ~~~ However, your workaround gave me an idea and I have done the following instead: 1) Extract the keystore: ~~~ $ mkdir custom-ca $ oc rsync jenkins-<pod-id>:/etc/pki/ca-trust/extracted/java/cacerts ./custom-java/ ~~~ 2) Concatenate RootCA + IntermediateCA and import them into the keystore as "trustcacerts" (I'm not importing the endpoint cert, just the proper CAs): ~~~ $ cat /path/to/intermed-ca.cert.pem /path/to/root-ca.cert.pem > ./custom-java/rootCA.pem $ sudo keytool -import -trustcacerts -noprompt -keystore ./custom-java/cacerts -storepass changeit -alias custom-ingress -file ./custom-java/rootCA.pem ~~~ 3) Create the configmap: ~~~ $ oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts ~~~ 4) Edit the deploymentconfig but including only the mount volume part to load the new configmap into the proper "/etc/pki/ca-trust/extracted/java/", which is the real path where the cacert is stored: ~~~ $ oc edit deploymentconfig.apps.openshift.io/jenkins spec: containers: - env: - name: OPENSHIFT_ENABLE_OAUTH value: "true" [...] volumeMounts: - mountPath: /etc/pki/ca-trust/extracted/java/ name: jenkins-custom-keystore dnsPolicy: ClusterFirst restartPolicy: Always [...] volumes: - configMap: defaultMode: 420 name: jenkins-custom-keystore name: jenkins-custom-keystore ~~~ With those steps, the new pod properly starts and loads the correct keystore, I was able to successfully login into the Jenkins oAuth on this manner :) ~~~ [...] 2020-02-28 09:51:45.049+0000 [id=21] INFO o.o.j.p.o.OpenShiftSetOAuth#setOauth: OpenShift OAuth: enable oauth set to true force true lastCheck Thu Jan 01 00:00:00 UTC 1970 2020-02-28 09:51:45.049+0000 [id=21] INFO o.o.j.p.o.OpenShiftSetOAuth#setOauth: OpenShift OAuth: configured security realm on startup: hudson.security.HudsonPrivateSecurityRealm@7992d4cc last check Thu Jan 01 00:00:00 UTC 1970 2020-02-28 09:51:47.538+0000 [id=21] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#populateDefaults: OpenShift OAuth: provider: OpenShiftProviderInfo: issuer: https://oauth-openshift.apps.<clusterid>.<domain> auth ep: https://oauth-openshift.apps.<clusterid>.<domain>/oauth/authorize token ep: https://oauth-openshift.apps.<clusterid>.<domain>/oauth/token 2020-02-28 09:51:47.558+0000 [id=21] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#useProviderOAuthEndpoint: OpenShift OAuth server is 4.x, specifically OpenShiftVersionInfo: major: 1 minor: 16+ gitVersion: v1.16.2 2020-02-28 09:51:47.559+0000 [id=21] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#initializeHttpsProxyAuthenticator: Checking if HTTPS proxy initialization is required ... 2020-02-28 09:51:47.802+0000 [id=21] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#transportToUse: OpenShift OAuth got an SSL error when accessing the issuer's token endpoint when using the SA certificate 2020-02-28 09:51:48.207+0000 [id=21] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#transportToUse: OpenShift OAuth was able to complete the SSL handshake when accessing the issuer's token endpoint using the JVMs default keystore [...] ~~~ Can you please corroborate this workaround? Best Regards.
Hello @Pedro, This looks good. Please see if you can write a KCS for this, I will be working on a fix for this. Regards, Vibhav
(In reply to vbobade from comment #9) > Hello @Pedro, > > This looks good. Please see if you can write a KCS for this, I will be > working on a fix for this. > > > Regards, > Vibhav Sure @Vibhav, I'm waiting for CU confirmation in order to properly document the workaround, please keep me posted on the fix progress, thanks.
Hi @Vibhav, I have documented the workarouns, here you have the KCS[1], but please note the following feedback from the CU: ~~~ I can see that the workarround will work, but is not a solution for us. I don't want to use configmaps for all of our customers (200+ Jenkins instnances) then we have to change our template of let the customers make their configmaps in all the projects. So, i can see that it will work, but it is not a solution. What can work for me is that i can change the s2i assemble script that wil do an update of the Java keystore, but our build are not run as root, so that also will not work. ~~~ [1] - https://access.redhat.com/solutions/4879851 Best Regards.
Hi again @Vibhav, please note that our workaround[1] is not feasible for the CU due to their business logic (200+ Jenkins instances). The other possible workaround would be to modify the assemble script on the following manner: ~~~ [...] if [ -d ${JENKINS_INSTALL_DIR}/ca-cert ]; then echo "CA-Certificates found, copying them in /etc/pki/ca-trust/source/anchors/" cp ${JENKINS_INSTALL_DIR}/ca-cert/* /etc/pki/ca-trust/source/anchors/ update-ca-trust fi ~~~ But this requires root privileges during the assembly phase, which in turn requires some specific build label[2] called "io.openshift.s2i.assemble-user" and this is also not feasible because it would require a new Dockerfile. Any help is welcome, thanks. [1] - https://access.redhat.com/solutions/4879851 [2] - https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md#assemble
Hi @Vibhav, FWIW, I saw also BZ#1802297 with a similar issue for 4.2 where the ca-bundle was not being pushed by the kube-controller-manager, this was solved in 4.2.22[1] but seems that it was already present since 4.3.0 as per BZ#1782819, which makes sense because our error was not reproducible when only using RootCA + Custom certificate. I just wanted to mention it in case there is something you can extrapolate from that patch to solve also the IntermediateCA problem, because AFAIK, the full RootCA + IntermediateCA chain pushed by the custom PKI[2] should be stored in the same ca-bundle, right? [1] - https://access.redhat.com/errata/RHBA-2020:0685 [2] - https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#configuring-a-custom-pki Best Regards.
Setting this to 4.5.0 as target release to not be a blocker for 4.4.0 This needs a KCS for explaining the procedure
*** Bug 1805091 has been marked as a duplicate of this bug. ***
(In reply to Akram Ben Aissi from comment #14) > Setting this to 4.5.0 as target release to not be a blocker for 4.4.0 > > This needs a KCS for explaining the procedure Hi Akram, there is a KCS[1] already linked with the BZ, is this one not sufficient? can you please elaborate on what is missing? [1] - https://access.redhat.com/solutions/4879851 Best Regards.
Hi Pedro, ok that's perfect. I just saw that you worked with Vibhav to make it published. I missed the link. We can close it then.
(In reply to Akram Ben Aissi from comment #17) > Hi Pedro, > > ok that's perfect. I just saw that you worked with Vibhav to make it > published. > I missed the link. > > We can close it then. Apologies for the confusion Akram but the KCS is just for the workaround, which BTW was not valid for the customer due to the amount of Jenkins instances (200+), so we are still waiting for Vibhav's feedback about the pending bugfix. FWIW, they have found another workaround modifying the Dockerfile on the following manner: ~~~ Dockerfile FROM (openshift/jenkins:2) COPY ./contrib/ca-cert/*.pem /etc/pki/ca-trust/source/anchors/ RUN update-ca-trust ~~~ NOTE: This is temporarily alleviating the problem but the bugfix is still pending, so please do not close this BZ. Regards.
Hi all, any progress here? Best Regards.
Hello Pedro, We are still working on this fix. Currently it is in our backlog. We are moving it to our current items for work. Regards, Vibhav
Thanks for the confirmation Vibhav, please keep me posted if any progress. Best Regards.
we have pushed the PR to solve this issue. It is pending merge: https://github.com/openshift/jenkins/pull/1045
Thanks for the update @Akram, please note that we'll also need to backport it to 4.3 (if possible) when finally merged into master branch. Best Regards.
Verifed
*** Bug 1817436 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2871
I just upgraded from 4.4.11 to 4.4.12, but i still see the same ssl cert error. Was this supposed to be fixed in 4.4.12? -Vikash
Hi all, I'm reopening this BZ because we have confirmed that this bug is still present in 4.4.12, can you please help us determine what happened here? ~~~ javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1362) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1337) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) ... ~~~ FWIW, I can see that this bugfix was pushed 2 times for version 4.4 but the bug is still present, maybe some regression? BZ#1826174 (4.4.9) BZ#1804345 (4.4.12) Regards.
Hi all, please note the following comment from the linked solution[1]: ~~~ This isn't fully fixed yet - while 4.4.8 includes the fixed Jenkins images, that allow it to take advantage of trusted CA certificates injected by the network operator, the Jenkins templates shipped by the samples operator still don't create the ConfigMap into which the certificates can be injected. https://bugzilla.redhat.com/show_bug.cgi?id=1852997 ~~~ [1] - https://access.redhat.com/solutions/4879851#comment-1864681