Description of problem: Cannot login to Jenkins after configuring custom Router base domain wildcard cert. Customer has configured the custom PKI via documentation but still has issues: https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki The following error is seen in Jenkins logs after attempting login: Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm populateDefaults INFO: OpenShift OAuth: provider: OpenShiftProviderInfo: issuer: https://oauth-openshift.apps.prod.openshift.example.com auth ep: https://oauth-openshift.apps.prod.openshift.example.com/oauth /authorize token ep: https://oauth-openshift.apps.prod.openshift.example.com/oauth/token Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm useProviderOAuthEndpoint INFO: OpenShift OAuth server is 4.x, specifically OpenShiftVersionInfo: major: 1 minor: 14+ gitVersion: v1.14.6+b294fe5 Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm initializeHttpsProxyAuthenticator INFO: Checking if HTTPS proxy initialization is required ... Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse INFO: OpenShift OAuth got an SSL error when accessing the issuer's token endpoint when using the SA certificate Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse INFO: OpenShift OAuth provider token endpoint failed unexpectedly using the JVMs default keystore javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) .... Version-Release number of selected component (if applicable): 4.2.16 How reproducible: Always Steps to Reproduce: 1. Configure custom cert for Router base domain (*.apps.example.com) 2. Configure trustedCA per docs link in description above. 3. Deploy Jenkins (oc new-app jenkins-ephemeral) 4. Attempt to login to Jenkins using OpenShift OAuth Additional info: The OpenShift Login Plugin generates a keystore based on the Service Account Secret mount point in the pod (specifically, /run/secrets/kubernetes.io/serviceaccount/ca.crt): https://github.com/openshift/jenkins-openshift-login-plugin/blob/master/src/main/java/org/openshift/jenkins/plugins/openshiftlogin/OpenShiftOAuth2SecurityRealm.java#L358-L366 This ca.crt doesn't include the trustedCA. Customer expected this would allow Jenkins to verify TLS for OpenShift OAuth Route.
I think the issue here is that the jenkins jvm, keystore needs to be updated with the right certificates (CA chain). https://access.redhat.com/solutions/310913
This was fixed with this bug https://bugzilla.redhat.com/show_bug.cgi?id=1782819 in 4.3.
I still could reproduce the issue with payload: 4.2.0-0.nightly-2020-02-25-171913. 1) Configure custom cert for Router base domain (*.apps.yinzhou-bug.qe.devcluster.openshift.com) 2) Follow doc to configure onfigure trustedCA: https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#nw-proxy-configure-object_configuring-a-custom-pki 3. Deploy Jenkins (oc new-app jenkins-ephemeral) 4. Attempt to login to Jenkins using OpenShift OAuth Met error: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ......
Hello Zhow, Currently Jenkins does not support customCA explicitly, but to make it easier I am working on a feature for the Login Plugin it should make things easier. 1> Get the default keyStore. oc rsync jenkins-1-8zbx2:/etc/pki/java ./custom-java 2> Add certificate to keystore sudo keytool -keystore ./custom-java/cacerts -import -alias custom-ingress -file ./example.crt 3> Create a ConfigMap from the custom keystore oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts 4> Edit DeploymentConfig for using the custom Keystore with the following changes. spec: template: spec: containers: - env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts" volumeMounts: - mountPath: /etc/pki/java/cacerts name: jenkins-custom-keystore volumes: - name: jenkins-custom-keystore configMap: name: jenkins-custom-keystore Hope this helps, Regards,
(In reply to vbobade from comment #15) > Hello Zhow, > > Currently Jenkins does not support customCA explicitly, but to make it > easier I am working on a feature for the Login Plugin it should make things > easier. > > 1> Get the default keyStore. > > oc rsync jenkins-1-8zbx2:/etc/pki/java ./custom-java > > 2> Add certificate to keystore > > sudo keytool -keystore ./custom-java/cacerts -import -alias custom-ingress > -file ./example.crt > > 3> Create a ConfigMap from the custom keystore > > oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts > > 4> Edit DeploymentConfig for using the custom Keystore with the following > changes. > > spec: > template: > spec: > containers: > - env: > - name: JAVA_TOOL_OPTIONS > value: "-XX:+UnlockExperimentalVMOptions > -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true > -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts" > volumeMounts: > - mountPath: /etc/pki/java/cacerts > name: jenkins-custom-keystore > volumes: > - name: jenkins-custom-keystore > configMap: > name: jenkins-custom-keystore > > Hope this helps, > > Regards, Thanks for help, I'll try again.
I'll move to qa since Zhow is already looking at it.
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1809924
@yinzhou, bug https://bugzilla.redhat.com/show_bug.cgi?id=1809924 have been verified, pls help to verify it ASAP, and inform me or @wsun if there is any issue, thanks
Confirmed with payload: 4.2.0-0.nightly-2020-03-08-215456, the issue has fixed: 1) Follow https://docs.openshift.com/container-platform/4.2/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress , to setting a custom default certificate; 2) Follow https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#nw-proxy-configure-object_configuring-a-custom-pki , to enabling the cluster-wide proxy; 3) Create jenkins apps; 4) Extract the keystore from the running Jenkins pod: Raw $ mkdir custom-java $ oc rsync jenkins-<pod-id>:/etc/pki/ca-trust/extracted/java/cacerts ./custom-java/ 5) Concatenate "RootCA + IntermediateCA" and import them into the keystore as "trustcacerts": Raw $ cat /path/to/intermed-ca.cert.pem /path/to/root-ca.cert.pem > ./custom-java/rootCA.pem $ sudo keytool -import -trustcacerts -noprompt -keystore ./custom-java/cacerts -storepass changeit -alias custom-ingress -file ./custom-java/rootCA.pem 6) Create the proper configmap: Raw $ oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts 7)Edit the deploymentconfig including the volumeMounts and volumes sections as follows: Raw $ oc edit deploymentconfig.apps.openshift.io/jenkins [...] spec: containers: - env: - name: OPENSHIFT_ENABLE_OAUTH value: "true" [...] volumeMounts: - mountPath: /etc/pki/ca-trust/extracted/java/ name: jenkins-custom-keystore dnsPolicy: ClusterFirst restartPolicy: Always [...] volumes: - name: jenkins-custom-keystore configMap: name: jenkins-custom-keystore 8) Check the jenkins apps from webconsole by route.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0685
Re-opening this BZ, since this issue is not really fixed in 4.2.22
Based on comment 15 this is not an issue with kcm but with jenkins, so I'm moving this accordingly. KCM has all the necessary bits.
Anecdotally, even using the workaround from comment 29 Jenkins doesn't completely trust the injected certificate. Creating a new pipeline via the Blue Ocean interface failed to validate an HTTPS git URL whose validation chain depended on the injected root. As a user I expect the trust settings from the cluster-wide proxy configuration to be automatically trusted by all components managed by Red Hat. For Jenkins to ignore those settings runs against the expectation of what "cluster-wide" proxy settings should impact.
Hi all, we have pushed the PR to solve this issue. It is pending merge: https://github.com/openshift/jenkins/pull/1045
Marked for upcoming sprint.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days