Created attachment 1812294 [details] jenkins UI login hits Oops Description of problem: Starting 4.9, it can custom hostname and certificate for OAuth route. But after customized, when visit jenkins console UI, the UI cannot be displayed successfully Version-Release number of selected component (if applicable): 4.9 How reproducible: 100% Steps to Reproduce: 1. generate custom CA cert and signed server cert 2. create secret in openshift-config using the generated server cert 3. append the custom CA cert to original admin kubeconfig certificate authority field 4. configure the ingress.config to contain custom route settings $ oc edit ingresses.config.openshift.io cluster spec: domain: <cluster domain name> componentRoutes: - name: oauth-openshift namespace: openshift-authentication hostname: <custom-oauth-server-hostname> servingCertKeyPairSecret: name: <custom-oauth-cert> 5. wait for the authentication operator to pick up the changes 6. login in and create jenkins-persistent app $ oc login -u NORMAL_USER -p *** $ oc new-project my_proj $ oc new-app jenkins-persistent # wait for the jenkins pods to be ready 7. browse the route, Log in with OpenShift $ oc get route Actual results: jenkins UI cannot be displayed and please check the attachment. Expected results: jenkins UI can be displayed successfully Additional info: from the jenkins pod logs, there is below error message Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) ... 132 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
This is found when testing https://issues.redhat.com/browse/AUTH-13 (follow-up of https://github.com/openshift/enhancements/blob/master/enhancements/ingress/custom-route-configuration.md ), for which we should ensure: when this feature is used by customers, all login methods should not be broken, including: oc, Management Console UI, Prometheus UI, Jenkins UI etc. So if you have issue, maybe contacting auth / ingress Devs could get help :)
the CA to sign the certificate that you specifiy for custom route is not know by Jenkins. to do so, you can follow the same procedure as here: https://docs.openshift.com/container-platform/4.8/networking/configuring-a-custom-pki.html#nw-proxy-configure-object_configuring-a-custom-pki this will make this additional trustedCA available to the Network Operator, which in turn inject it in all pods of namespaces containing an empty configMap annotated with config.openshift.io/inject-trusted-cabundle="true" We use this approach when installing Jenkins through the template, and so, we do that for your Jenkins instances. But, in return you need to be sure that CA certs are correctly managed by Network Operator There seem to be also a discrepancy in documentation and the way Network Operator and Custom Route Configuration work: The first accepts configMap to specify additional CA. The second one uses TLS secrets. Discussion has been started on the following slack channel with the OpenShift Network Edge team. https://coreos.slack.com/archives/CCH60A77E/p1628521935045800?thread_ts=1628521358.045500&cid=CCH60A77E
Based on additional discussions with @Standa , the observed behavior is not specific to Jenkins and this will be common to any workload using incomplete CA trust. In the case of Custom Routes, it is possible to get the certificates in the configmap openshift-config-managed/oauth-serving-cert or follow this PR https://github.com/openshift/oauth-proxy/pull/220 to determine a way to set up the certificates for user workloads.
@Akram, now the documentation is ready with statement "The Cluster Authentication Operator publishes the OAuth server's serving certificate in the oauth-serving-cert config map in the openshift-config-managed namespace. You can find the certificate in the data.ca-bundle.crt key of the config map". As talked in the slack, without code change from Jenkins team, Jenkins UI cannot work for this case. Please help to check.
Here is the workaround: In this example, `ca.pem` has the custom CA certificate. As part of step 6, after you create jenkins namespace, but before creating the app. 1. Create configmap that contains the custom CA: oc create configmap oauth-jenkins-ca --from-file=ca.crt=./ca.pem 2. Download the jenkins app template from github: curl -L https://github.com/openshift/origin/blob/master/examples/jenkins/jenkins-persistent-template.json -O 3. Modify template to use new configmap together with the existing one as a projected volume: "volumeMounts": [ { "mountPath": "/var/lib/jenkins", "name": "${JENKINS_SERVICE_NAME}-data" }, { "mountPath": "/etc/pki/ca-trust/source/anchors", "name": "all-ca-certs" } ] <...> "volumes": [ { "name": "${JENKINS_SERVICE_NAME}-data", "persistentVolumeClaim": { "claimName": "${JENKINS_SERVICE_NAME}" } }, { "name": "all-ca-certs", "projected": { "sources": [ { "configMap": { "name": "${JENKINS_SERVICE_NAME}-trusted-ca-bundle", "items": [ { "key": "ca-bundle.crt", "path": "ca-bundle.crt" } ] } }, { "configMap": { "name": "oauth-jenkins-ca", "items": [ { "key": "ca.crt", "path": "ca.crt" } ] } } ] } } ] 4. Finally after that, you can deploy the jenkins application, and it will use CA certificates correctly: oc new-app --file=jenkins-persistent-template.json
Closing this as CURRENTRELEASE - we are documenting the workaround in https://access.redhat.com/solutions/6470721
Release noted in 4.10: https://github.com/openshift/openshift-docs/pull/40899
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days