Bug 1991448 - [AUTH-13] Jenkins UI login hits Ooops when oauth server is customized
Summary: [AUTH-13] Jenkins UI login hits Ooops when oauth server is customized
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Jenkins
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Alice Rum
QA Contact: Jitendar Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-09 07:53 UTC by liyao
Modified: 2023-09-15 01:13 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
* This release contains a known issue with Jenkins. If you customize the hostname and certificate of the OpenShift OAuth route, Jenkins no longer trusts the OAuth server endpoint. As a result, users cannot log in to the Jenkins console if they rely on the OpenShift OAuth integration to manage identity and access. + Workaround: See the Red Hat Knowledgebase solution, link:https://access.redhat.com/solutions/6470721[Deploy Jenkins on OpenShift with Custom OAuth Server URL]. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1991448[*BZ#1991448*])
Clone Of:
Environment:
Last Closed: 2022-01-28 16:30:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
jenkins UI login hits Oops (113.69 KB, image/png)
2021-08-09 07:53 UTC, liyao
no flags Details

Description liyao 2021-08-09 07:53:08 UTC
Created attachment 1812294 [details]
jenkins UI login hits Oops

Description of problem:
Starting 4.9, it can custom hostname and certificate for OAuth route. But after customized, when visit jenkins console UI, the UI cannot be displayed successfully

Version-Release number of selected component (if applicable):
4.9

How reproducible:
100%

Steps to Reproduce:
1. generate custom CA cert and signed server cert
2. create secret in openshift-config using the generated server cert
3. append the custom CA cert to original admin kubeconfig certificate authority field
4. configure the ingress.config to contain custom route settings
$ oc edit ingresses.config.openshift.io cluster
spec:
  domain: <cluster domain name>
  componentRoutes:
    - name: oauth-openshift
      namespace: openshift-authentication
      hostname: <custom-oauth-server-hostname>
      servingCertKeyPairSecret:
        name: <custom-oauth-cert>
5. wait for the authentication operator to pick up the changes
6. login in and create jenkins-persistent app
$ oc login -u NORMAL_USER -p ***
$ oc new-project my_proj
$ oc new-app jenkins-persistent
# wait for the jenkins pods to be ready  
7. browse the route, Log in with OpenShift
$ oc get route

Actual results:
jenkins UI cannot be displayed and please check the attachment.

Expected results:
jenkins UI can be displayed successfully


Additional info:
from the jenkins pod logs, there is below error message 
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
    at java.base/sun.security.validator.Validator.validate(Validator.java:264)
    at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341)
    ... 132 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)

Comment 1 Xingxing Xia 2021-08-09 10:24:41 UTC
This is found when testing https://issues.redhat.com/browse/AUTH-13 (follow-up of https://github.com/openshift/enhancements/blob/master/enhancements/ingress/custom-route-configuration.md ), for which we should ensure: when this feature is used by customers, all login methods should not be broken, including: oc, Management Console UI, Prometheus UI, Jenkins UI etc. So if you have issue, maybe contacting auth / ingress Devs could get help :)

Comment 2 Akram Ben Aissi 2021-08-09 16:12:39 UTC
the CA to sign the certificate that you specifiy for custom route is not know by Jenkins.
to do so,  you can follow the same procedure as here: https://docs.openshift.com/container-platform/4.8/networking/configuring-a-custom-pki.html#nw-proxy-configure-object_configuring-a-custom-pki


this will make this additional trustedCA available to the Network Operator, which in turn inject it in all pods of namespaces containing an empty configMap  annotated with
config.openshift.io/inject-trusted-cabundle="true"

We use this approach when installing Jenkins through the template, and so, we do that for your Jenkins instances.
But, in return you need to be sure that CA certs are correctly managed by Network Operator



There seem to be also a discrepancy in documentation and the way Network Operator and Custom Route Configuration work:
The first accepts configMap to specify additional CA.
The second one uses TLS secrets.

Discussion has been started on the following slack channel with the OpenShift Network Edge team.
https://coreos.slack.com/archives/CCH60A77E/p1628521935045800?thread_ts=1628521358.045500&cid=CCH60A77E

Comment 3 Akram Ben Aissi 2021-08-10 13:03:35 UTC
Based on additional discussions with @Standa , the observed behavior is not specific to Jenkins and this will be common to any workload using incomplete CA trust.

In the case of Custom Routes, it is possible to get the certificates in the configmap openshift-config-managed/oauth-serving-cert

or follow this PR https://github.com/openshift/oauth-proxy/pull/220 to determine a way to set up the certificates for user workloads.

Comment 4 liyao 2021-09-08 05:17:17 UTC
@Akram, now the documentation is ready with statement "The Cluster Authentication Operator publishes the OAuth server's serving certificate in the oauth-serving-cert config map in the openshift-config-managed namespace. You can find the certificate in the data.ca-bundle.crt key of the config map". As talked in the slack, without code change from Jenkins team, Jenkins UI cannot work for this case. Please help to check.

Comment 9 Alice Rum 2021-10-27 11:25:58 UTC
Here is the workaround:

In this example, `ca.pem` has the custom CA certificate.

As part of step 6, after you create jenkins namespace, but before creating the app.
1. Create configmap that contains the custom CA:
   oc create configmap oauth-jenkins-ca --from-file=ca.crt=./ca.pem

2. Download the jenkins app template from github:
   curl -L https://github.com/openshift/origin/blob/master/examples/jenkins/jenkins-persistent-template.json -O

3. Modify template to use new configmap together with the existing one as a projected volume:

"volumeMounts": [
	{
		"mountPath": "/var/lib/jenkins",
		"name": "${JENKINS_SERVICE_NAME}-data"
	},
	{
		"mountPath": "/etc/pki/ca-trust/source/anchors",
		"name": "all-ca-certs"
	}
]

<...>

"volumes": [
	{
		"name": "${JENKINS_SERVICE_NAME}-data",
		"persistentVolumeClaim": {
			"claimName": "${JENKINS_SERVICE_NAME}"
		}
	},
	{
		"name": "all-ca-certs",
		"projected": {
			"sources": [
				{
					"configMap": {
						"name": "${JENKINS_SERVICE_NAME}-trusted-ca-bundle",
						"items": [
							{
								"key": "ca-bundle.crt",
								"path": "ca-bundle.crt"
							}
						]
					}
				},
				{
					"configMap": {
						"name": "oauth-jenkins-ca",
						"items": [
							{
								"key": "ca.crt",
								"path": "ca.crt"
							}
						]
					}
				}
			]
		}
	}
]

4. Finally after that, you can deploy the jenkins application, and it will use CA certificates correctly:
   oc new-app --file=jenkins-persistent-template.json

Comment 12 Adam Kaplan 2022-01-28 16:30:14 UTC
Closing this as CURRENTRELEASE - we are documenting the workaround in https://access.redhat.com/solutions/6470721

Comment 13 Rolfe Dlugy-Hegwer 2022-02-17 20:29:38 UTC
Release noted in 4.10: https://github.com/openshift/openshift-docs/pull/40899

Comment 14 Red Hat Bugzilla 2023-09-15 01:13:23 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.