Bug 1802297 - OpenShift Login Plugin doesn't support custom PKI [NEEDINFO]
Summary: OpenShift Login Plugin doesn't support custom PKI
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Jenkins
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.z
Assignee: Akram Ben Aissi
QA Contact: Jitendar Singh
URL:
Whiteboard:
Depends On: 1782819 1809924
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-12 20:23 UTC by Robert Bost
Modified: 2020-07-21 14:22 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-21 14:22:30 UTC
Target Upstream Version:
rbost: needinfo? (vbobade)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-kube-controller-manager-operator pull 350 None closed Bug 1802297: Inject kube-controller-manager pods trust stores with trusted ca bundle 2020-08-04 09:36:27 UTC
Github openshift jenkins pull 1087 None closed Bug 1802297: handle ingress certificate signed by custom ca 2020-08-04 09:36:26 UTC
Red Hat Knowledge Base (Solution) 4897631 None None None 2020-03-11 17:08:14 UTC
Red Hat Product Errata RHBA-2020:0685 None None None 2020-03-10 11:41:29 UTC

Description Robert Bost 2020-02-12 20:23:52 UTC
Description of problem:

Cannot login to Jenkins after configuring custom Router base domain wildcard cert. Customer has configured the custom PKI via documentation but still has issues:

  https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

The following error is seen in Jenkins logs after attempting login:

Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm populateDefaults
INFO: OpenShift OAuth: provider: OpenShiftProviderInfo: issuer: https://oauth-openshift.apps.prod.openshift.example.com auth ep: https://oauth-openshift.apps.prod.openshift.example.com/oauth
/authorize token ep: https://oauth-openshift.apps.prod.openshift.example.com/oauth/token
Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm useProviderOAuthEndpoint
INFO: OpenShift OAuth server is 4.x, specifically OpenShiftVersionInfo: major: 1 minor: 14+ gitVersion: v1.14.6+b294fe5
Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm initializeHttpsProxyAuthenticator
INFO: Checking if HTTPS proxy initialization is required ... 
Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse
INFO: OpenShift OAuth got an SSL error when accessing the issuer's token endpoint when using the SA certificate
Jan 29, 2020 10:02:08 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm transportToUse
INFO: OpenShift OAuth provider token endpoint failed unexpectedly using the JVMs default keystore
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
        ....


Version-Release number of selected component (if applicable): 4.2.16


How reproducible: Always

Steps to Reproduce:
1. Configure custom cert for Router base domain (*.apps.example.com)
2. Configure trustedCA per docs link in description above.
3. Deploy Jenkins (oc new-app jenkins-ephemeral)
4. Attempt to login to Jenkins using OpenShift OAuth



Additional info:
The OpenShift Login Plugin generates a keystore based on the Service Account Secret mount point in the pod (specifically, /run/secrets/kubernetes.io/serviceaccount/ca.crt):

  https://github.com/openshift/jenkins-openshift-login-plugin/blob/master/src/main/java/org/openshift/jenkins/plugins/openshiftlogin/OpenShiftOAuth2SecurityRealm.java#L358-L366

This ca.crt doesn't include the trustedCA. Customer expected this would allow Jenkins to verify TLS for OpenShift OAuth Route.

Comment 2 Eric Rich 2020-02-12 20:43:15 UTC
I think the issue here is that the jenkins jvm, keystore needs to be updated with the right certificates (CA chain). 

https://access.redhat.com/solutions/310913

Comment 4 Maciej Szulik 2020-02-13 12:53:54 UTC
This was fixed with this bug https://bugzilla.redhat.com/show_bug.cgi?id=1782819 in 4.3.

Comment 10 zhou ying 2020-02-26 09:51:11 UTC
I still could reproduce the issue with payload: 4.2.0-0.nightly-2020-02-25-171913. 

1) Configure custom cert for Router base domain (*.apps.yinzhou-bug.qe.devcluster.openshift.com)
2) Follow doc to configure onfigure trustedCA:
https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#nw-proxy-configure-object_configuring-a-custom-pki

3. Deploy Jenkins (oc new-app jenkins-ephemeral)
4. Attempt to login to Jenkins using OpenShift OAuth

Met error: 
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)

......

Comment 15 Vibhav Bobade 2020-02-28 03:41:29 UTC
Hello Zhow,

Currently Jenkins does not support customCA explicitly, but to make it easier I am working on a feature for the Login Plugin it should make things easier.

1> Get the default keyStore.

oc rsync jenkins-1-8zbx2:/etc/pki/java ./custom-java

2> Add certificate to keystore

sudo keytool -keystore ./custom-java/cacerts -import -alias custom-ingress -file ./example.crt

3> Create a ConfigMap from the custom keystore

oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts

4> Edit DeploymentConfig for using the custom Keystore with the following changes.

spec:
  template:
    spec:
      containers:
      - env:
        - name: JAVA_TOOL_OPTIONS
          value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts"
        volumeMounts: 
        - mountPath: /etc/pki/java/cacerts
          name: jenkins-custom-keystore
      volumes:
      - name: jenkins-custom-keystore
        configMap:
          name: jenkins-custom-keystore

Hope this helps,

Regards,

Comment 17 zhou ying 2020-02-28 05:38:53 UTC
(In reply to vbobade from comment #15)
> Hello Zhow,
> 
> Currently Jenkins does not support customCA explicitly, but to make it
> easier I am working on a feature for the Login Plugin it should make things
> easier.
> 
> 1> Get the default keyStore.
> 
> oc rsync jenkins-1-8zbx2:/etc/pki/java ./custom-java
> 
> 2> Add certificate to keystore
> 
> sudo keytool -keystore ./custom-java/cacerts -import -alias custom-ingress
> -file ./example.crt
> 
> 3> Create a ConfigMap from the custom keystore
> 
> oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts
> 
> 4> Edit DeploymentConfig for using the custom Keystore with the following
> changes.
> 
> spec:
>   template:
>     spec:
>       containers:
>       - env:
>         - name: JAVA_TOOL_OPTIONS
>           value: "-XX:+UnlockExperimentalVMOptions
> -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true
> -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts"
>         volumeMounts: 
>         - mountPath: /etc/pki/java/cacerts
>           name: jenkins-custom-keystore
>       volumes:
>       - name: jenkins-custom-keystore
>         configMap:
>           name: jenkins-custom-keystore
> 
> Hope this helps,
> 
> Regards,

Thanks for help, I'll try again.

Comment 18 Maciej Szulik 2020-02-28 11:20:48 UTC
I'll move to qa since Zhow is already looking at it.

Comment 27 zhou ying 2020-03-06 08:38:07 UTC
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1809924

Comment 28 ge liu 2020-03-08 03:54:34 UTC
@yinzhou, bug https://bugzilla.redhat.com/show_bug.cgi?id=1809924 have been verified, pls help to verify it ASAP, and inform me or @wsun if there is any issue, thanks

Comment 29 zhou ying 2020-03-09 06:14:15 UTC
Confirmed with payload: 4.2.0-0.nightly-2020-03-08-215456, the issue has fixed:

1) Follow https://docs.openshift.com/container-platform/4.2/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress   , to setting a custom default certificate;

2) Follow https://docs.openshift.com/container-platform/4.3/networking/configuring-a-custom-pki.html#nw-proxy-configure-object_configuring-a-custom-pki , to enabling the cluster-wide proxy;

3) Create jenkins apps;

4) Extract the keystore from the running Jenkins pod:
Raw
$ mkdir custom-java
$ oc rsync jenkins-<pod-id>:/etc/pki/ca-trust/extracted/java/cacerts ./custom-java/

5) Concatenate "RootCA + IntermediateCA" and import them into the keystore as "trustcacerts":
Raw
$ cat /path/to/intermed-ca.cert.pem /path/to/root-ca.cert.pem > ./custom-java/rootCA.pem
$ sudo keytool -import -trustcacerts -noprompt -keystore ./custom-java/cacerts -storepass changeit -alias custom-ingress -file ./custom-java/rootCA.pem
6) Create the proper configmap:
Raw
$ oc create configmap jenkins-custom-keystore --from-file=./custom-java/cacerts

7)Edit the deploymentconfig including the volumeMounts and volumes sections as follows:
Raw
$ oc edit deploymentconfig.apps.openshift.io/jenkins
 
[...]
    spec:
      containers:
      - env:
        - name: OPENSHIFT_ENABLE_OAUTH
          value: "true"
[...]
        volumeMounts:
        - mountPath: /etc/pki/ca-trust/extracted/java/
          name: jenkins-custom-keystore
      dnsPolicy: ClusterFirst
      restartPolicy: Always
[...]
      volumes:
      - name: jenkins-custom-keystore
        configMap:
          name: jenkins-custom-keystore

8) Check the jenkins apps from webconsole by route.

Comment 31 errata-xmlrpc 2020-03-10 11:41:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0685

Comment 33 Anand Paladugu 2020-04-06 13:20:12 UTC
Re-opening this BZ, since this issue is not really fixed in 4.2.22

Comment 34 Maciej Szulik 2020-04-06 13:33:22 UTC
Based on comment 15 this is not an issue with kcm but with jenkins, so I'm moving this accordingly. 
KCM has all the necessary bits.

Comment 35 Chet Hosey 2020-04-14 08:31:25 UTC
Anecdotally, even using the workaround from comment 29 Jenkins doesn't completely trust the injected certificate. Creating a new pipeline via the Blue Ocean interface failed to validate an HTTPS git URL whose validation chain depended on the injected root.

As a user I expect the trust settings from the cluster-wide proxy configuration to be automatically trusted by all components managed by Red Hat. For Jenkins to ignore those settings runs against the expectation of what "cluster-wide" proxy settings should impact.

Comment 36 Akram Ben Aissi 2020-04-14 08:46:38 UTC
Hi all,

we have pushed the PR to solve this issue. It is pending merge: https://github.com/openshift/jenkins/pull/1045

Comment 41 Vibhav Bobade 2020-07-09 10:31:20 UTC
Marked for upcoming sprint.


Note You need to log in before you can comment on or make changes to this bug.