Bug 1269070 - Custom cert for web console
Custom cert for web console
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Management Console (Show other bugs)
3.0.0
Unspecified Unspecified
urgent Severity high
: ---
: ---
Assigned To: Jordan Liggitt
yapei
:
Depends On:
Blocks: 1267746
  Show dependency treegraph
 
Reported: 2015-10-06 04:39 EDT by Takayoshi Kimura
Modified: 2015-12-09 00:52 EST (History)
12 users (show)

See Also:
Fixed In Version: atomic-openshift-3.0.2.901-0.git.61.568adb6.el7aos.x86_64
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 09:26:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1259150 None None None Never
Red Hat Knowledge Base (Solution) 1605473 None None None Never
Origin (Github) 4009 None None None Never

  None (edit)
Description Takayoshi Kimura 2015-10-06 04:39:11 EDT
Some customers want to avoid self-signed cert for user facing UI and use cert signed by corporate CA instead.

Add an ability to adopt custom cert for OpenShift web console.
Comment 1 Takayoshi Kimura 2015-10-06 04:43:08 EDT
Upstream issue related to this:

Generate CSRs to streamline signing with custom CA #4009 https://github.com/openshift/origin/issues/4009

There are 2 solutions proposed:

1) Custom cert on differnt port (not implemneted yet)
2) Apache frontend proxy with custom cert

For #2 workaround, we need to confirm if it's a supported option in OSE.
Comment 3 Takayoshi Kimura 2015-10-06 05:06:26 EDT
This is initially raised as a doc bug BZ 1259150 but it has various topics so I created this BZ focusing on web console cert.

https://bugzilla.redhat.com/show_bug.cgi?id=1259150
Comment 4 Jordan Liggitt 2015-10-06 11:17:35 EDT
There are a few issues here:

1. The master API and web console share a cert/key if they are both configured to bind to the same interface/port. If you bind the UI to a separate interface or port, you can specify a different serving cert/key, but you might need to update the assetConfig.publicURL (oauthConfig.assetPublicURL) to point to the new port.

2. Even if the UI uses a different cert, the browser is still directed to the API publicMasterURL to log in, and to make API requests, so the serving cert/key for the API server would still be encountered by the web console.

To use a corporate CA for the serving cert (for the API), the options are:

  a. manually generate one with the same common name/subjectAltNames as the OpenShift cert, replace the cert/key, and append the corporate CA chain to all the places the OpenShift CA is supplied to infrastructure (in client kubeconfig files, in the CA provided to service accounts, etc)

  b. wait for ansible/openshift to be able to produce CSR's and consume custom CAs

To use a public CA for the serving cert, OpenShift would need to be enhanced to allow multiple certs for the API (since no CA will sign a serving cert for private IP ranges).
Comment 5 jooho lee 2015-10-06 20:20:36 EDT
Hi Jordan,

As for first/second issues that you worried, is it real issues or can be configurable?
 
During test, I changed 4 parameters, do you think those modifications can cause other issues? :
~~~
assetConfig:
  logoutURL: ""
  masterPublicURL: https://master.test.com:6443            <====== Change
  publicURL: https://master.test.com:6443/console/         <====== Change
..
..
oauthConfig:
  assetPublicURL: https://master.test.com:6443/console/    <====== Change
  ...
  masterPublicURL: https://master.test.com:6443            <====== Change
~~~

I am not sure I understand correctly about the rest of your comments but I think there is no field for IP ranges to get public CA for the serving cert. By the way, without enhancement of openshift to use multiple CA, it is not possible to apply custom module only for web console, am I right?
Comment 7 Jordan Liggitt 2015-10-06 21:34:02 EDT
Those four config changes look correct to make browser clients use https://master.test.com:6443 for the API and https://master.test.com:6443/console for the web console

Did you test the oc command line client through the proxy or only the web console?

quick things to test with the web console to make sure the proxy is working:
* Web console login works (and all redirects are to https://master.test.com:6443/...)
* Web sockets work correctly in the web console (changes appear live on a project overview page without refreshing)

quick things to test with the CLI to make sure the proxy is working:
* Login: `oc login https://master.test.com:6443` prompts and completes
* Watch: `oc get pods --watch` sees updates to pods
* SPDY protocols: `oc exec -ti <podname> bash`
Comment 9 Michael Napolis 2015-10-11 22:30:03 EDT
Hi all,

Adding in Josep 'Pep' Turro Mauri 's findings and contributions related to this issue

===

I also did some testing and found that the Apache reverse proxy configuration mentioned has problems: websockets don't work; the console doesn't update dynamically, exec doesn't work...

Instead of fighting Apache config I tried with nginx instead of Apache. It works better, including console dynamic content and oc exec/rsh

I used the SCL's nginx16-nginx package with this in the configuration:

    server {
        listen       443;
        server_name  mygateway.example.com;

        ssl                  on;
        ssl_certificate      /etc/pki/tls/certs/corporate.crt;
        ssl_certificate_key  /etc/pki/tls/private/corporate.key;

        ssl_session_timeout  5m;

        ssl_protocols  SSLv2 SSLv3 TLSv1;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers   on;

        location / {
            proxy_pass   https://master.example.com:8443;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

    }

===

We'll need Engineering assistance to come up and verify a supported workaround for the strategic customer to use.
Comment 10 Jordan Liggitt 2015-10-12 14:33:47 EDT
Long-term, I'd like to allow setting multiple certs, so the customer could set a separate cert for the external-facing hostname

Short-term, can the customer create a certificate with all the required subjectAltNames? A certificate with the following subjectAltNames would be needed:

DNS:public-master.example.com (replace with public-master hostname)
DNS:master.example.com (replace with master hostname)
DNS:kubernetes
DNS:kubernetes.default
DNS:kubernetes.default.svc
DNS:kubernetes.default.svc.cluster.local
DNS:openshift
DNS:openshift.default
DNS:openshift.default.svc
DNS:openshift.default.svc.cluster.local
DNS:x.x.x.x (replace with master IP)
DNS:172.30.0.1 (replace with kubernetes service IP, first in service IP range)
IP Address:x.x.x.x (replace with master IP)
IP Address:172.30.0.1 (replace with kubernetes service IP, first in service IP range)

The signing CA for the custom cert would need to be appended to the ca referenced by the master-config.yaml, node-config.yaml, and kubeconfig files.

Something like this would add in a custom certificate on the master:

export CONFIG_DIR=/etc/openshift
export MASTER_CONFIG_DIR=/etc/openshift/master

$ cp custom-server.crt ${MASTER_CONFIG_DIR}/server.crt
$ cp custom-server.key ${MASTER_CONFIG_DIR}/server.key
$ cat custom-ca.crt >> ${MASTER_CONFIG_DIR}/ca.crt

kubeconfigs=($(find ${CONFIG_DIR} -name '*.kubeconfig'))
for kubeconfig in ${kubeconfigs[@]-}; do
  echo "Updating ${kubeconfig}..."
  clusters=($(oc config view --config=${kubeconfig} -o template '--template={{range .clusters}}{{printf "%s\n" .name}}{{end}}'))
  for cluster in ${clusters[@]-}; do
    echo "  Updating ${cluster}..."
    oc config set-cluster ${cluster} --certificate-authority=${MASTER_CONFIG_DIR}/ca.crt --config=${kubeconfig}
  done
done


The same "oc config set-cluster" update would need to be done in the kubeconfig files used on the nodes to contact the master
Comment 15 Jordan Liggitt 2015-10-16 11:55:27 EDT
Support for adding custom certificates used for specific hostnames added upstream in https://github.com/openshift/origin/pull/5097

Targeted for 3.1
Comment 18 weiwei jiang 2015-10-19 05:57:36 EDT
Checked with https://trello.com/c/Gc3FDSK8/524-2-allow-setting-custom-serving-certs-for-specific-hostnames-in-api-console-using-sni on atomic-openshift-3.0.2.901-0.git.61.568adb6.el7aos.x86_64, and found no issues.
Comment 28 Brenton Leanhardt 2015-11-23 09:26:40 EST
This fix is available in OpenShift Enterprise 3.1.

Note You need to log in before you can comment on or make changes to this bug.