Bug 1269070

Upstream issue related to this:

Generate CSRs to streamline signing with custom CA #4009 https://github.com/openshift/origin/issues/4009

There are 2 solutions proposed:

1) Custom cert on differnt port (not implemneted yet)
2) Apache frontend proxy with custom cert

For #2 workaround, we need to confirm if it's a supported option in OSE.

This is initially raised as a doc bug BZ 1259150 but it has various topics so I created this BZ focusing on web console cert.

https://bugzilla.redhat.com/show_bug.cgi?id=1259150

There are a few issues here:

1. The master API and web console share a cert/key if they are both configured to bind to the same interface/port. If you bind the UI to a separate interface or port, you can specify a different serving cert/key, but you might need to update the assetConfig.publicURL (oauthConfig.assetPublicURL) to point to the new port.

2. Even if the UI uses a different cert, the browser is still directed to the API publicMasterURL to log in, and to make API requests, so the serving cert/key for the API server would still be encountered by the web console.

To use a corporate CA for the serving cert (for the API), the options are:

  a. manually generate one with the same common name/subjectAltNames as the OpenShift cert, replace the cert/key, and append the corporate CA chain to all the places the OpenShift CA is supplied to infrastructure (in client kubeconfig files, in the CA provided to service accounts, etc)

  b. wait for ansible/openshift to be able to produce CSR's and consume custom CAs

To use a public CA for the serving cert, OpenShift would need to be enhanced to allow multiple certs for the API (since no CA will sign a serving cert for private IP ranges).

Hi Jordan,

As for first/second issues that you worried, is it real issues or can be configurable?
 
During test, I changed 4 parameters, do you think those modifications can cause other issues? :
~~~
assetConfig:
  logoutURL: ""
  masterPublicURL: https://master.test.com:6443            <====== Change
  publicURL: https://master.test.com:6443/console/         <====== Change
..
..
oauthConfig:
  assetPublicURL: https://master.test.com:6443/console/    <====== Change
  ...
  masterPublicURL: https://master.test.com:6443            <====== Change
~~~

I am not sure I understand correctly about the rest of your comments but I think there is no field for IP ranges to get public CA for the serving cert. By the way, without enhancement of openshift to use multiple CA, it is not possible to apply custom module only for web console, am I right?

Those four config changes look correct to make browser clients use https://master.test.com:6443 for the API and https://master.test.com:6443/console for the web console

Did you test the oc command line client through the proxy or only the web console?

quick things to test with the web console to make sure the proxy is working:
* Web console login works (and all redirects are to https://master.test.com:6443/...)
* Web sockets work correctly in the web console (changes appear live on a project overview page without refreshing)

quick things to test with the CLI to make sure the proxy is working:
* Login: `oc login https://master.test.com:6443` prompts and completes
* Watch: `oc get pods --watch` sees updates to pods
* SPDY protocols: `oc exec -ti <podname> bash`

Hi all,

Adding in Josep 'Pep' Turro Mauri 's findings and contributions related to this issue

===

I also did some testing and found that the Apache reverse proxy configuration mentioned has problems: websockets don't work; the console doesn't update dynamically, exec doesn't work...

Instead of fighting Apache config I tried with nginx instead of Apache. It works better, including console dynamic content and oc exec/rsh

I used the SCL's nginx16-nginx package with this in the configuration:

    server {
        listen       443;
        server_name  mygateway.example.com;

        ssl                  on;
        ssl_certificate      /etc/pki/tls/certs/corporate.crt;
        ssl_certificate_key  /etc/pki/tls/private/corporate.key;

        ssl_session_timeout  5m;

        ssl_protocols  SSLv2 SSLv3 TLSv1;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers   on;

        location / {
            proxy_pass   https://master.example.com:8443;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

    }

===

We'll need Engineering assistance to come up and verify a supported workaround for the strategic customer to use.

Long-term, I'd like to allow setting multiple certs, so the customer could set a separate cert for the external-facing hostname

Short-term, can the customer create a certificate with all the required subjectAltNames? A certificate with the following subjectAltNames would be needed:

DNS:public-master.example.com (replace with public-master hostname)
DNS:master.example.com (replace with master hostname)
DNS:kubernetes
DNS:kubernetes.default
DNS:kubernetes.default.svc
DNS:kubernetes.default.svc.cluster.local
DNS:openshift
DNS:openshift.default
DNS:openshift.default.svc
DNS:openshift.default.svc.cluster.local
DNS:x.x.x.x (replace with master IP)
DNS:172.30.0.1 (replace with kubernetes service IP, first in service IP range)
IP Address:x.x.x.x (replace with master IP)
IP Address:172.30.0.1 (replace with kubernetes service IP, first in service IP range)

The signing CA for the custom cert would need to be appended to the ca referenced by the master-config.yaml, node-config.yaml, and kubeconfig files.

Something like this would add in a custom certificate on the master:

export CONFIG_DIR=/etc/openshift
export MASTER_CONFIG_DIR=/etc/openshift/master

$ cp custom-server.crt ${MASTER_CONFIG_DIR}/server.crt
$ cp custom-server.key ${MASTER_CONFIG_DIR}/server.key
$ cat custom-ca.crt >> ${MASTER_CONFIG_DIR}/ca.crt

kubeconfigs=($(find ${CONFIG_DIR} -name '*.kubeconfig'))
for kubeconfig in ${kubeconfigs[@]-}; do
  echo "Updating ${kubeconfig}..."
  clusters=($(oc config view --config=${kubeconfig} -o template '--template={{range .clusters}}{{printf "%s\n" .name}}{{end}}'))
  for cluster in ${clusters[@]-}; do
    echo "  Updating ${cluster}..."
    oc config set-cluster ${cluster} --certificate-authority=${MASTER_CONFIG_DIR}/ca.crt --config=${kubeconfig}
  done
done


The same "oc config set-cluster" update would need to be done in the kubeconfig files used on the nodes to contact the master

Support for adding custom certificates used for specific hostnames added upstream in https://github.com/openshift/origin/pull/5097

Targeted for 3.1

Checked with https://trello.com/c/Gc3FDSK8/524-2-allow-setting-custom-serving-certs-for-specific-hostnames-in-api-console-using-sni on atomic-openshift-3.0.2.901-0.git.61.568adb6.el7aos.x86_64, and found no issues.

This fix is available in OpenShift Enterprise 3.1.