Bug 1565720 - Can't add "namedCertificates" to "oc cluster up --use-existing-config" config in 3.10 like was possible in 3.9
Summary: Can't add "namedCertificates" to "oc cluster up --use-existing-config" config...
Alias: None
Product: OKD
Classification: Red Hat
Component: Master
Version: 3.x
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: David Eads
QA Contact: Wang Haoran
Depends On:
TreeView+ depends on / blocked
Reported: 2018-04-10 15:22 UTC by Derek Whatley
Modified: 2018-04-12 22:48 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-04-12 22:48:24 UTC
Target Upstream Version:

Attachments (Terms of Use)
Directory containing oc cluster up configuration files. (1.47 MB, application/x-gzip)
2018-04-10 15:22 UTC, Derek Whatley
no flags Details

Description Derek Whatley 2018-04-10 15:22:14 UTC
Created attachment 1419953 [details]
Directory containing oc cluster up configuration files.

Description of problem:
In 3.9 and previous, it was possible to edit "openshift.local.clusterup/master/master-config.yaml" to contain a "namedCertificates" section which enabled us to get a nice green TLS lock in the browser.

It looked something like this:

    - certFile: /tmp/openshift.local.clusterup/console-fullchain.pem
      keyFile: /tmp/openshift.local.clusterup/console-privkey.pem
      - "openshift.dwhatley.com"

Where "/tmp/openshift.local.clusterup/console-full{chain,key}.pem" pointed at the desired cert and key to be used. 

In 3.10, trying to add the paths of a cert/key pair in the new component location, "openshift.local.clusterup/kube-apiserver/master-config.yaml", results in an repeated "Exited (255) on two pods during "oc cluster up --use-existing-config":

1f178e8909e3 k8s_controllers_kube-controller-manager
13b3b5a84cf6 k8s_scheduler_kube-scheduler

The error is:
[ec2-user@ip-10-0-0-77 ~]$ docker logs 1f178e8909e3                            
Invalid MasterConfig master-config.yaml                                        
  servingInfo.namedCertificates[0].certFile: Invalid value: "/tmp/openshift.local.clusterup/console-fullchain.pem": could not read file: stat /tmp/openshift.local.clusterup/console-fullchain.pem: no such file or directory                                                                                                 
  servingInfo.namedCertificates[0].keyFile: Invalid value: "/tmp/openshift.local.clusterup/console-privkey.pem": could not read file: stat /tmp/openshift.local.clusterup/console-privkey.pem: no such file or directory    

It seems to me like a regression that 3.10 doesn't copy/mount the cert/key pair into the required containers. The controller-manager and scheduler pods try to access a path that only exists on the base system.

Version-Release number of selected component (if applicable):
oc v3.10.0-alpha.0+4035345-541
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

How reproducible:

Steps to Reproduce:
1. Run oc cluster up with 3.10
2. Run oc cluster down
3. Create a cert/key pair for your domain (we use LetsEncrypt)
4. Edit openshift.local.clusterup/kube-apiserver/master-config.yaml to contain a "namedCertificates" section pointing at created cert/key pair
5. Run oc cluster up --use-existing-config

Actual results:
Run /usr/bin/oc cluster up --routing-suffix=dwhatley.aws.dog8code.com --public-hostname=dwhatley.aws.dog8code.com --base-dir=/tmp/openshift.local.clusterup --image=docker.io/openshift/origin --tag=latest --enable=service-catalog --us

Gets stuck while spinning up kube-apiserver containers

Expected results:
Doesn't get stuck spinning up kube-apiserver container, gives "green lock" icon when accessing Web Console signifying provided certs are actually being used.

Additional info:

Comment 1 Derek Whatley 2018-04-10 15:26:34 UTC
ALSO, I've omitted the cert/key from the listed locations in the tar file "/tmp/openshift.local.clusterup/console-privkey.pem", but they were there while running oc cluster up.

Comment 2 David Eads 2018-04-10 15:50:11 UTC
You're trying to point to files which are not mounted in the container.  The only directory mounted into the container (may be copied in the future) is the `CWD/openshift.local.cluster/kube-apiserver`.  You'll want to put your certs in that directory and reference them by the relative path, which is only `console-privkey.pem`

Comment 3 Derek Whatley 2018-04-12 22:48:24 UTC
Solution described by David solved this problem. Marking as closed.

Note You need to log in before you can comment on or make changes to this bug.