Bug 1565720

Summary: Can't add "namedCertificates" to "oc cluster up --use-existing-config" config in 3.10 like was possible in 3.9
Product: OKD Reporter: Derek Whatley <dwhatley>
Component: MasterAssignee: David Eads <deads>
Status: CLOSED NOTABUG QA Contact: Wang Haoran <haowang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.xCC: aos-bugs, deads, jmatthew, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-12 22:48:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Directory containing oc cluster up configuration files. none

Description Derek Whatley 2018-04-10 15:22:14 UTC
Created attachment 1419953 [details]
Directory containing oc cluster up configuration files.

Description of problem:
========================
In 3.9 and previous, it was possible to edit "openshift.local.clusterup/master/master-config.yaml" to contain a "namedCertificates" section which enabled us to get a nice green TLS lock in the browser.

It looked something like this:

servingInfo:
...
  namedCertificates:
    - certFile: /tmp/openshift.local.clusterup/console-fullchain.pem
      keyFile: /tmp/openshift.local.clusterup/console-privkey.pem
      names:
      - "openshift.dwhatley.com"
...

Where "/tmp/openshift.local.clusterup/console-full{chain,key}.pem" pointed at the desired cert and key to be used. 

In 3.10, trying to add the paths of a cert/key pair in the new component location, "openshift.local.clusterup/kube-apiserver/master-config.yaml", results in an repeated "Exited (255) on two pods during "oc cluster up --use-existing-config":

1f178e8909e3 k8s_controllers_kube-controller-manager
13b3b5a84cf6 k8s_scheduler_kube-scheduler


The error is:
[ec2-user@ip-10-0-0-77 ~]$ docker logs 1f178e8909e3                            
Invalid MasterConfig master-config.yaml                                        
  servingInfo.namedCertificates[0].certFile: Invalid value: "/tmp/openshift.local.clusterup/console-fullchain.pem": could not read file: stat /tmp/openshift.local.clusterup/console-fullchain.pem: no such file or directory                                                                                                 
  servingInfo.namedCertificates[0].keyFile: Invalid value: "/tmp/openshift.local.clusterup/console-privkey.pem": could not read file: stat /tmp/openshift.local.clusterup/console-privkey.pem: no such file or directory    


It seems to me like a regression that 3.10 doesn't copy/mount the cert/key pair into the required containers. The controller-manager and scheduler pods try to access a path that only exists on the base system.


Version-Release number of selected component (if applicable):
==============================================================
oc v3.10.0-alpha.0+4035345-541
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

How reproducible:
==================
100%

Steps to Reproduce:
1. Run oc cluster up with 3.10
2. Run oc cluster down
3. Create a cert/key pair for your domain (we use LetsEncrypt)
4. Edit openshift.local.clusterup/kube-apiserver/master-config.yaml to contain a "namedCertificates" section pointing at created cert/key pair
5. Run oc cluster up --use-existing-config

Actual results:
================
Run /usr/bin/oc cluster up --routing-suffix=dwhatley.aws.dog8code.com --public-hostname=dwhatley.aws.dog8code.com --base-dir=/tmp/openshift.local.clusterup --image=docker.io/openshift/origin --tag=latest --enable=service-catalog --us
e-existing-config

Gets stuck while spinning up kube-apiserver containers

Expected results:
==================
Doesn't get stuck spinning up kube-apiserver container, gives "green lock" icon when accessing Web Console signifying provided certs are actually being used.

Additional info:

Comment 1 Derek Whatley 2018-04-10 15:26:34 UTC
ALSO, I've omitted the cert/key from the listed locations in the tar file "/tmp/openshift.local.clusterup/console-privkey.pem", but they were there while running oc cluster up.

Comment 2 David Eads 2018-04-10 15:50:11 UTC
You're trying to point to files which are not mounted in the container.  The only directory mounted into the container (may be copied in the future) is the `CWD/openshift.local.cluster/kube-apiserver`.  You'll want to put your certs in that directory and reference them by the relative path, which is only `console-privkey.pem`

Comment 3 Derek Whatley 2018-04-12 22:48:24 UTC
Solution described by David solved this problem. Marking as closed.