Bug 1397757

Summary: The custom CA files were overwritten during installation
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: InstallerAssignee: Tim Bielawa <tbielawa>
Status: CLOSED NOTABUG QA Contact: Johnny Liu <jialiu>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.4.0CC: abutcher, aos-bugs, gpei, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-05 19:16:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Gaoyun Pei 2016-11-23 10:15:07 UTC
Description of problem:
Specify custom ca cert in ansible inventory, trigger an installation.
openshift_master_ca_certificate={"certfile": "/path/to/ca.crt", "keyfile": "/path/to/ca.key"}

The custom ca cert were overwritten by newly generated ca when creating the master certificates.


Version-Release number of selected component (if applicable):
openshift-ansible-3.4.26-1.git.0.882474b.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1.Prepare a self-signed ca cert, set openshift_master_ca_certificate option in inventory file, run the install playbook
openshift_master_ca_certificate={"certfile": "/path/to/ca.crt", "keyfile": "/path/to/ca.key"}
[root@ip-172-18-11-122 ~]# openssl x509 -in /path/to/ca.crt -text |grep Subject:
        Subject: C=CN, ST=beijing, L=beijing, O=redhat, OU=openshift, CN=qe-test


2.The ca certs files were copied to /etc/origin/master/ dir, but were replaced by new ca cert when creating the master certificates with commnd:
oc adm create-master-certs --hostnames=kubernetes.default,kubernetes.default.svc.cluster.local,kubernetes,openshift.default,x.com,ip-172-18-11-122.ec2.internal,openshift.default.svc,172.30.0.1,54.164.64.179,172.18.11.122,openshift.default.svc.cluster.local,kubernetes.default.svc,openshift -master=https://ip-172-18-11-122.ec2.internal:8443 --public-master=https://x.com:8443 --cert-dir=/etc/origin/master --overwrite=false


Actual results:
After installation, check the CN of /etc/origin/master/ca.crt
[root@ip-172-18-11-122 master]# openssl x509 -in ca.crt -text |grep Subject
        Subject: CN=openshift-signer@1479886468

ca-bundle.crt has the same content as ca.crt


Expected results:
Custom ca cert should be used

Additional info:
Related ansible installer logs:
TASK [openshift_ca : Deploy master ca certificate] *****************************
Wednesday 23 November 2016  03:16:46 +0000 (0:00:00.193)       0:06:53.160 **** 
changed: [x.com -> x.com] => (item={u'dest': u'ca.crt', u'src': u'/path/to/ca.crt'}) => {"changed": true, "checksum": "aa036a66bdd7e5d6e336a6848ee3659e36982024", "dest": "/etc/origin/master/ca.crt", "gid": 0, "group": "root", "item": {"dest": "ca.crt", "src": "/path/to/ca.crt"}, "md5sum": "b47bee28be185efc25e022fda45a35f0", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 1322, "src": "/root/.ansible/tmp/ansible-tmp-1479871008.02-181141957834103/source", "state": "file", "uid": 0}
changed: [x.com -> x.com] => (item={u'dest': u'ca.key', u'src': u'/path/to/ca.key'}) => {"changed": true, "checksum": "8d0aa1f9e9273fefd2ab9070cf5712f84bb066dc", "dest": "/etc/origin/master/ca.key", "gid": 0, "group": "root", "item": {"dest": "ca.key", "src": "/path/to/ca.key"}, "md5sum": "7b753021411c457ceaf2d694797d3c35", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 1834, "src": "/root/.ansible/tmp/ansible-tmp-1479871012.0-266515015458432/source", "state": "file", "uid": 0}

TASK [openshift_ca : Create ca serial] *****************************************
Wednesday 23 November 2016  03:16:54 +0000 (0:00:08.139)       0:07:01.300 **** 
changed: [x.com -> x.com] => {"changed": true, "checksum": "356a192b7913b04c54574d18c28d46e6395428ab", "dest": "/etc/origin/master/ca.serial.txt", "gid": 0, "group": "root", "md5sum": "c4ca4238a0b923820dcc509a6f75849b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 1, "src": "/root/.ansible/tmp/ansible-tmp-1479871016.16-31779106527266/source", "state": "file", "uid": 0}

TASK [openshift_ca : Create the master certificates if they do not already exist] ***
Wednesday 23 November 2016  03:16:58 +0000 (0:00:04.138)       0:07:05.439 **** 
changed: [x.com -> ] => {"changed": true, "cmd": ["oc", "adm", "create-master-certs", "--hostnames=kubernetes.default,kubernetes.default.svc.cluster.local,kubernetes,openshift.default,x.com,ip-172-18-11-122.ec2.internal,openshift.default.svc,172.30.0.1,54.164.64.179,172.18.11.122,openshift.default.svc.cluster.local,kubernetes.default.svc,openshift", "--master=https://ip-172-18-11-122.ec2.internal:8443", "--public-master=https://x.com:8443", "--cert-dir=/etc/origin/master", "--overwrite=false"], "delta": "0:00:06.914696", "end": "2016-11-22 22:17:07.301075", "rc": 0, "start": "2016-11-22 22:17:00.386379", "stderr": "Command \"create-master-certs\" is deprecated, Use 'oc adm ca' instead.", "stdout": "Generated new key pair as /etc/origin/master/serviceaccounts.public.key and /etc/origin/master/serviceaccounts.private.key", "stdout_lines": ["Generated new key pair as /etc/origin/master/serviceaccounts.public.key and /etc/origin/master/serviceaccounts.private.key"], "warnings": []}

Comment 1 Tim Bielawa 2016-11-28 16:06:52 UTC
Attempting to reproduce this now.

Comment 2 Tim Bielawa 2016-11-28 17:10:38 UTC
I could not reproduce this error on a fresh OCP 3.4 installation with this inventory:

> [OSEv3:children]
> nodes
> masters
> nfs
> etcd
>
> [OSEv3:vars]
> openshift_master_cluster_public_hostname=m01.example.com
> ansible_ssh_user=root
> openshift_master_cluster_hostname=m01.example.com
> deployment_type=openshift-enterprise
>
> openshift_master_ca_certificate={"certfile": "/tmp/tmp.ewjTUUOvnj/bz1397757.crt", "keyfile": "/tmp/tmp.ewjTUUOvnj/bz1397757.key"}
> openshift_master_default_subdomain=m01.example.com
> openshift_release=3
> openshift_additional_repos=[{'id': 'ose-devel', 'name': 'ose-devel', 'baseurl': 'http://download.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.4/latest/x86_64/os/', 'enabled': 1, 'gpgcheck': 0}]


With this self-generated certificate:

> [/tmp/tmp.ewjTUUOvnj] 9:07:55 
> $ openssl x509 -in ./bz1397757.crt -text | grep Subject
>        Subject: C=US, ST=NV, L=Henderson, O=Tim Blabla, CN=m01.example.com/emailAddress=tbielawa

Here is my software summary:

> $ rpm -q openshift-ansible ansible
> openshift-ansible-3.4.17-1.git.164.b45db4c.fc23.noarch
> ansible-2.2.0.0-3.fc23.noarch


I'll try to reproduce again using the same commit as you, 882474b.

Comment 3 Tim Bielawa 2016-11-28 17:16:44 UTC
Just realized I forgot to set the openshift release to 3.4...

Comment 4 Tim Bielawa 2016-11-28 17:31:57 UTC
Re-ran the test to reproduce again and was still unable to reproduce the error.

> [root@m01 master]# openssl x509 -in ca.crt -text |grep Subject
>         Subject: C=US, ST=NV, L=Henderson, O=Tim Blabla, CN=m01.example.com/emailAddress=tbielawa
>
> [root@m01 master]# ls -l ca.crt 
> -rw-r--r--. 1 root root 1391 Nov 28 12:21 ca.crt
>
> [root@m01 master]# date
> Mon Nov 28 12:28:35 EST 2016
>
> [root@m01 master]# oc version
> oc v3.4.0.29+ca980ba
> kubernetes v1.4.0+776c994
> features: Basic-Auth GSSAPI Kerberos SPNEGO
>
> Server https://m01.example.com:8443
> openshift v3.4.0.29+ca980ba
> kubernetes v1.4.0+776c994


This was on OCP 3.4 and the same openshift-ansible package version as you:

> $ rpm -q openshift-ansible ansible
> openshift-ansible-3.4.26-1.git.0.882474b.fc23.noarch
> ansible-2.2.0.0-3.fc23.noarch

Comment 5 Tim Bielawa 2016-11-28 17:34:48 UTC
@Gaoyun Pei, I'm not sure what you're doing differently than I am. Can you please provide additional information such as a full inventory file?

Comment 10 Andrew Butcher 2016-12-05 19:16:08 UTC
Hey Gaoyun, the attached CA has a pass phrase set and this is why OpenShift is replacing it. Can we update the test case to ensure that a pass phrase is not set for the CA?

Comment 11 Gaoyun Pei 2016-12-06 09:32:40 UTC
(In reply to Andrew Butcher from comment #10)
> Hey Gaoyun, the attached CA has a pass phrase set and this is why OpenShift
> is replacing it. Can we update the test case to ensure that a pass phrase is
> not set for the CA?

Tried again with removing the pass phrase on CA key file, it could work as expected. So openshift doesn't support custom CA with pass phrase added, have updated the test case, thanks Andrew!