Bug 1668970

Summary: Router does not properly concatenate certificates generated by Red Hat IdM due to missing nonprintable ('\n')
Product: OpenShift Container Platform Reporter: Brian J. Beaudoin <bbeaudoi>
Component: InstallerAssignee: Scott Dodson <sdodson>
Installer sub component: openshift-ansible QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: unspecified CC: bbeaudoi, gpei
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
If the specified router certificate, key, or CA did not end with a new line character the router deployment would fail. A new line is now appended to each of the input files ensuring this problem doesn't occur.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-20 14:11:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brian J. Beaudoin 2019-01-24 03:15:29 UTC
Description of problem:

When using a certificate (cert.pem) downloaded from Red Hat IdM, the router deployment and redeploy-router-certificates.yml playbooks, an error is generated stating the certificate and key must be in the same file due to a missing '\n' at the end of the file.

Version-Release number of the following components:
$ rpm -q openshift-ansible
openshift-ansible-3.11.59-1.git.0.ba8e948.el7.noarch
$ rpm -q ansible
ansible-2.6.11-1.el7ae.noarch
$ ansible --version
ansible 2.6.11
  config file = /home/bbeaudoin/.ansible.cfg
  configured module search path = [u'/home/bbeaudoin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Steps to Reproduce:
1. Generate a certificate using Red Hat IdM (or any other tool) without a '\n' after -----END CERTIFICATE-----
2. Run the installation or use the redeploy-router-certificates.yml playbook

Actual results:
__main__.RouterException: Could not perform router preparation: error: router could not be created: the default cert must contain a private key

Expected results:
Safe concatenation of files that may be generated (rather than cut-and-pasted) that appear to be correct with exception of a missing nonprintable character.

Additional info:

Concatenation resulted in a temporary file that contained this string in the middle:

-----END CERTIFICATE----------BEGIN PRIVATE KEY-----

As the error suggested the certificate and key should have been concatenated before running the playbook, the missing nonprintable character may not be obvious if the file is not closely inspected.

Examining the certificate using `openssl x509` or other certificate utilities show a valid certificate file. Opening in a text editor does not reveal an issue due to the missing character being nonprintable.

If the files are concatenated by the user (in response to the error) before running the playbook without fixing the error, the playbooks will happily write or replace the router certificates (as the private key now exists in the concatenated file due to double-concatenation) but despite the playbook running to completion without error the router deployment will fail.

Traceback (most recent call last):
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3252, in <module>
    main()
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3243, in main
    results = Router.run_ansible(module.params, module.check_mode)
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3172, in run_ansible
    if not ocrouter.needs_update():
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3010, in needs_update
    not Utils.check_def_equal(self.prepared_router['ServiceAccount']['obj'].yaml_dict,
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 2729, in prepared_router
    raise RouterException('Could not perform router preparation: %s' % results['stderr'])
__main__.RouterException: Could not perform router preparation: error: router could not be created: the default cert must contain a private key

Comment 2 Scott Dodson 2019-01-25 14:35:58 UTC
Change has merged to release-3.11 branch, can you validate that resolves the issue with the problematic inputs you were using?

Comment 3 Brian J. Beaudoin 2019-01-28 13:44:02 UTC
Works as expected. Ran the playbook ensuring the secret was replaced, the contents were valid, and the router deployed properly.

Comment 4 Scott Dodson 2019-01-28 13:52:05 UTC
In openshift-ansible-3.11.75-1, why didn't this got ON_QA when the new tag was cut?

Comment 6 Gaoyun Pei 2019-01-29 13:44:04 UTC
Could reproduce this issue with openshift-ansible-3.11.59-1.git.0.ba8e948.el7.noarch

1. Prepare a custom router cert file which doesn't have line break in the bottom line.

[root@gpei-preserve-ansible-slave host]# cat -e /path/to/custom_router_1_no_end_linebreak.crt
-----BEGIN CERTIFICATE-----$
MIIDQjCCAioCCQDnQLeTFb3pAjANBgkqhkiG9w0BAQUFADBrMQswCQYDVQQGEwJY$
...
/QTdf8gk3AiClnpJsM3bnuIy2lqzY7mebky//JBz9M3RwCfKDz7EJuZ12mqu8p+S$
xfqh3NbO4g355/CppBlIX3BjwLtcbg==$
-----END CERTIFICATE-----[root@gpei-preserve-ansible-slave host]# 


2. Set the custom router cert and key in inventory file
openshift_hosted_router_certificate={"certfile": "/path/to/custom_router_1_no_end_linebreak.crt", "keyfile": "/path/to/custom_router_1.key", "cafile": "/path/to/custom_router_1_rootca.crt"}


3. Run redeploy-router-certificates.yml playbook against a 3.11 cluster

TASK [openshift_hosted : Create OpenShift router] **********************************************************************************************************************************************************
failed: [host-8-245-45.host.centralci.eng.rdu2.redhat.com] ...
"module_stderr": "Shared connection to host-8-245-45.host.centralci.eng.rdu2.redhat.com closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible/ansible-tmp-1548767802.39-160700364467405/AnsiballZ_oc_adm_router.py\", line 113, in <module>\r\n    _ansiballz_main()\r\n  File \"/tmp/ansible/ansible-tmp-1548767802.39-160700364467405/AnsiballZ_oc_adm_router.py\", line 105, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/tmp/ansible/ansible-tmp-1548767802.39-160700364467405/AnsiballZ_oc_adm_router.py\", line 48, in invoke_module\r\n    imp.load_module('__main__', mod, module, MOD_DESC)\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3252, in <module>\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3243, in main\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3172, in run_ansible\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3010, in needs_update\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 2729, in prepared_router\r\n__main__.RouterException: Could not perform router preparation: error: router could not be created: the default cert must contain a private key\r\n\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}


With openshift-ansible-3.11.75-1.git.0.95e8e2a.el7.noarch, the redeploy-router-certificates.yml playbook could be finished without such error.
router-certs secret was updated to use the new provided cert and key, router pod was running well after redeployment.

Comment 9 errata-xmlrpc 2019-02-20 14:11:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0326