Bug 1668970 - Router does not properly concatenate certificates generated by Red Hat IdM due to missing nonprintable ('\n')
Summary: Router does not properly concatenate certificates generated by Red Hat IdM du...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: All
OS: Linux
Target Milestone: ---
: 3.11.z
Assignee: Scott Dodson
QA Contact: Gaoyun Pei
Depends On:
TreeView+ depends on / blocked
Reported: 2019-01-24 03:15 UTC by Brian J. Beaudoin
Modified: 2019-02-20 14:11 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
If the specified router certificate, key, or CA did not end with a new line character the router deployment would fail. A new line is now appended to each of the input files ensuring this problem doesn't occur.
Clone Of:
Last Closed: 2019-02-20 14:11:02 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0326 0 None None None 2019-02-20 14:11:07 UTC

Description Brian J. Beaudoin 2019-01-24 03:15:29 UTC
Description of problem:

When using a certificate (cert.pem) downloaded from Red Hat IdM, the router deployment and redeploy-router-certificates.yml playbooks, an error is generated stating the certificate and key must be in the same file due to a missing '\n' at the end of the file.

Version-Release number of the following components:
$ rpm -q openshift-ansible
$ rpm -q ansible
$ ansible --version
ansible 2.6.11
  config file = /home/bbeaudoin/.ansible.cfg
  configured module search path = [u'/home/bbeaudoin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Steps to Reproduce:
1. Generate a certificate using Red Hat IdM (or any other tool) without a '\n' after -----END CERTIFICATE-----
2. Run the installation or use the redeploy-router-certificates.yml playbook

Actual results:
__main__.RouterException: Could not perform router preparation: error: router could not be created: the default cert must contain a private key

Expected results:
Safe concatenation of files that may be generated (rather than cut-and-pasted) that appear to be correct with exception of a missing nonprintable character.

Additional info:

Concatenation resulted in a temporary file that contained this string in the middle:


As the error suggested the certificate and key should have been concatenated before running the playbook, the missing nonprintable character may not be obvious if the file is not closely inspected.

Examining the certificate using `openssl x509` or other certificate utilities show a valid certificate file. Opening in a text editor does not reveal an issue due to the missing character being nonprintable.

If the files are concatenated by the user (in response to the error) before running the playbook without fixing the error, the playbooks will happily write or replace the router certificates (as the private key now exists in the concatenated file due to double-concatenation) but despite the playbook running to completion without error the router deployment will fail.

Traceback (most recent call last):
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3252, in <module>
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3243, in main
    results = Router.run_ansible(module.params, module.check_mode)
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3172, in run_ansible
    if not ocrouter.needs_update():
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 3010, in needs_update
    not Utils.check_def_equal(self.prepared_router['ServiceAccount']['obj'].yaml_dict,
  File \"/tmp/ansible_fCMFq7/ansible_module_oc_adm_router.py\", line 2729, in prepared_router
    raise RouterException('Could not perform router preparation: %s' % results['stderr'])
__main__.RouterException: Could not perform router preparation: error: router could not be created: the default cert must contain a private key

Comment 2 Scott Dodson 2019-01-25 14:35:58 UTC
Change has merged to release-3.11 branch, can you validate that resolves the issue with the problematic inputs you were using?

Comment 3 Brian J. Beaudoin 2019-01-28 13:44:02 UTC
Works as expected. Ran the playbook ensuring the secret was replaced, the contents were valid, and the router deployed properly.

Comment 4 Scott Dodson 2019-01-28 13:52:05 UTC
In openshift-ansible-3.11.75-1, why didn't this got ON_QA when the new tag was cut?

Comment 6 Gaoyun Pei 2019-01-29 13:44:04 UTC
Could reproduce this issue with openshift-ansible-3.11.59-1.git.0.ba8e948.el7.noarch

1. Prepare a custom router cert file which doesn't have line break in the bottom line.

[root@gpei-preserve-ansible-slave host]# cat -e /path/to/custom_router_1_no_end_linebreak.crt
-----END CERTIFICATE-----[root@gpei-preserve-ansible-slave host]# 

2. Set the custom router cert and key in inventory file
openshift_hosted_router_certificate={"certfile": "/path/to/custom_router_1_no_end_linebreak.crt", "keyfile": "/path/to/custom_router_1.key", "cafile": "/path/to/custom_router_1_rootca.crt"}

3. Run redeploy-router-certificates.yml playbook against a 3.11 cluster

TASK [openshift_hosted : Create OpenShift router] **********************************************************************************************************************************************************
failed: [host-8-245-45.host.centralci.eng.rdu2.redhat.com] ...
"module_stderr": "Shared connection to host-8-245-45.host.centralci.eng.rdu2.redhat.com closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible/ansible-tmp-1548767802.39-160700364467405/AnsiballZ_oc_adm_router.py\", line 113, in <module>\r\n    _ansiballz_main()\r\n  File \"/tmp/ansible/ansible-tmp-1548767802.39-160700364467405/AnsiballZ_oc_adm_router.py\", line 105, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/tmp/ansible/ansible-tmp-1548767802.39-160700364467405/AnsiballZ_oc_adm_router.py\", line 48, in invoke_module\r\n    imp.load_module('__main__', mod, module, MOD_DESC)\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3252, in <module>\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3243, in main\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3172, in run_ansible\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 3010, in needs_update\r\n  File \"/tmp/ansible_oc_adm_router_payload_S2pU6a/__main__.py\", line 2729, in prepared_router\r\n__main__.RouterException: Could not perform router preparation: error: router could not be created: the default cert must contain a private key\r\n\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

With openshift-ansible-3.11.75-1.git.0.95e8e2a.el7.noarch, the redeploy-router-certificates.yml playbook could be finished without such error.
router-certs secret was updated to use the new provided cert and key, router pod was running well after redeployment.

Comment 9 errata-xmlrpc 2019-02-20 14:11:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.