Description of problem: When a cluster upgrade is tried, it failed at certificate expiry check phase. openshift_cert_expiry module failed and produced the following python stack trace at its stdout (sanitized from ansible output): Traceback (most recent call last): File "/tmp/ansible_koJMzO/ansible_module_openshift_cert_expiry.py", line 874, in <module> main() File "/tmp/ansible_koJMzO/ansible_module_openshift_cert_expiry.py", line 770, in main issuer) = load_and_handle_cert(router_c, now, base64decode=True, ans_module=module) File "/tmp/ansible_koJMzO/ansible_module_openshift_cert_expiry.py", line 280, in load_and_handle_cert OpenSSL.crypto.FILETYPE_PEM, _cert_string) UnicodeEncodeError: 'ascii' codec can't encode character u'\u0151' in position 151033: ordinal not in range(128) This issue is also reproducible doing a simple certificate expiration check. Setting different openshift_certificate_expiry_warning_days makes no difference and there are no certificates about to expire. The following PR seems to fix this issue: https://github.com/openshift/openshift-ansible/pull/11318 Version-Release number of the following components: $ rpm -q openshift-ansible openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch $ rpm -q ansible ansible-2.6.14-1.el7ae.noarch $ ansible --version ansible 2.6.14 config file = /home/ansible/git/ocp-ansible/ansible.cfg configured module search path = [u'/home/ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] How reproducible: On certain environments, but always. Steps to Reproduce: 1. Do a certificate check (easy-mode.yml playbook) or run any other playbook that invokes the module (like the upgrade playbook). Actual results: Certificate check fails although certificates are not about to expire. Error actually seems to have nothing with expiry. Expected results: Certificate checks should pass and allow update to continue Additional info: Please attach logs from ansible-playbook with the -vvv flag
Upstream PR https://github.com/openshift/openshift-ansible/pull/11318 fixes the problem and should merge soon. OpenSSL.load_certificate() takes a parameter of type bytes. We were passing a string, so python would convert to bytes. It seems the default encoding is ascii, which throws an error with a non-ascii character; I can reproduce this problem, but cannot find a definitive doc reference. Specifying the encoding resolves the issue.
Not able to reproduce this issue with openshift-ansible-3.11.82-1 Steps: 1. Create certificate with non-ascii character in "ST", "O" and "CN" fields. 2. Set openshift_master_ca_certificate={'certfile': '/root/cus_1.cert.pem', 'keyfile':'/root/cus_1.key.pem'} 3. After installation, confirm the customized ca cert file applied [root@ip-172-18-11-196 master]# openssl x509 -in ca.crt -subject -noout subject= /C=GB/ST=\xC3\x85\xC2\x91\xC3\x85\xC2\x91\xC3\x85\xC2\x91\xC3\x85\xC2\x91/O=\xC3\x85\xC2\x91/CN=\xC3\x85\xC2\x91\xC3\x85\xC2\x91-test 4. Run easy-mode.yml playbook, the playbook finished without any error. Did I miss something here?
Gaoyun Pei if you create a certificate with non-ascii characters, they may get encoded in the PEM file as ascii characters (as they are base-64 encoded). The issue seems to reproduce when there are non-ascii characters in the file itself, for example, at its comments.
Thanks for the info, Pablo! I'm able to reproduce this issue after adding non-ascii character to the cert file as comment. 1. echo "#TÜRKTRUST" >> /etc/origin/node/client-ca.crt 2. Run easy-mode.yml playbook TASK [openshift_certificate_expiry : Check cert expirys on host] ************************************************************************************************************************************************** fatal: [vm-10-0-77-8.hosted.upshift.rdu2.redhat.com]: FAILED! => {"changed": false, "module_stderr": "Shared connection to vm-10-0-77-8.hosted.upshift.rdu2.redhat.com closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_Nnrfpy/ansible_module_openshift_cert_expiry.py\", line 874, in <module>\r\n main()\r\n File \"/tmp/ansible_Nnrfpy/ansible_module_openshift_cert_expiry.py\", line 577, in main\r\n issuer) = load_and_handle_cert(cert, now, ans_module=module)\r\n File \"/tmp/ansible_Nnrfpy/ansible_module_openshift_cert_expiry.py\", line 280, in load_and_handle_cert\r\n OpenSSL.crypto.FILETYPE_PEM, _cert_string)\r\nUnicodeEncodeError: 'ascii' codec can't encode character u'\\xdc' in position 2: ordinal not in range(128)\r\n", "msg": "MODULE FAILURE", "rc": 1} Verified with openshift-ansible-3.11.97-1.git.0.5bb60b0.el7.noarch.rpm, no such error in this step. TASK [openshift_certificate_expiry : Check cert expirys on host] ************************************************************************************************************************************************** ok: [vm-10-0-76-48.hosted.upshift.rdu2.redhat.com] => {"changed": false, "check_results": {"etcd": [], "kubeconfigs": [], "meta": {"checked_at_time": "2019-03-19 00:20:10.952262", "show_all": "True", "warn_before_date": "2020-03-18 00:20:10.952262", "warning_days": 365}, "ocp_certs": [{"cert_cn": "CN:openshift-signer@1552964964", "days_remaining": 1825, "expiry": "2024-03-17 03:09:24", "health": "ok", "issuer": "CN=openshift-signer@1552964964 ", "path": "/etc/origin/node/client-ca.crt", "serial": 1, "serial_hex": "0x1"}, {"cert_cn": "CN:openshift-signer@1552964964", "days_remaining": 1825, "expiry": "2024-03-17 03:09:24", "health": "ok", "issuer": "CN=openshift-signer@1552964964 ", "path": "/etc/origin/node/client-ca.crt", "serial": 1, "serial_hex": "0x1"}], "registry": [], "router": []}, "msg": "Checked 2 total certificates. Expired/Warning/OK: 0/0/2. Warning window: 365 days", "rc": 0, "summary": {"etcd_certificates": 0, "expired": 0, "kubeconfig_certificates": 0, "ok": 2, "registry_certs": 0, "router_certs": 0, "system_certificates": 2, "total": 2, "warning": 0}, "warn_certs": false}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0636