Bug 1743950 - full chain in custom CA causes controllers to crashloopback
Summary: full chain in custom CA causes controllers to crashloopback
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Joseph Callen
QA Contact: Gaoyun Pei
Depends On:
TreeView+ depends on / blocked
Reported: 2019-08-21 04:18 UTC by Vladislav Walek
Modified: 2019-09-24 08:08 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-09-24 08:08:09 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 11880 0 None closed Bug 1743950: Validate openshift_master_ca_certificate and ca.crt 2020-04-25 10:04:30 UTC
Red Hat Product Errata RHBA-2019:2816 0 None None None 2019-09-24 08:08:17 UTC

Description Vladislav Walek 2019-08-21 04:18:08 UTC
Description of problem:

the controllers service fails to start due:
E0821 00:05:30.608953       1 controllermanager.go:483] Error starting "csrsigning"
F0821 00:05:30.608976       1 controllermanager.go:188] error starting controllers: failed to start certificate controller: error parsing CA cert file "/etc/origin/master/ca.crt": {"code":1003,"message":"the PEM file should contain only one object"}

my ca actually contains two certificates the full chain - root and ca.
The docs are actually saying that the:

If the CA certificate is issued by an intermediate CA, the bundled certificate must contain the full chain (the intermediate and root certificates) for the CA in order to validate child certificates.

Version-Release number of selected component (if applicable):
OpenShift Container Platfrom 3.11

How reproducible:
reproducible on my lab

Steps to Reproduce:
1. create pem cert containing the full chain certs
2. redeploy with playbook
3. check the controllers service

Actual results:

Expected results:

Additional info:
actually customer is hitting the same issue  - running reproducer for customer

Comment 3 David Eads 2019-08-22 18:38:40 UTC
I don't know.  I'd try the install team.

Comment 10 Gaoyun Pei 2019-09-17 10:59:59 UTC
Verify this bug with openshift-ansible-3.11.146-1.git.0.fcedb45.el7.noarch.rpm

1) With a ca-chain cert set in openshift_master_ca_certificate, run playbooks/openshift-master/redeploy-openshift-ca.yml
openshift_master_ca_certificate={"certfile": "/path/to/files/ca/ca-chain.cert.pem", "keyfile": "/path/to/files/ca/intermediate.key.pem"}

It will fail as below:

TASK [Validate openshift_master_ca_certificate when defined] *******************
fatal: [ci-vm-10-0-151-103.hosted.upshift.rdu2.redhat.com]: FAILED! => {"changed": false, "msg": "If defined, openshift_master_ca_certificate must include two parameters: certfile and keyfile. The certfile parameter must contain only the single certificate that signs the OpenShift Container Platform certificates. If you have intermediate certificates in your chain, you must bundle them into a different file. See https://docs.openshift.org/latest/install_config/redeploying_certificates.html#redeploying-new-custom-ca\n"}
	to retry, use: --limit @/home/slave6/workspace/Run-Ansible-Playbooks-Nextge/private-openshift-ansible/playbooks/openshift-master/redeploy-openshift-ca.retry

2) With a ca-chain cert set in openshift_additional_ca, and intermediate CA set in openshift_master_ca_certificate
openshift_master_ca_certificate={"certfile": "/path/to/files/ca/intermediate.cert.pem", "keyfile": "/path/to/files/ca/intermediate.key.pem"}

playbooks/openshift-master/redeploy-openshift-ca.yml could finished successfully.

Check on the master:
* /etc/origin/master/ca.crt was updated as the specified intermediate CA
* /etc/origin/master/additional_ca.crt was updated as the specified ca-chain CA
* /etc/origin/master/ca-bundle.crt was updated as old openshift CA & the specified ca-chain CA
* /etc/origin/node/client-ca.crt was updated as the new content of /etc/origin/master/ca-bundle.crt

Comment 12 errata-xmlrpc 2019-09-24 08:08:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.