Bug 1809780 - openshift-installer does only import a singe CA from clouds.yaml "cacert" bundle for OpenShift on OpenStack
Summary: openshift-installer does only import a singe CA from clouds.yaml "cacert" bun...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.3.z
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 4.5.0
Assignee: Martin André
QA Contact: David Sanz
Depends On:
Blocks: 1810491
TreeView+ depends on / blocked
Reported: 2020-03-03 21:01 UTC by Robert Heinzmann
Modified: 2020-07-13 17:18 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When the CA cert file referenced from the clouds.yaml contained more than one certificate, the complete cert bundle was added to the bootstrap ignition file, but only the first one was actually trusted to gather the ignition file. Consequence: It was not possible to install OCP on OpenStack environment using intermediate certificates signed by non-default trusted authority. Fix: Split the CA cert bundle into separate certs and add them to the ignition file as separate entries. Result: It is now possible to install OCP on OpenStack environment using intermediate certificates signed by non-default trusted authorities.
Clone Of:
: 1810491 1810492 (view as bug list)
Last Closed: 2020-07-13 17:17:45 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Github openshift installer pull 3226 None closed Bug 1809780: OpenStack: pass all CA from cert bundle to bootstrap ign… 2020-09-17 21:05:52 UTC
Red Hat Product Errata RHBA-2020:2409 None None None 2020-07-13 17:18:07 UTC

Description Robert Heinzmann 2020-03-03 21:01:22 UTC
Description of problem:

When installing OpenShift 4.3.2 using IPI on an OpenStack 13 cluster that uses a custom PKI infrastructure, the retrieval of the included ignition file (ignition.config.append[0].source: https://swift_url/path/to/ignition_file) by the bootstrap node fails with 

  "x509: certificate signed by unknown authority". 

This is also true for the 4.3.2 release, which includes support for custom PKI / self signed certificates.

It seems that the installer creates a ignition shim (provided via user-data) with only a single ignition.security.tls.certificateAuthorities[] CA certificate entry. This entry is the whole content of the file specified in the clouds.yaml file with "clouds.${cloudname}.cacert". If the cert bundle specified in the clouds.yaml cacert section contains multiple certificates (e.g. RootCA and Intermedia Issuing CA), ignition will only import the first one listed in this file. If the endpoint of OpenStack does not properly return the Intermediate CA required, SSL can not verify the trust chain, as a intermediate certificate is missing and retrieval of ignition file fails.

Respective code links: 

- Ignition expects a JSON LIST of certificates to add to the truststore:  https://github.com/coreos/ignition/blob/b757ecd7569952bea1b7232872d551ccdd44ed60/internal/resource/http.go#L106-L125

- However the installer only adds a single one: https://github.com/openshift/installer/blob/release-4.3/pkg/tfvars/openstack/bootstrap_ignition.go#L143-L145 

How reproducible:

Steps to Reproduce:

1. Setup an OpenStack cluster using a custom PKI infrastructure using a RootCA, an IntermediateCA and server / endpoint certs signed by the IntermediateCA. Make sure the IntermediateCA is not returned by the Swift or OpenStack endpoints, only the server cert.
2. Create a cert bundle including the root ca and then the intermediate certificate
3. Configure clouds.yaml with the cert bundle configured in 2.
4. Run the Openshift installer using IPI

Actual results:

Bootstrap node can not pull ignition file (2nd stage included by the shim), as the intermediate certificate is not added to the truststore of the bootstrap node

Expected results:

Openshift cluster is created, Bootstrap node installs the cluster.

Additional info:

Openshift Version 4.3.2
Openstack 13

Comment 5 David Sanz 2020-03-30 14:32:13 UTC
Verified on 4.5.0-0.nightly-2020-03-30-083935

Comment 6 Matt 2020-04-21 21:37:44 UTC
Not surprisingly, running into this on 4.4.0-rc.9 installer as well. Would love to get this fixed in 4.4.0 so I can actually deploy a cluster. :)

Comment 7 Matt 2020-04-22 04:01:26 UTC
For my environment, putting the intermediate CA first in the cacert file worked for ignition boot strap. The root CA needed to be in the file as well for the openstack commands to work.

So, cat cacert.pem:
Mxxxx Intermediate CA that signed Openstack endpoint certificates
Mxxxxx Root CA that signed intermediate CA certificate

After that, the bundle in install-config.yaml seemed to work as the proxy certs, which are not signed by the same intermediate CA, were file for pulling containers and whatnot.

Comment 8 Martin André 2020-04-22 10:35:03 UTC
You need to put all the certs required to verify the openstack endpoints certificates in your PEM encoded file.

There was an issue where only the first certificate from the cert bundle was used by the bootstrap node to retrieve the ignition file. This is what this BZ is about. The fix is present in 4.4 also, see https://bugzilla.redhat.com/show_bug.cgi?id=1810491.

Comment 10 errata-xmlrpc 2020-07-13 17:17:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.