Description of problem: When installing OpenShift 4.3.2 using IPI on an OpenStack 13 cluster that uses a custom PKI infrastructure, the retrieval of the included ignition file (ignition.config.append[0].source: https://swift_url/path/to/ignition_file) by the bootstrap node fails with "x509: certificate signed by unknown authority". This is also true for the 4.3.2 release, which includes support for custom PKI / self signed certificates. It seems that the installer creates a ignition shim (provided via user-data) with only a single ignition.security.tls.certificateAuthorities[] CA certificate entry. This entry is the whole content of the file specified in the clouds.yaml file with "clouds.${cloudname}.cacert". If the cert bundle specified in the clouds.yaml cacert section contains multiple certificates (e.g. RootCA and Intermedia Issuing CA), ignition will only import the first one listed in this file. If the endpoint of OpenStack does not properly return the Intermediate CA required, SSL can not verify the trust chain, as a intermediate certificate is missing and retrieval of ignition file fails. Respective code links: - Ignition expects a JSON LIST of certificates to add to the truststore: https://github.com/coreos/ignition/blob/b757ecd7569952bea1b7232872d551ccdd44ed60/internal/resource/http.go#L106-L125 - However the installer only adds a single one: https://github.com/openshift/installer/blob/release-4.3/pkg/tfvars/openstack/bootstrap_ignition.go#L143-L145 How reproducible: Steps to Reproduce: 1. Setup an OpenStack cluster using a custom PKI infrastructure using a RootCA, an IntermediateCA and server / endpoint certs signed by the IntermediateCA. Make sure the IntermediateCA is not returned by the Swift or OpenStack endpoints, only the server cert. 2. Create a cert bundle including the root ca and then the intermediate certificate 3. Configure clouds.yaml with the cert bundle configured in 2. 4. Run the Openshift installer using IPI Actual results: Bootstrap node can not pull ignition file (2nd stage included by the shim), as the intermediate certificate is not added to the truststore of the bootstrap node Expected results: Openshift cluster is created, Bootstrap node installs the cluster. Additional info: Openshift Version 4.3.2 Openstack 13
Verified on 4.5.0-0.nightly-2020-03-30-083935
Not surprisingly, running into this on 4.4.0-rc.9 installer as well. Would love to get this fixed in 4.4.0 so I can actually deploy a cluster. :)
For my environment, putting the intermediate CA first in the cacert file worked for ignition boot strap. The root CA needed to be in the file as well for the openstack commands to work. So, cat cacert.pem: ----- BEGIN CERTIFICATE ----- Mxxxx Intermediate CA that signed Openstack endpoint certificates ----- END CERTIFICATE ----- ----- BEGIN CERTIFICATE ----- Mxxxxx Root CA that signed intermediate CA certificate ----- END CERTIFICATE ----- After that, the bundle in install-config.yaml seemed to work as the proxy certs, which are not signed by the same intermediate CA, were file for pulling containers and whatnot.
You need to put all the certs required to verify the openstack endpoints certificates in your PEM encoded file. There was an issue where only the first certificate from the cert bundle was used by the bootstrap node to retrieve the ignition file. This is what this BZ is about. The fix is present in 4.4 also, see https://bugzilla.redhat.com/show_bug.cgi?id=1810491.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409