Bug 1809780

Summary: openshift-installer does only import a singe CA from clouds.yaml "cacert" bundle for OpenShift on OpenStack
Product: OpenShift Container Platform Reporter: Robert Heinzmann <rheinzma>
Component: InstallerAssignee: Martin André <m.andre>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: asimonel, jialiu, m.andre, matthew.piechota, pkanthal, pprinett, rsandu
Version: 4.3.z   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: When the CA cert file referenced from the clouds.yaml contained more than one certificate, the complete cert bundle was added to the bootstrap ignition file, but only the first one was actually trusted to gather the ignition file. Consequence: It was not possible to install OCP on OpenStack environment using intermediate certificates signed by non-default trusted authority. Fix: Split the CA cert bundle into separate certs and add them to the ignition file as separate entries. Result: It is now possible to install OCP on OpenStack environment using intermediate certificates signed by non-default trusted authorities.
 Story Points: ---
Clone Of:
: 1810491 1810492 (view as bug list) Environment:
Last Closed: 2020-07-13 17:17:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1810491    

Description Robert Heinzmann 2020-03-03 21:01:22 UTC 
Description of problem:

When installing OpenShift 4.3.2 using IPI on an OpenStack 13 cluster that uses a custom PKI infrastructure, the retrieval of the included ignition file (ignition.config.append[0].source: https://swift_url/path/to/ignition_file) by the bootstrap node fails with 

  "x509: certificate signed by unknown authority". 

This is also true for the 4.3.2 release, which includes support for custom PKI / self signed certificates.

It seems that the installer creates a ignition shim (provided via user-data) with only a single ignition.security.tls.certificateAuthorities[] CA certificate entry. This entry is the whole content of the file specified in the clouds.yaml file with "clouds.${cloudname}.cacert". If the cert bundle specified in the clouds.yaml cacert section contains multiple certificates (e.g. RootCA and Intermedia Issuing CA), ignition will only import the first one listed in this file. If the endpoint of OpenStack does not properly return the Intermediate CA required, SSL can not verify the trust chain, as a intermediate certificate is missing and retrieval of ignition file fails.

Respective code links: 

- Ignition expects a JSON LIST of certificates to add to the truststore:  https://github.com/coreos/ignition/blob/b757ecd7569952bea1b7232872d551ccdd44ed60/internal/resource/http.go#L106-L125

- However the installer only adds a single one: https://github.com/openshift/installer/blob/release-4.3/pkg/tfvars/openstack/bootstrap_ignition.go#L143-L145 

How reproducible:

Steps to Reproduce:

1. Setup an OpenStack cluster using a custom PKI infrastructure using a RootCA, an IntermediateCA and server / endpoint certs signed by the IntermediateCA. Make sure the IntermediateCA is not returned by the Swift or OpenStack endpoints, only the server cert.
2. Create a cert bundle including the root ca and then the intermediate certificate
3. Configure clouds.yaml with the cert bundle configured in 2.
4. Run the Openshift installer using IPI

Actual results:

Bootstrap node can not pull ignition file (2nd stage included by the shim), as the intermediate certificate is not added to the truststore of the bootstrap node

Expected results:

Openshift cluster is created, Bootstrap node installs the cluster.

Additional info:

Openshift Version 4.3.2
Openstack 13
Comment 5 David Sanz 2020-03-30 14:32:13 UTC 
Verified on 4.5.0-0.nightly-2020-03-30-083935
Comment 6 Matt 2020-04-21 21:37:44 UTC 
Not surprisingly, running into this on 4.4.0-rc.9 installer as well. Would love to get this fixed in 4.4.0 so I can actually deploy a cluster. :)
Comment 7 Matt 2020-04-22 04:01:26 UTC 
For my environment, putting the intermediate CA first in the cacert file worked for ignition boot strap. The root CA needed to be in the file as well for the openstack commands to work.

So, cat cacert.pem:
----- BEGIN CERTIFICATE -----
Mxxxx Intermediate CA that signed Openstack endpoint certificates
----- END CERTIFICATE -----
----- BEGIN CERTIFICATE -----
Mxxxxx Root CA that signed intermediate CA certificate
----- END CERTIFICATE -----

After that, the bundle in install-config.yaml seemed to work as the proxy certs, which are not signed by the same intermediate CA, were file for pulling containers and whatnot.
Comment 8 Martin André 2020-04-22 10:35:03 UTC 
You need to put all the certs required to verify the openstack endpoints certificates in your PEM encoded file.

There was an issue where only the first certificate from the cert bundle was used by the bootstrap node to retrieve the ignition file. This is what this BZ is about. The fix is present in 4.4 also, see https://bugzilla.redhat.com/show_bug.cgi?id=1810491.
Comment 10 errata-xmlrpc 2020-07-13 17:17:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409