Description of problem: During the testing of OCP on self-signed OSP16, we found that only one CA in caReference will be added to the trust pool [ 9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs After research, we found that https://github.com/coreos/ignition/blob/master/internal/resource/http.go#L117, here we use x509.ParseCertificate for the ca-bundle we provided, and this always only read the first CA in the ca-bundle. Version-Release number of selected component (if applicable): 44.81.202002241126-0 ./openshift-install 4.4.0-0.nightly-2020-02-27-230850 built from commit e172a64692cfcb7d9ec37c2b80dfcecd46cd6d22 release image registry.svc.ci.openshift.org/ocp/release@sha256:12c12ad90b10c278f5d299efeca48bf39d496dfd7584339cdfa1cc85143cb08f How reproducible: Always Steps to Reproduce: 1. try to setup a IPI on self-signed OSP with likely the following clouds.yaml which cacert is a ca-bundle contain multiple certs and make sure your CA is not the first one : --- clouds: openstack: auth: auth_url: https://10.8.100.190:13000/v3 username: shiftstack_user password: HIDDEN project_id: e4452838e3494c44a926a5a9863095b4 project_name: shiftstack user_domain_name: Default region_name: regionOne interface: public identity_api_version: 3 cacert: "/home/jenkins/workspace/Launch Environment Flexy/workdir/cacert.crt.20200228-471-9ac12d" 2. Check bootstrap server log 3. Actual results: [ 8.197779] ignition[1073]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5 [ 9.003506] ignition[1073]: GET result: OK [ 9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs [ 9.034710] ignition[1073]: GET https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: attempt #1 [ 62.119530] ignition[1073]: GET error: Get https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: x509: certificate signed by unknown authority [ 62.318905] ignition[1073]: GET https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: attempt #2 [ 62.324235] ignition[1073]: GET error: Get https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: x509: certificate signed by unknown authority Expected results: It should work well Additional info: one workaround is to put the necessary ca be the first one for cacert file
(In reply to weiwei jiang from comment #0) > Actual results: > [ 8.197779] ignition[1073]: GET > http://169.254.169.254/openstack/latest/user_data: attempt #5 > > [ 9.003506] ignition[1073]: GET result: OK > > [ 9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs > [ 9.034710] ignition[1073]: GET > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/ > file: attempt #1 > [ 62.119530] ignition[1073]: GET error: Get > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/ > file: x509: certificate signed by unknown authority This looks like it may be that Ignition does not have the CA cert provided early enough to fetch the next files it needs. You should be able to provide CA certs to Ignition itself via the `security.tls.certificateAuthorities` section of the Ignition config. https://github.com/coreos/ignition/blob/spec2x/doc/configuration-v2_2.md ``` { "ignition": { "config": {}, "security": { "tls": { "certificateAuthorities": [ { "source": "http://some.example.com/path/to/cacert.pem", "verification": {} }, { "source": "http://some.example.com/path/to/anothercacert.pem", "verification": {} } ] } }, "timeouts": {}, "version": "2.2.0" }, ... ``` See also https://bugzilla.redhat.com/show_bug.cgi?id=1735192 where the use of self-signed certs in OSP is discussed at length.
(In reply to Micah Abbott from comment #2) > (In reply to weiwei jiang from comment #0) > > Actual results: > > [ 8.197779] ignition[1073]: GET > > http://169.254.169.254/openstack/latest/user_data: attempt #5 > > > > [ 9.003506] ignition[1073]: GET result: OK > > > > [ 9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs > > [ 9.034710] ignition[1073]: GET > > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/ > > file: attempt #1 > > [ 62.119530] ignition[1073]: GET error: Get > > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/ > > file: x509: certificate signed by unknown authority > > This looks like it may be that Ignition does not have the CA cert provided > early enough to fetch the next files it needs. > > You should be able to provide CA certs to Ignition itself via the > `security.tls.certificateAuthorities` section of the Ignition config. > > https://github.com/coreos/ignition/blob/spec2x/doc/configuration-v2_2.md > > ``` > { > "ignition": { > "config": {}, > "security": { > "tls": { > "certificateAuthorities": [ > { > "source": "http://some.example.com/path/to/cacert.pem", > "verification": {} > }, > { > "source": "http://some.example.com/path/to/anothercacert.pem", > "verification": {} > } > ] > } > }, > "timeouts": {}, > "version": "2.2.0" > }, > ... > ``` > > See also https://bugzilla.redhat.com/show_bug.cgi?id=1735192 where the use > of self-signed certs in OSP is discussed at length. Yes when we split the ca-bundle to multiple caReference, then work well. But does that mean we do not support a ca-bundle in a single certificateAuthorities? any docs?
(In reply to weiwei jiang from comment #3) > (In reply to Micah Abbott from comment #2) > > (In reply to weiwei jiang from comment #0) > > > Actual results: > > > [ 8.197779] ignition[1073]: GET > > > http://169.254.169.254/openstack/latest/user_data: attempt #5 > > > > > > [ 9.003506] ignition[1073]: GET result: OK > > > > > > [ 9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs > > > [ 9.034710] ignition[1073]: GET > > > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/ > > > file: attempt #1 > > > [ 62.119530] ignition[1073]: GET error: Get > > > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/ > > > file: x509: certificate signed by unknown authority > > > > This looks like it may be that Ignition does not have the CA cert provided > > early enough to fetch the next files it needs. > > > > You should be able to provide CA certs to Ignition itself via the > > `security.tls.certificateAuthorities` section of the Ignition config. > > > > https://github.com/coreos/ignition/blob/spec2x/doc/configuration-v2_2.md > > > > ``` > > { > > "ignition": { > > "config": {}, > > "security": { > > "tls": { > > "certificateAuthorities": [ > > { > > "source": "http://some.example.com/path/to/cacert.pem", > > "verification": {} > > }, > > { > > "source": "http://some.example.com/path/to/anothercacert.pem", > > "verification": {} > > } > > ] > > } > > }, > > "timeouts": {}, > > "version": "2.2.0" > > }, > > ... > > ``` > > > > See also https://bugzilla.redhat.com/show_bug.cgi?id=1735192 where the use > > of self-signed certs in OSP is discussed at length. > > Yes when we split the ca-bundle to multiple caReference, then work well. > But does that mean we do not support a ca-bundle in a single > certificateAuthorities? > any docs? or does that means we also need split for a certificate chain?
Micah has started an upstream discussion on this at https://github.com/coreos/ignition/issues/931. It doesn't seem like a regression (and from upstream discussions, might end up being NOTABUG). My suggestion is to retarget this for 4.4.z or 4.5.0.
We are unlikely to have this behavior changed in the 4.4 time frame. The current suggestion is to split the certificate bundle into separate entries. Please see the upstream issue for additional context.
The upstream discussion has progressed towards supporting the use of bundled CAs in Ignition, but it is unlikely we will have this support for 4.5 Currently, there is no 4.6 Target Release for BZ, but that would be the earliest we could deliver the requested functionality.
Note that https://github.com/coreos/ignition/pull/968 is for Ignition spec 3, and will need to be backported to the spec2x branch.
spec2x backport - https://github.com/coreos/ignition/pull/983
Fixed in ignition-0.35.1-13.rhaos4.6.gitb4d18ad.el8
Using the changes to `kola` from @arithx in https://bugzilla.redhat.com/show_bug.cgi?id=1842538#c6, I was able to verify the fix in RHCOS 46.82.202006190240-0 ``` $ bin/kola run -d -b rhcos --ignition-version v2 -p aws --aws-region us-east-1 --aws-ami ami-0139896443a30a831 --aws-profile openshift-dev coreos.ignition.security.* 2020-06-19T21:34:26Z cli: Started logging at level DEBUG 2020-06-19T21:34:26Z cli: Started logging at level DEBUG === RUN coreos.ignition.security.tls.bundle === RUN coreos.ignition.security.tls 2020-06-19T21:34:38Z platform/api/aws: created security group sg-047c385f99203a67f --- PASS: coreos.ignition.security.tls.bundle (198.01s) cluster.go:141: Signature ok cluster.go:141: subject=CN = 172.31.52.86 cluster.go:141: Getting Private key cluster.go:141: Signature ok cluster.go:141: subject=CN = 172.31.52.86 cluster.go:141: Getting Private key --- PASS: coreos.ignition.security.tls (174.50s) PASS, output in _kola_temp/aws-2020-06-19-1734-3381068 ```
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196