Bug 1808286

Summary: Ignition can not handle multiple CAs in one caReference
Product: OpenShift Container Platform Reporter: weiwei jiang <wjiang>
Component: RHCOSAssignee: Sohan Kunkerkar <skunkerk>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: bbreard, bgilbert, dustymabe, imcleod, jlebon, jligon, miabbott, nstielau, smilner
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1842538 (view as bug list) Environment:
Last Closed: 2020-10-27 15:55:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1842538    

Description weiwei jiang 2020-02-28 07:33:28 UTC 
Description of problem:
During the testing of OCP on self-signed OSP16, we found that only one CA in caReference will be added to the trust pool
[    9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs

After research, we found that https://github.com/coreos/ignition/blob/master/internal/resource/http.go#L117,
here we use x509.ParseCertificate for the ca-bundle we provided, and this always only read the first CA in the ca-bundle.



Version-Release number of selected component (if applicable):
44.81.202002241126-0

./openshift-install 4.4.0-0.nightly-2020-02-27-230850
built from commit e172a64692cfcb7d9ec37c2b80dfcecd46cd6d22
release image registry.svc.ci.openshift.org/ocp/release@sha256:12c12ad90b10c278f5d299efeca48bf39d496dfd7584339cdfa1cc85143cb08f

How reproducible:
Always

Steps to Reproduce:
1. try to setup a IPI on self-signed OSP with likely the following clouds.yaml which cacert is a ca-bundle contain multiple certs and make sure your CA is not the first one :
---
clouds:
  openstack:
    auth:
      auth_url: https://10.8.100.190:13000/v3
      username: shiftstack_user
      password: HIDDEN
      project_id: e4452838e3494c44a926a5a9863095b4
      project_name: shiftstack
      user_domain_name: Default
    region_name: regionOne
    interface: public
    identity_api_version: 3
    cacert: "/home/jenkins/workspace/Launch Environment Flexy/workdir/cacert.crt.20200228-471-9ac12d"
2. Check bootstrap server log
3.

Actual results:
[    8.197779] ignition[1073]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5                                                                                                                   
[    9.003506] ignition[1073]: GET result: OK                                                                                                                                                                      
[    9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs 
[    9.034710] ignition[1073]: GET https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: attempt #1     
[   62.119530] ignition[1073]: GET error: Get https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: x509: certificate signed by unknown authority                                        
[   62.318905] ignition[1073]: GET https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: attempt #2                                                                                      
[   62.324235] ignition[1073]: GET error: Get https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/file: x509: certificate signed by unknown authority    

Expected results:
It should work well

Additional info:
one workaround is to put the necessary ca be the first one for cacert file
Comment 2 Micah Abbott 2020-03-03 21:46:59 UTC 
(In reply to weiwei jiang from comment #0)
> Actual results:
> [    8.197779] ignition[1073]: GET
> http://169.254.169.254/openstack/latest/user_data: attempt #5               
> 
> [    9.003506] ignition[1073]: GET result: OK                               
> 
> [    9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs 
> [    9.034710] ignition[1073]: GET
> https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/
> file: attempt #1     
> [   62.119530] ignition[1073]: GET error: Get
> https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/
> file: x509: certificate signed by unknown authority                         

This looks like it may be that Ignition does not have the CA cert provided early enough to fetch the next files it needs.

You should be able to provide CA certs to Ignition itself via the `security.tls.certificateAuthorities` section of the Ignition config.

https://github.com/coreos/ignition/blob/spec2x/doc/configuration-v2_2.md

```
{
  "ignition": {
    "config": {},
    "security": {
      "tls": {
         "certificateAuthorities": [
           {
             "source": "http://some.example.com/path/to/cacert.pem",
             "verification": {}
           },
           {
             "source": "http://some.example.com/path/to/anothercacert.pem",
             "verification": {}
           }
         ]
       }
    },
    "timeouts": {},
    "version": "2.2.0"
  },
...
```

See also https://bugzilla.redhat.com/show_bug.cgi?id=1735192 where the use of self-signed certs in OSP is discussed at length.
Comment 3 weiwei jiang 2020-03-04 01:29:07 UTC 
(In reply to Micah Abbott from comment #2)
> (In reply to weiwei jiang from comment #0)
> > Actual results:
> > [    8.197779] ignition[1073]: GET
> > http://169.254.169.254/openstack/latest/user_data: attempt #5               
> > 
> > [    9.003506] ignition[1073]: GET result: OK                               
> > 
> > [    9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs 
> > [    9.034710] ignition[1073]: GET
> > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/
> > file: attempt #1     
> > [   62.119530] ignition[1073]: GET error: Get
> > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/
> > file: x509: certificate signed by unknown authority                         
> 
> This looks like it may be that Ignition does not have the CA cert provided
> early enough to fetch the next files it needs.
> 
> You should be able to provide CA certs to Ignition itself via the
> `security.tls.certificateAuthorities` section of the Ignition config.
> 
> https://github.com/coreos/ignition/blob/spec2x/doc/configuration-v2_2.md
> 
> ```
> {
>   "ignition": {
>     "config": {},
>     "security": {
>       "tls": {
>          "certificateAuthorities": [
>            {
>              "source": "http://some.example.com/path/to/cacert.pem",
>              "verification": {}
>            },
>            {
>              "source": "http://some.example.com/path/to/anothercacert.pem",
>              "verification": {}
>            }
>          ]
>        }
>     },
>     "timeouts": {},
>     "version": "2.2.0"
>   },
> ...
> ```
> 
> See also https://bugzilla.redhat.com/show_bug.cgi?id=1735192 where the use
> of self-signed certs in OSP is discussed at length.

Yes when we split the ca-bundle to multiple caReference, then work well.
But does that mean we do not support a ca-bundle in a single certificateAuthorities?
any docs?
Comment 4 weiwei jiang 2020-03-04 01:46:42 UTC 
(In reply to weiwei jiang from comment #3)
> (In reply to Micah Abbott from comment #2)
> > (In reply to weiwei jiang from comment #0)
> > > Actual results:
> > > [    8.197779] ignition[1073]: GET
> > > http://169.254.169.254/openstack/latest/user_data: attempt #5               
> > > 
> > > [    9.003506] ignition[1073]: GET result: OK                               
> > > 
> > > [    9.032773] ignition[1073]: Adding "Installer-QE-CA" to list of CAs 
> > > [    9.034710] ignition[1073]: GET
> > > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/
> > > file: attempt #1     
> > > [   62.119530] ignition[1073]: GET error: Get
> > > https://10.8.100.190:13292/v2/images/e838663c-cc1c-4e8c-8593-5d8992e62439/
> > > file: x509: certificate signed by unknown authority                         
> > 
> > This looks like it may be that Ignition does not have the CA cert provided
> > early enough to fetch the next files it needs.
> > 
> > You should be able to provide CA certs to Ignition itself via the
> > `security.tls.certificateAuthorities` section of the Ignition config.
> > 
> > https://github.com/coreos/ignition/blob/spec2x/doc/configuration-v2_2.md
> > 
> > ```
> > {
> >   "ignition": {
> >     "config": {},
> >     "security": {
> >       "tls": {
> >          "certificateAuthorities": [
> >            {
> >              "source": "http://some.example.com/path/to/cacert.pem",
> >              "verification": {}
> >            },
> >            {
> >              "source": "http://some.example.com/path/to/anothercacert.pem",
> >              "verification": {}
> >            }
> >          ]
> >        }
> >     },
> >     "timeouts": {},
> >     "version": "2.2.0"
> >   },
> > ...
> > ```
> > 
> > See also https://bugzilla.redhat.com/show_bug.cgi?id=1735192 where the use
> > of self-signed certs in OSP is discussed at length.
> 
> Yes when we split the ca-bundle to multiple caReference, then work well.
> But does that mean we do not support a ca-bundle in a single
> certificateAuthorities?
> any docs?

or does that means we also need split for a certificate chain?
Comment 5 Jonathan Lebon 2020-03-05 16:37:35 UTC 
Micah has started an upstream discussion on this at https://github.com/coreos/ignition/issues/931.

It doesn't seem like a regression (and from upstream discussions, might end up being NOTABUG). My suggestion is to retarget this for 4.4.z or 4.5.0.
Comment 6 Micah Abbott 2020-03-05 17:25:20 UTC 
We are unlikely to have this behavior changed in the 4.4 time frame.  The current suggestion is to split the certificate bundle into separate entries.

Please see the upstream issue for additional context.
Comment 7 Micah Abbott 2020-04-20 15:17:44 UTC 
The upstream discussion has progressed towards supporting the use of bundled CAs in Ignition, but it is unlikely we will have this support for 4.5

Currently, there is no 4.6 Target Release for BZ, but that would be the earliest we could deliver the requested functionality.
Comment 8 Benjamin Gilbert 2020-05-14 23:22:12 UTC 
Note that https://github.com/coreos/ignition/pull/968 is for Ignition spec 3, and will need to be backported to the spec2x branch.
Comment 9 Micah Abbott 2020-05-22 13:50:22 UTC 
spec2x backport - https://github.com/coreos/ignition/pull/983
Comment 10 Micah Abbott 2020-06-01 13:40:34 UTC 
Fixed in ignition-0.35.1-13.rhaos4.6.gitb4d18ad.el8
Comment 13 Micah Abbott 2020-06-19 21:45:32 UTC 
Using the changes to `kola` from @arithx in https://bugzilla.redhat.com/show_bug.cgi?id=1842538#c6, I was able to verify the fix in RHCOS 46.82.202006190240-0


```
$ bin/kola run -d -b rhcos --ignition-version v2 -p aws --aws-region us-east-1 --aws-ami ami-0139896443a30a831 --aws-profile openshift-dev coreos.ignition.security.*
2020-06-19T21:34:26Z cli: Started logging at level DEBUG
2020-06-19T21:34:26Z cli: Started logging at level DEBUG
=== RUN   coreos.ignition.security.tls.bundle
=== RUN   coreos.ignition.security.tls
2020-06-19T21:34:38Z platform/api/aws: created security group sg-047c385f99203a67f
--- PASS: coreos.ignition.security.tls.bundle (198.01s)
        cluster.go:141: Signature ok
        cluster.go:141: subject=CN = 172.31.52.86
        cluster.go:141: Getting Private key
        cluster.go:141: Signature ok
        cluster.go:141: subject=CN = 172.31.52.86
        cluster.go:141: Getting Private key
--- PASS: coreos.ignition.security.tls (174.50s)
PASS, output in _kola_temp/aws-2020-06-19-1734-3381068
```
Comment 15 errata-xmlrpc 2020-10-27 15:55:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196