Bug 1735192 - [OSP] allow retrieval of ignition files from behind an encrypted endpoint which uses a self-signed certificate
Summary: [OSP] allow retrieval of ignition files from behind an encrypted endpoint whi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.2.0
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ---
: 4.4.0
Assignee: egarcia
QA Contact: David Sanz
URL:
Whiteboard:
Depends On: 1769879
Blocks: 1796822
TreeView+ depends on / blocked
 
Reported: 2019-07-31 19:10 UTC by August Simonelli
Modified: 2023-10-06 18:27 UTC (History)
29 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1796822 (view as bug list)
Environment:
Last Closed: 2020-05-13 21:51:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-api-provider-openstack pull 79 0 None closed Bug 1769879: SSC refractor 2021-02-18 17:10:20 UTC
Github openshift cluster-image-registry-operator pull 445 0 None closed Add support for OpenStack clouds with self-signed certificates 2021-02-18 17:10:19 UTC
Github openshift installer pull 2587 0 'None' closed Bug 1735192: OpenStack Self Signed Certs Fix 2021-02-18 17:10:19 UTC
Github openshift installer pull 2932 0 None closed OpenStack: Add user CA certificate to cloud-config configmap 2021-02-18 17:10:20 UTC
Github openshift machine-api-operator pull 473 0 None closed OpenStack: allow cluster-api-provider-openstack to read configmaps 2021-02-18 17:10:21 UTC
Github openshift machine-config-operator pull 1392 0 None closed OpenStack: Add ca cert to disk on all nodes so kubelet can start up 2021-02-18 17:10:21 UTC
Red Hat Bugzilla 1769879 0 high CLOSED [IPI][OSP] Machine-api cannot create workers on osp envs installed with self-signed certs 2023-10-06 18:45:10 UTC
Red Hat Knowledge Base (Solution) 4735631 0 None None None 2020-02-19 11:49:32 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-13 21:52:05 UTC

Description August Simonelli 2019-07-31 19:10:11 UTC
Description of problem:

When a CoreOS image needs to request ignition files from behind an endpoint (swift, webserver, etc) that is encrypted with a self-signed certificate it fails with an error.

Version-Release number of the following components:

/home/stack/src/github.com/openshift/installer/bin/openshift-install unreleased-master-1469-ge5b5d5efd1d43f5d9449efc798b7f8cde12bb0bb
built from commit e5b5d5efd1d43f5d9449efc798b7f8cde12bb0bb
release image registry.svc.ci.openshift.org/origin/release:4.2

How reproducible:

Place ignition files behind an encrypted endpoint that uses a self-signed or internal CA.

Steps to Reproduce:
1. deploy OCP with terraform
2. attempt to retrieve ignition configs from behind encrypted location that uses a self-signed cert
3.

Actual results:

certificate signed by unknown authority

Here is a specific example from an OSP install using swift:

[   49.236893] ignition[609]: GET error: Get https://192.168.1.20:13808/v1/AUTH_3dd52f403c0c477ba541c54f4055ea90/rhte-tprhz/load-balancer.ign?temp_url_sig=1dae57439c0a8c55cd737ee8cee3dddeabfe0b6a&temp_url_expires=1564598715: x509: certificate signed by unknown authority

Expected results:

Retrieval of ignition files should be possible even when using encryption with a self-signed certificate

Additional info:

Comment 1 Abhinav Dahiya 2019-07-31 19:30:58 UTC
ignition has docs on how to provide ca when requesting... see docshttp://coreos.com/ignition/docs/latest/

Comment 2 August Simonelli 2019-07-31 20:03:44 UTC
(In reply to Abhinav Dahiya from comment #1)
> ignition has docs on how to provide ca when requesting... see
> docshttp://coreos.com/ignition/docs/latest/

Can you point me where it shows this? It can't be IN ignition as this is BEFORE we grab it.
Are you saying COreOS has a way to ASK for ignition files without a CA? Can you link me?

Comment 3 August Simonelli 2019-07-31 20:08:19 UTC
or to specify the CA.
we can't seem to make it work for OpenStack.

Comment 5 Nate Revo 2019-07-31 20:50:13 UTC
We can see where to provide the cacert file in the ignition file.  However, this is in relation to the v4 openshift-installer.  How do we get a CA file injected into ignition from the config.yml

Comment 6 August Simonelli 2019-07-31 20:53:38 UTC
reopening as OpenStack specific for now.

Comment 7 Nate Revo 2019-07-31 22:22:19 UTC
Turns out that OpenStack uses one additional ignition config that other platforms might not.   

In the terraform it is called `data "ignition_config" "redirect"` and is a shim ignition file, pushed via user_data, that then tries to pull the rest of the contents from an object store where the actual bootstrap.ign (and others) are stored.   If this object store (accessed via http) is encrypted with tls and uses a self signed certificate the "shim/redirect" ignition file fails with a tls x.509 error because the CA is not "trusted".   

The installer needs to push the certificate provided by the `additionalTrustBundle` configuration attribute, similar to the way it does for bootstrap.ign, master.ign and node.ign.

This is the part that's causing the bug.

Comment 8 Eric Duen 2019-08-01 19:07:03 UTC
Assign to Emilio for initial review.

Comment 9 Martin André 2019-08-20 09:53:50 UTC
The terraform ignition provider only supports ignition v2.1.0 for now, while we need at least ignition format v2.2.0.

https://github.com/openshift/installer/issues/2225 describes the issue in detail.

There's a feature request to support ignition v2.2.0 in the terraform ignition provider issue tracker at https://github.com/terraform-providers/terraform-provider-ignition/issues/42 with the beginning of a patch. It has been opened for almost a year and has no recent activity.

It's unlikely this will make it for the 4.2 release.

Comment 11 egarcia 2019-10-23 15:01:28 UTC
Design Doc: https://github.com/openshift/enhancements/pull/72
Code: https://github.com/openshift/installer/pull/2544

Awaiting Approval and Review. On target to be delivered in 4.3

Comment 12 Gyanendra Kumar 2019-10-28 15:16:33 UTC
Do we have a workaround for this?  I am installing OCP 4.2 on RH openstack13( self signed TLS) and hitting the same error.

Comment 13 egarcia 2019-10-28 15:34:15 UTC
There is not currently a workaround.

Comment 16 David Sanz 2019-11-07 15:38:15 UTC
Verified on 4.3.0-0.nightly-2019-11-07-113138

[    8.001872] ignition[1096]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5
[    9.181010] ignition[1096]: GET result: OK
[    9.204317] ignition[1096]: Adding "10.0.0.101" to list of CAs
[    9.208204] ignition[1096]: GET https://10.0.0.101:13808/v1/AUTH_19b91a0c7bd44aac9489948c255ee0be/morenod-ocp-mhxnq/72k6kgxfmz4c795q: attempt #1
[...]
[   54.123584] ignition[1096]: GET result: OK
[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (52s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (53s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (53s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (55s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (55s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (56s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (56s / no limit)[   60.198298] ignition[1096]: Adding "10.0.0.101" to list of CAs
[   60.202079] ignition[1096]: Adding "10.0.0.101" to list of CAs
[K[[0;32m  OK[   60.206515] systemd[1]: Started Ignition (fetch).
  [0m] Started Ignition (fetch).
         Startin[   60.211563] ignition[1096]: fetch: fetch complete

Comment 17 David Sanz 2019-11-07 16:04:59 UTC
Sorry, reopening

Ignition file is correctly downloaded, but machine-api controller cannot create workers

I1107 15:47:57.510026       1 controller.go:164] Reconciling Machine "morenod-ocp-mhxnq-worker-xv7p4"
I1107 15:47:57.510361       1 controller.go:376] Machine "morenod-ocp-mhxnq-worker-xv7p4" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
E1107 15:47:57.524429       1 controller.go:279] Failed to check if machine "morenod-ocp-mhxnq-worker-xv7p4" exists: Error checking if instance exists (machine/actuator.go 346): 
Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: Post https://10.0.0.101:13000//v3/auth/tokens: x509: certificate signed by unknown authority
I

Comment 21 egarcia 2019-12-02 19:00:13 UTC
For clarification, this issue is resolved. Instances can all retrieve their ignition configs from services that use self signed certificates in their authentication. There is a further issue that needs to be resolved in order to fully support self signed certificates that is being tracked in the following bug https://bugzilla.redhat.com/show_bug.cgi?id=1769879. This is a separate technical issue and so we chose to track it independently.

Comment 22 Max de Bayser 2019-12-09 17:45:45 UTC
If I understood this is solved in 4.3. But at the present moment the latest release that we can get at RH is 4.2.9. So I'll just leave my workaround here if anyone else can't wait for the next release.

First I've tried adding "interface: internal" to clouds.yaml to point to the http endoints, but that was ignored. What worked was create a new region using the openstack CLI and copy all internal endpoints from the other region (RegionOne) to this one but as public endpoints. With that the terraform openstack provider stopped returning the https URL to the ignition file. This is not very secure, but in a test environment where you can't afford to create signed certificates it works.

Comment 34 Pierre Prinetti 2020-01-31 11:42:23 UTC
This feature is now code-complete.

Comment 35 Pierre Prinetti 2020-01-31 11:45:47 UTC
> Ideally we would've linked all the PRs that addressed this against this bug, the bot is smart enough to move the PR to modified once the last PR linked to the bug merges

OK! Nice to know, thanks. I moved to ON_QA by request by David, I am then going to retroactively link the PRs like you suggested.

Comment 37 David Sanz 2020-01-31 16:30:02 UTC
Verified ignition downloaded from SSL endpoint with self-signed on 4.4.0-0.nightly-2020-01-31-144949

Verified also that workers are correctly created
Verified also that image-registry swift container is created
Verified also worker scalation using machineset

[   11.258522] ignition[1075]: GET https://10.0.0.101:13292/v2/images/2780452b-a6df-41da-a3b8-e350c13b7d8f/file: attempt #1
[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (11s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (12s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (12s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (13s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (13s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (14s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (15s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (15s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (16s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (16s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (17s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (17s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (18s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (19s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (19s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (20s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (20s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (21s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (22s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (22s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (23s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (23s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (24s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (24s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (25s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (26s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (26s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (27s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (27s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (28s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (29s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (29s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (30s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (30s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (31s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (32s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (32s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (33s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (33s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (34s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (34s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (35s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (36s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (36s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (37s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (37s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (38s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (39s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (39s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (40s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (40s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (41s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (41s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (42s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (42s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (43s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (43s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (44s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (44s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (45s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (45s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (46s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (46s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (47s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (47s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (48s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (48s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (49s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (49s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (50s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (50s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (51s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (52s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (52s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (53s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (53s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (54s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (55s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (55s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (56s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (56s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (57s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (57s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (58s / no limit)[   61.262408] random: crng init done
[   61.267303] random: 7 urandom warning(s) missed due to ratelimiting
[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (58s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (59s / no limit)[   62.733167] ignition[1075]: GET result: OK
[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (59s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (1min / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (1min / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (1min 1s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (1min 1s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (1min 2s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (1min 2s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (1min 3s / no limit)[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (1min 3s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (1min 4s / no limit)[K[[0m[0;31m*     [0m] A start job is running for Ignition (fetch) (1min 4s / no limit)[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (fetch) (1min 5s / no limit)[   68.336746] ignition[1075]: Adding "192.168.24.2" to list of CAs
[   68.340338] ignition[1075]: Adding "192.168.24.2" to list of CAs
[   68.343766] ignition[1075]: fetch: fetch complete
[K[[0;32m  OK  [0m] Started Ignition (fetch).
[   68.348433] systemd[1]: Started Ignition (fetch).
         Starting Check for FIPS mode...
[   68.352481] ignition[1075]: fetch: fetch passed
[   68.355295] systemd[1]: Starting Check for FIPS mode...
[   68.358383] ignition[1075]: Ignition finished successfully
[   68.407576] rhcos-fips[1213]: Found /etc/ignition-machine-config-encapsulated.json in Ignition config
[   68.413052] rhcos-fips[1213]: FIPS mode not requested
[   68.418686] systemd[1]: Started Check for FIPS mode.
[[0;32m  OK  [0m] Started Check for FIPS mode.
         Starting Ignition (disks)...
[   68.425113] systemd[1]: Starting Ignition (disks)...
[   68.454513] ignition[1231]: Ignition 0.35.0
[   68.457187] ignition[1231]: Stage: disks
[   68.459718] ignition[1231]: reading system config file "/usr/lib/ignition/base.ign"
[   68.480285] ignition[1231]: Adding "192.168.24.2" to list of CAs
[   68.483681] ignition[1231]: disks: disks passed
[   68.486469] ignition[1231]: Ignition finished successfully
[   68.489632] systemd[1]: Started Ignition (disks).
[[0;32m  OK  [0m] Started Ignition (disks).
         Starting CoreOS Firstboot encryption of root device...
[   68.496423] systemd[1]: Starting CoreOS Firstboot encryption of root device...
[   68.576459] loop: module loaded
[   68.583300] coreos-cryptfs[1241]: coreos-cryptfs: Fetching clevis config
[   68.593805] coreos-cryptfs[1241]: coreos-cryptfs: No Clevis config provided
[   68.597732] coreos-cryptfs[1241]: coreos-cryptfs: No Clevis pin provided, encryption is not enabled
[[0;32m  OK  [0m] Started CoreOS Firstboot encryption of root device.
[   68.613041] systemd[1]: Started CoreOS Firstboot encryption of root device.
[   68.616634] systemd[1]: Starting CoreOS LUKS Opener...
         Starting CoreOS LUKS Opener...
[   68.696894] coreos-cryptfs[1263]: coreos-cryptfs: /dev/vda4 uses the 'cipher_null-ecb'
[   68.701763] coreos-cryptfs[1263]: coreos-cryptfs: /dev/vda4 is not encrypted. Device will be mounted as device-mapper linear target.
[   69.017632]  vda: vda1 vda2 vda3 vda4
[   69.023420] coreos-cryptfs[1263]: CHANGED: partition=4 start=1050624 old: size=5937152 end=6987776 new: size=82835423,end=83886047
[   69.061945] systemd[1]: Started CoreOS LUKS Opener.
[[0;32m  OK  [0m] Started CoreOS LUKS Opener.
[   69.088261] systemd[1]: Starting Ignition OSTree: Mount (firstboot) /sysroot...
         Starting Ignition OSTree: Mount (firstboot) /sysroot...
[   69.234469] SGI XFS with ACLs, security attributes, no debug enabled
[   69.241964] XFS (dm-0): Mounting V5 Filesystem
[   69.613513] XFS (dm-0): Ending clean mount
[   69.616634] XFS (dm-0): Quotacheck needed: Please wait.
[   69.708289] XFS (dm-0): Quotacheck: Done.
[[0;32m  OK  [[   69.714682] systemd[1]: Started Ignition OSTree: Mount (firstboot) /sysroot.
0m] Started Ignition OSTree: Mount (firstboot) /sysroot.
         Startin[   69.722795] systemd[1]: Starting OSTree Prepare OS/...
g OSTree Prepare OS/...
[   69.740395] ostree-prepare-root[1350]: Resolved OSTree target to: /sysroot/ostree/deploy/rhcos/deploy/f61524fda480c611dcd25629fd15eb6de27a306689261c211dbc8e88c19a5219.0
[[0;32m  OK  [[   69.751063] systemd[1]: Started OSTree Prepare OS/.
0m] Started OSTree Prepare OS/.
[[0;32m  OK  [[   69.754981] systemd[1]: Reached target Initrd Root File System.
0m] Reached target Initrd Root File System.
         Startin[   69.760379] systemd[1]: Starting Mount OSTree /var...
g Mount OSTree /var...
[   69.770309] ignition-ostree-mount-var[1353]: Mounting /sysroot/sysroot/ostree/deploy/rhcos/var
[[0;32m  OK  [[   69.776407] systemd[1]: Started Mount OSTree /var.
0m] Started Mount OSTree /var.
         Startin[   69.781154] systemd[1]: Starting Ignition (mount) [stub]...
g Ignition (mount) [stub]...
[[0;32m  OK  [[   69.785899] systemd[1]: Started Ignition (mount) [stub].
0m] Started Ignition (mount) [stub].
[[0;32m  OK  [0m] Reached target Ignition Complete.
[   69.791701] systemd[1]: Reached target Ignition Complete.




[stack@undercloud-0 morenod]$ openstack server list
+--------------------------------------+--------------------------------+--------+-----------------------------------------------------+-------------------------+-----------+
| ID                                   | Name                           | Status | Networks                                            | Image                   | Flavor    |
+--------------------------------------+--------------------------------+--------+-----------------------------------------------------+-------------------------+-----------+
| e47e3de6-b2b1-4833-90fc-60f69011ac2d | morenod-ssl-htrmd-worker-kckq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.20             | morenod-ssl-htrmd-rhcos | m4.xlarge |
| c21436b6-fdf5-4305-b5a5-ae8040140123 | morenod-ssl-htrmd-worker-hqxx5 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.19             | morenod-ssl-htrmd-rhcos | m4.xlarge |
| 2e4b2056-2292-42b4-8e45-e4dbd0f155e4 | morenod-ssl-htrmd-worker-2ssq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.17             | morenod-ssl-htrmd-rhcos | m4.xlarge |
| 5003b464-f8ce-42d3-8198-fc78e0c5d08f | morenod-ssl-htrmd-master-0     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.21             | morenod-ssl-htrmd-rhcos | m4.xlarge |
| e8f364f4-0464-41bc-8b10-3609f5066654 | morenod-ssl-htrmd-master-2     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.16             | morenod-ssl-htrmd-rhcos | m4.xlarge |
| b5313ee0-bbce-4772-986a-56250af47bcd | morenod-ssl-htrmd-master-1     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.24             | morenod-ssl-htrmd-rhcos | m4.xlarge |
| c77af594-3540-46af-b517-a894cc80c9eb | morenod-ssl-htrmd-bootstrap    | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.14, 10.0.0.202 | morenod-ssl-htrmd-rhcos | m4.xlarge |
+--------------------------------------+--------------------------------+--------+-----------------------------------------------------+-------------------------+-----------+

$ openstack container list
+----------------------------------------------------------------+
| Name                                                           |
+----------------------------------------------------------------+
| morenod-ssl-htrmd-image-registry-amdmgtavlccstxdnxsaoyepwbeyqx |
+----------------------------------------------------------------+

NAME                             STATUS   ROLES    AGE     VERSION
morenod-ssl-htrmd-master-0	 Ready    master   38m     v1.17.1
morenod-ssl-htrmd-master-1	 Ready    master   38m     v1.17.1
morenod-ssl-htrmd-master-2	 Ready    master   38m     v1.17.1
morenod-ssl-htrmd-worker-2ssq2   Ready    worker   16m     v1.17.1
morenod-ssl-htrmd-worker-42c8w   Ready    worker   3m21s   v1.17.1
morenod-ssl-htrmd-worker-hqxx5   Ready    worker   16m     v1.17.1
morenod-ssl-htrmd-worker-kckq2   Ready    worker   17m     v1.17.1
NAME                             PHASE          TYPE        REGION   ZONE   AGE
morenod-ssl-htrmd-master-0	 Running        m4.xlarge            nova   40m
morenod-ssl-htrmd-master-1	 Running        m4.xlarge            nova   40m
morenod-ssl-htrmd-master-2	 Running        m4.xlarge            nova   40m
morenod-ssl-htrmd-worker-2ssq2   Running        m4.xlarge            nova   33m
morenod-ssl-htrmd-worker-42c8w   Running        m4.xlarge            nova   8m5s
morenod-ssl-htrmd-worker-hqxx5   Running        m4.xlarge            nova   33m
morenod-ssl-htrmd-worker-kckq2   Running        m4.xlarge            nova   33m

[stack@undercloud-0 morenod]$ openstack server list
+--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+
| ID                                   | Name                           | Status | Networks                                | Image                   | Flavor    |
+--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+
| 83c515da-2719-4251-8270-57c5b93f20da | morenod-ssl-htrmd-worker-42c8w | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.14 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| e47e3de6-b2b1-4833-90fc-60f69011ac2d | morenod-ssl-htrmd-worker-kckq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.20 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| c21436b6-fdf5-4305-b5a5-ae8040140123 | morenod-ssl-htrmd-worker-hqxx5 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.19 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| 2e4b2056-2292-42b4-8e45-e4dbd0f155e4 | morenod-ssl-htrmd-worker-2ssq2 | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.17 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| 5003b464-f8ce-42d3-8198-fc78e0c5d08f | morenod-ssl-htrmd-master-0     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.21 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| e8f364f4-0464-41bc-8b10-3609f5066654 | morenod-ssl-htrmd-master-2     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.16 | morenod-ssl-htrmd-rhcos | m4.xlarge |
| b5313ee0-bbce-4772-986a-56250af47bcd | morenod-ssl-htrmd-master-1     | ACTIVE | morenod-ssl-htrmd-openshift=10.196.0.24 | morenod-ssl-htrmd-rhcos | m4.xlarge |
+--------------------------------------+--------------------------------+--------+-----------------------------------------+-------------------------+-----------+

Comment 38 dlbewley 2020-01-31 19:52:27 UTC
Did the location of the tls bundle in install-config.yaml change? 

Previously it worked at .additionalTrustBundle. I know am back to seeing this on bootstrap node.

`[  281.539052] ignition[1067]: GET error: Get https://openstack.domain.com:13292/v2/images/98cf7fd6-e2d8-4279-83e6-e7de32f3793b/file: x509: certificate signed by unknown authority`

Testing with
openshift-install v4.4.0
built from commit e1a6a984b73a818a48f800eecc05ebbc526fc5b9
release image quay.io/openshift-release-dev/ocp-release-nightly@sha256:e1f4190d3a28efb01edb3a42aa138651cab813dc92e8aeef0d5be67fadd0f987

Comment 39 dlbewley 2020-01-31 20:27:55 UTC
Forgive me. I found https://github.com/openshift/installer/commit/2d9ae15a28981a01a6c7503458725c6a14f0ea4a#diff-cf67b77422d61ae5593ac620f6e9b054 and learned to place the cacert in my clouds-public.yaml. I now see worker nodes being created!

Comment 48 Vincent S. Cojot 2020-04-08 02:04:46 UTC
I'm hitting this on OSP13z11 with a Certificate signed by my own CA (obviously, I'm not with Verisign so my CA chain needs to be added somewhere).

1) after preparing the OSP enviroment (in a similar fashion to [1] and [2]), I ran:
./openshift-install create cluster --log-level debug

2) Terraform starts and then hangs after a few minutes:

DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.openshift.lasthome.solace.krynn:6443/version?timeout=32s: dial tcp 10.0.130.215:6443: connect: connection refused

3) In the console of the bootstrap node, I see this:
  532.452676] ignition[724]: GET error: Get https://10.0.130.77:13808/swift/v1/AUTH_f7e33616a3fe42729523b5d296a42111/ocp4-m5fgd/2fspx766mqj86crv: x509: certificate signed by unknown authority
[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (fetch) (8min 44s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (8min 45s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 45s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (8min 46s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (8min 46s / no limit)[K[     [0;31m*[0m] A start job is running for Ignition (fetch) (8min 47s / no limit)[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (fetch) (8min 47s / no limit)[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (fetch) (8min 48s / no limit)[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (fetch) (8min 48s / no limit)[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (fetch) (8min 49s / no limit)[  537.453694] ignition[724]: GET https://10.0.130.77:13808/swift/v1/AUTH_f7e33616a3fe42729523b5d296a42111/ocp4-m5fgd/2fspx766mqj86crv: attempt #108
[  537.495145] ignition[724]: GET error: Get https://10.0.130.77:13808/swift/v1/AUTH_f7e33616a3fe42729523b5d296a42111/ocp4-m5fgd/2fspx766mqj86crv: x509: certificate signed by unknown authority

The problem here is that I seem to be using OCP 4.3.9 so this shouldn't be happening, right?

(overcloud) [stack@osp13p osd-ocp-demo]$ ./openshift-install version
./openshift-install 4.3.9
built from commit 64fccd954517812eab166d38c7fc5bf71b219b7e
release image quay.io/openshift-release-dev/ocp-release@sha256:f0fada3c8216dc17affdd3375ff845b838ef9f3d67787d3d42a88dcd0f328eea

[1] : https://developers.redhat.com/blog/2020/02/06/red-hat-openshift-4-2-ipi-on-openstack-13-all-in-one-setup/
[2] : https://kdjlab.com/openshift-4-2-on-red-hat-openstack-platform-13

Comment 49 August Simonelli 2020-04-08 05:17:17 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1735192#c48 was missing the cacert reference in clouds.yaml. Resolved when that was added and self signed cert worked.

Comment 50 Vincent S. Cojot 2020-04-08 12:16:58 UTC
Thank you August. Yes, the missing cacert was my issue.
For the record, here's a copy of my clouds.yaml:

clouds:
  openstack:
    auth:
      auth_url: https://<some ip>:13000/v3
      username: "ocp-user"
      password: "<password>"
      project_id: f7e3[..digits...]42111
      project_name: "ocp-tenant"
      user_domain_name: "Default"
    cacert: /home//stack/OSP/ca.crt.pem <======================================
    region_name: "regionOne"
    interface: "public"
    identity_api_version: 3

Comment 52 errata-xmlrpc 2020-05-13 21:51:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.